Empirical Studies (Live Experiment)

REFSQ’11 offers a place for empirical research. As part of the Empirical Track the Empirical Studies contain a live experiment in which everyone can participate. Below you will find more information about the experiment and about the Empirical Studies in general.

Live Experiment about Risk Estimation

The live experiment will be a controlled experiment about the estimation of IT-related risk. Such risk estimation is the basis for risk-based requirements prioritization, for such requirements which contribute to the system value by reducing risk.

Research questions: The experiment investigates the influence of the following two factors on the quality of risk estimation:

  1. Fuzziness: Are the estimates achieved for measurable risks better than those for fuzzy or abstract risks?
  2. Scales: Are damage and probability of risks easier to estimate on a scale of points than in percent or Euro?

Participants´ benefits: The participants can make a practical experience with the estimation of requirements-related risk, what is more difficult than they probably expect.

Profile of intended participants: No special knowledge or experience is needed. Former experiments have indicated that more experience does not automatically mean better risk estimates.

Agenda of the experiment:

  • 14:00-14:20 introductory presentation
  • 14:20-14:50 risk estimation on questionnaires
  • 14:50-15:10 break
  • 15:10-15:30 discussion of preliminary results and method

Study organizer:

Andrea Herrmann, Axivion GmbH

Replication Package:

This is the replication package for the risk-estimation experiment executed and published at REFSQ 2011.

The ppt slides are the introduction presentation.

Each participant gets the following material, depending on whether (s)he belongs
to Group 1 or Group 2:

For typing in the data from the questionnaires – or alternatively to execute
the experiment on computers – you can use the excel templates:

for Group 1/2 respectively.

Call for Participation

The discussion at recent REFSQs have confirmed the strong need for empirical validation of the effectiveness for our RE methods, but the literature to date, including that of REFSQ, could show more of this validation. This lack is assumed to be at least partly due to the difficulty of finding and persuading the participation of a sufficient number of suitable experimental subjects. Therefore, REFSQ 2011 will offer an opportunity to conduct a small number of empirical studies during the conference itself. The goals of this opportunity, besides that of permitting the conduct of some experiments, are to raise awareness for the necessity and benefits of empirical studies and to show that participating in them is not dangerous to one’s health. Furthermore, we want to bring together the community of researchers and practitioners who are interested in empirical studies.

Therefore, we support empirical studies to be conducted during the REFSQ conference. Possibilities include and are not restricted to

  • a controlled experiment, designed to require no more than 90 minutes
  • a series of interviews, each interview shall be designed to require no more than 30 minutes
  • an in-situ survey (questionnaire), designed to require no more than 30 minutes

The featured studies are designed to be performed during conference itself with REFSQ participants from industry and academia volunteering to be subjects. The results of the study will be published one month after the conference!

More information regarding the controlled experiment about the estimation of IT-related risk can be found here.

Event co-chair

Joerg Doerr, Fraunhofer IESE, Germany

PC members

Ian Alexander, Scenario Plus, UK

Claudia P. Ayala, UPC

Joy Beatty, Seilevel, USA

Marcus Ciolkowski, Fraunhofer IESE, Germany

Samuel Fricker, University of Zürich, Switzerland

Frank Houdek, Daimler AG, Germany

Isabel John, Fraunhofer IESE, Germany

Natalia Juristo, Universidad Politécnica de Madrid, Spain

Søren Lauesen, IT-University of Copenhagen, Denmark

Anne Persson, University of Skövde, Sweden

Gil Regev, EPFL and Itecor, Switzerland

Björn Regnell, Lund University, Sweden

Camille Salinesi, Univ. Paris 1 – Panthéon Sorbonne, France

Erik Simmons, Intel, USA

Roel Wieringa, University of Twente, Netherlands