We provide an assessment of the status of empirical software research by analyzing all refereed articles that appeared in the Journal of Empirical Software Engineering from its first issue in January 1996 through June 2006. The journal publishes empirical software research exclusively and it is the only journal to do so. The main findings are: 1. The dominant empirical methods are experiments and case studies. Other methods (correlational studies, meta analysis, surveys, descriptive approaches, ex post facto studies) occur infrequently; long-term studies are missing. About a quarter of the experiments are replications. 2. Professionals are used somewhat more frequently than students as subjects. 3. The dominant topics studied are measurement/metrics and tools/methods/frameworks. Metrics research is dominated by correlational and case studies without any experiments. 4. Important topics are underrepresented or absent, for example: programming languages, model driven developpment, formal methods, and others. The narrow focus on a few empirically researched topics is in contrast to the broad scope of software research.
[1]
Venkataraman Ramesh,et al.
Research in software engineering: an analysis of the literature
,
2002,
Inf. Softw. Technol..
[2]
Paul Lukowicz,et al.
Experimental evaluation in computer science: A quantitative study
,
1995,
J. Syst. Softw..
[3]
Helen Sharp,et al.
The type of evidence produced by empirical software engineers
,
2005,
ACM SIGSOFT Softw. Eng. Notes.
[4]
Marvin V. Zelkowitz,et al.
Experimental validation in software engineering
,
1997,
Inf. Softw. Technol..
[5]
J. Segal,et al.
The type of evidence produced by empirical software engineers
,
2005
.
[6]
J. Saver,et al.
A Quantitative Study
,
2005
.
[7]
Amela Karahasanovic,et al.
A survey of controlled experiments in software engineering
,
2005,
IEEE Transactions on Software Engineering.