The Evaluation of Large Research Initiatives

Over the past few decades there has been a rise in the number of federally funded large scientific research initiatives, with increased calls to evaluate their processes and outcomes. This article describes efforts to evaluate such initiatives in one agency within the U.S. federal government. The authors introduce the Evaluation of Large Initiatives (ELI) project, a preliminary effort to explore how to accomplish such evaluation. They describe a pilot effort of this project to evaluate the Transdisciplinary Tobacco Use Research Center (TTURC) initiative of the National Cancer Institute. They present a summary of this pilot evaluation including the methods used (concept mapping, logic modeling, a detailed researcher survey, content analysis and systematic peer-evaluation of progress reports, bibliometric analysis and peer evaluation of publications and citations, and financial expenditures analysis) and a brief overview of results. Finally, they discuss several important lessons and recommendations that emerged from this work.

[1]  Pamela I. Clark,et al.  Greater Than the Sum: Systems Thinking in Tobacco control , 2007 .

[2]  Mary Kane,et al.  Concept Mapping for Planning and Evaluation , 2006 .

[3]  H. Skoie.  Bibliometrics—Some warnings from the north , 2006, Scientometrics.

[4]  J. Anetor,et al.  Large-Scale Biomedical Science: Exploring Strategies for Future Research , 2005 .

[5]  Concepción S. Wilson,et al.  The Literature of Bibliometrics, Scientometrics, and Informetrics , 2001, Scientometrics.

[6]  F. Manning,et al.  NIH Extramural Center Programs , 2004 .

[7]  Wolfgang Glänzel,et al.  Two decades of "Scientometrics". An interdisciplinary field represented by its leading journal , 2004, Scientometrics.

[8]  M. Clark,et al.  Evaluating transdisciplinary science. , 2003, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[9]  Nicolas Rasmussen,et al.  Of `Small Men', Big Science and Bigger Business: The Second World War and Biomedical Research in the United States , 2002 .

[10]  Donald W. Compton,et al.  The art, craft, and science of evaluation capacity building , 2002 .

[11]  Richard Smith,et al.  Measuring the social impact of research , 2001, BMJ : British Medical Journal.

[12]  Hervé Maisonneuve,et al.  Peer Review in Health Sciences , 2000, BMJ : British Medical Journal.

[13]  Elizabeth Whitmore,et al.  Understanding and Practicing Participatory Evaluation. , 1998 .

[14]  Peter Hernon,et al.  The government performance and results act , 1998, Gov. Inf. Q..

[15]  P. Seglen,et al.  Education and debate , 1999, The Ethics of Public Health.

[16]  Valerie J. Caracelli,et al.  Advances in mixed-method evaluation : the challenges and benefits of integrating diverse paradigms , 1997 .

[17]  Mary Jo Nye.  Before Big Science: The Pursuit Of Modern Chemistry And Physics, 1800-1940 , 1996 .

[18]  E. Funkhouser.  The Evaluative Use of Citation Analysis for Communication Journals , 1996 .

[19]  Farideh Osareh.  Bibliometrics, Citation Anatysis and Co-Citation Analysis: A Review of Literature II , 1996 .

[20]  FARIDEH OSAREH,et al.  Bibliometrics, Citation Analysis and Co-Citation Analysis: A Review of Literature I , 1996, Libri.

[21]  P. Sopp.  Cluster analysis. , 1996, Veterinary immunology and immunopathology.

[22]  Ronald N. Kostoff,et al.  Research requirements for research impact assessment , 1995 .

[23]  Neal F. Lane.  What is the future of research? The science policy of the USA , 1995 .

[24]  R P Lederman,et al.  Systematic Data Collection: I , 1994, MCN. The American journal of maternal child nursing.

[25]  Ronald N. Kostoff,et al.  Quantitative/qualitative federal research impact evaluation practices , 1994 .

[26]  P. Rosenfield,et al.  The potential of transdisciplinary research for sustaining and extending linkages between the health and social sciences. , 1992, Social science & medicine.

[27]  Valerie J. Caracelli,et al.  Toward a Conceptual Framework for Mixed-Method Evaluation Designs , 1989 .

[28]  R. Weber.  Basic Content Analysis , 1986 .

[29]  William M. K. Trochim,et al.  Conceptualization for Planning and Evaluation , 2022 .

[30]  William M. K. Trochim,et al.  Pattern Matching, Validity, and Conceptualization in Program Evaluation , 1985 .

[31]  William M. K. Trochim,et al.  Research Design for Program Evaluation: The Regression-Discontinuity Approach , 1984 .

[32]  B. Everitt,et al.  Cluster Analysis (2nd ed). , 1982 .

[33]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[34]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[35]  Moonja P. Kim,et al.  The Method of Sorting as a Data-Gathering Procedure in Multivariate Research. , 1975, Multivariate behavioral research.

[36]  Michael R. Anderberg,et al.  Cluster Analysis for Applications , 1973 .

[37]  E GARFIELD,et al.  Citation indexes for science; a new dimension in documentation through association of ideas. , 2006, Science.