Types of research output profiles: A multilevel latent class analysis of the Austrian Science Fund’s final project report data

Starting out from a broad concept of research output, this article looks at the question as to what research outputs can typically be expected from certain disciplines. Based on a secondary analysis of data from final project reports (ex post research evaluation) at the Austrian Science Fund (FWF), Austria's central funding organization for basic research, the goals are (1) to find, across all scientific disciplines, types of funded research projects with similar research output profiles; and (2) to classify the scientific disciplines in homogeneous segments bottom-up according to the frequency distribution of these research output profiles. The data comprised 1,742 completed, FWF-funded research projects across 22 scientific disciplines. The multilevel latent class (LC) analysis produced four LCs or types of research output profiles: 'Not Book', 'Book and Non-Reviewed Journal Article', 'Multiple Outputs', and 'Journal Article, Conference Contribution, and Career Development'. The class membership can be predicted by three covariates: project duration, requested grant sum, and project head's age. In addition, five segments of disciplines can be distinguished: 'Life Sciences and Medicine', 'Social Sciences/Arts and Humanities', 'Formal Sciences', 'Technical Sciences', and 'Physical Sciences'. In 'Social Sciences/Arts and Humanities' almost all projects are of the type 'Book and Non-Reviewed Journal Article', but, vice versa, not all projects of the 'Book and Non-reviewed Journal Article' type are in the 'Social Sciences/Arts and Humanities' segment. The research projects differ not only qualitatively in their output profile; they also differ quantitatively, so that projects can be ranked according to amount of output. Copyright The Author 2012. Published by Oxford University Press., Oxford University Press.

[1]  Rüdiger Mutz,et al.  University and student segmentation: multilevel latent-class analysis of students' attitudes towards research methods and statistics. , 2013, The British journal of educational psychology.

[2]  V. Neuhaus,et al.  Latent Class Analysis , 2010 .

[3]  Lutz Bornmann,et al.  Heterogeneity of Inter-Rater Reliabilities of Grant Peer Reviews and Its Determinants: A General Estimating Equations Approach , 2012, PloS one.

[4]  Lutz Bornmann,et al.  Measuring the societal impact of research , 2012, EMBO reports.

[5]  Alessandro Liberati,et al.  Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews , 2011, Health research policy and systems.

[6]  L. Butler,et al.  Evaluating University Research Performance Using Metrics , 2011 .

[7]  Lutz Bornmann,et al.  Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization , 2011, J. Informetrics.

[8]  C. Fischer,et al.  Factors Influencing Approval Probability in Austrian Science Fund (FWF) Decision-Making Procedures - FWF Stand-Alone Projects Programme, 1999 to 2008 , 2010 .

[9]  Jeroen K. Vermunt,et al.  Latent Class Modeling with Covariates: Two Improved Three-Step Approaches , 2010, Political Analysis.

[10]  Bengt Muthén,et al.  Multilevel Latent Class Analysis: An Application of Adolescent Smoking Typologies With Individual and Contextual Predictors , 2010, Structural equation modeling : a multidisciplinary journal.

[11]  Loet Leydesdorff,et al.  Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance , 2010, J. Informetrics.

[12]  Rüdiger Mutz,et al.  A Nomothetic Version of the Brunswikian Lens Model: A Variable- and Person-Oriented Approach , 2010 .

[13]  E. Hazelkorn.  Assessing Europe's University-Based Research , 2010 .

[14]  Kristi L Holmes,et al.  Beyond citation analysis: a model for assessment of research impact. , 2010, Journal of the Medical Library Association : JMLA.

[15]  Berthold Lausen,et al.  Advances in Data Analysis, Data Handling and Business Intelligence - Proceedings of the 32nd Annual Conference of the Gesellschaft für Klassifikation e.V., Joint Conference with the British Classification Society (BCS) and the Dutch/Flemish Classification Society (VOC), Helmut-Schmidt-University, Ha , 2010, GfKl.

[16]  A. J. M. Linmans,et al.  Why with bibliometrics the Humanities does not need to be the weakest link , 2010, Scientometrics.

[17]  Mike Thelwall,et al.  Google book search: Citation analysis for social science and the humanities , 2009, J. Assoc. Inf. Sci. Technol..

[18]  Fletcher T. H. Cole,et al.  Libcitations: A measure for comparative assessment of book publications in the humanities and social sciences , 2009, J. Assoc. Inf. Sci. Technol..

[19]  L. Butler,et al.  Metrics or Peer Review? Evaluating the 2001 UK Research Assessment Exercise in Political Science , 2009 .

[20]  Yu-Wei Chang,et al.  Characteristics of research output in social sciences and humanities: From a research evaluation perspective , 2008, J. Assoc. Inf. Sci. Technol..

[21]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research , 2008 .

[22]  Linda Butler,et al.  ICT assessment: Moving beyond journal outputs , 2008, Scientometrics.

[23]  Jeroen K. Vermunt,et al.  Determining the Number of Components in Mixture Models for Hierarchical Data , 2008, GfKl.

[24]  F. Veloso,et al.  The determinants of research output and impact: A study of Mexican researchers , 2007 .

[25]  R. Mutz,et al.  Entwicklung eines Hochschul-Rankingverfahrens mittels Mixed-Rasch-Modell und Mehrebenenanalyse am Beispiel der Psychologie , 2007 .

[26]  Vincent Larivière,et al.  Benchmarking scientific output in the social sciences and humanities: The limits of existing databases , 2006, Scientometrics.

[27]  Heavy Alcohol use in the “Fighting Back” Survey Sample: Separating Individual and Community Level Influences using Multilevel Latent Class Analysis , 2006 .

[28]  Thed N. van Leeuwen,et al.  The application of bibliometric analyses in the evaluation of social science research. Who benefits from it, and why it is still feasible , 2006, Scientometrics.

[29]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[30]  Michael Dinges.  THE AUSTRIAN SCIENCE FUND: EX POST EVALUATION AND PERFORMANCE OF FWF FUNDED RESEARCH PROJECTS , 2005 .

[31]  Charles Oppenheim,et al.  The Evolution of the UK’s Research Assessment Exercise: Publications, Performance and Perceptions , 2005 .

[32]  Paul Bourke,et al.  Publication types, citation rates and evaluation , 1996, Scientometrics.

[33]  Rolf A. Zwaan,et al.  Assessing the usefulness of bibliometric indicators for the humanities and the social and beha vioural sciences: A comparative study , 1989, Scientometrics.

[34]  Wolfgang Glänzel,et al.  A bibliometric approach to social sciences. National research performances in 6 selected social science areas, 1990–1992 , 2005, Scientometrics.

[35]  Diana Hicks,et al.  The Four Literatures of Social Sciences , 2005 .

[36]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research , 2005 .

[37]  Kathryn La Barre,et al.  Mickey Mouse and Milton: book publishing in the humanities , 2004, Learn. Publ..

[38]  Marcel Croon,et al.  Estimating Latent Structure Models with Categorical Variables: One-Step Versus Three-Step Estimators , 2004, Political Analysis.

[39]  J. Vermunt.  Latent Class Models , 2004 .

[40]  David Kaplan,et al.  The Sage handbook of quantitative methodology for the social sciences , 2004 .

[41]  D. Hicks.  The Four Literatures of Social Science , 2004 .

[42]  Tammo H. A. Bijmolt,et al.  Country and Consumer Segmentation: Multi-Level Latent Class Analysis of Financial Product Ownership , 2004 .

[43]  E. Govere,et al.  Educational legislation in colonial Zimbabwe (1899-1979) , 2003 .

[44]  Grant Lewison,et al.  Beyond outputs: new measures of biomedical research impact , 2003, Aslib Proc..

[45]  R. Mutz,et al.  Forstliches Engagement und forstliches Engagementpotenzial von Privatwaldbesitzern in Nordrhein-Westfalen − Analyse auf der Basis des Mixed-Rasch-Modells , 2002, Forstwissenschaftliches Centralblatt vereinigt mit Tharandter forstliches Jahrbuch.

[46]  Lisa A. Best,et al.  Scientific Graphs and the Hierarchy of the Sciences: , 2000 .

[47]  S. Hanney,et al.  Evaluating the NHS research and development programme: will the programme give value for money? , 1998, Journal of the Royal Society of Medicine.

[48]  H. Bozdogan.  Choosing the Number of Component Clusters in the Mixture-Model Using a New Informational Complexity Criterion of the Inverse-Fisher Information Matrix , 1993 .

[49]  Otto Opitz,et al.  Information and Classification , 1993 .

[50]  A. Nederhof.  Methods of coping with social desirability bias: A review. , 1985 .