Institutional repositories and research assessment

This study concerns the potential role of institutional repositories in supporting research assessment in universities with specific reference to the Research Assessment Exercises in the UK. After a brief look at research evaluation methods, it introduces the UK Research Assessment Exercise (RAE), focusing on its role in determining the distribution of research funding, the assessment process itself, and some concerns that have been raised by participants and observers. The study will then introduce institutional repositories and consider the ways in which they might be used to enhance the research assessment process in the UK. It will first consider the role of repositories in providing institutional support for the submission and review process. Secondly, the paper will consider the ways in which citation linking between papers in repositories might be used as the basis for generating quantitative data on research impact that could be used for assessment. Thirdly, this study will consider other ways in which repositories might be able to provide quantitative data, e.g. usage statistics or Webometric link data, which may be able to be used - together with other indicators - to support the evaluation of research.

[1]  P. Seglen,et al.  Education and debate , 1999, The Ethics of Public Health.

[2]  Mike Thelwall,et al.  Methods for reporting on the targets of links from national systems of university Web sites , 2004, Inf. Process. Manag..

[3]  Charles Oppenheim,et al.  The Correlation between citation counts and the 1992 Research Assessment Exercise Ratings for British Library and Information Science University departments , 1995, J. Documentation.

[4]  M. Strathern.  ‘Improving ratings’: audit in the British University system , 1997, European Review.

[5]  J. Johnes,et al.  The Research Performance of UK Universities: a Statistical Analysis of the Results of the 1989 Research Selectivity Exercise , 1993 .

[6]  Julian Warner.  Citation analysis and research assessment in the United Kingdom , 2005 .

[7]  Thomas V Perneger,et al.  Competing interests: None declared. Ethical approval: Ethics committee of Côte d’Ivoire’s Ministry of Public Health and the Institutional Review Board of the US Centers for Disease Control and Prevention , 2004 .

[8]  Lewis Elton,et al.  Goodhart's Law and Performance Indicators in Higher Education , 2004 .

[9]  S. Lawrence.  Free online availability substantially increases a paper's impact , 2001, Nature.

[10]  Charles Oppenheim,et al.  Journals, scholarly communication and the RAE: A case study of the business and management sector , 2001 .

[11]  Stevan Harnad,et al.  Earlier Web Usage Statistics as Predictors of Later Citation Impact , 2005, J. Assoc. Inf. Sci. Technol..

[12]  T. Jefferson,et al.  Effects of editorial peer review: a systematic review. , 2002, JAMA.

[13]  Henk F. Moed,et al.  Journal impact measures in bibliometric research , 2004, Scientometrics.

[14]  C. Lee Giles,et al.  Digital Libraries and Autonomous Citation Indexing , 1999, Computer.

[15]  Hugh Willmott,et al.  Commercialising Higher Education in the UK: The state, industry and peer review , 2003 .

[16]  Mike Thelwall,et al.  The relationship between the WIFs or inlinks of Computer Science Departments in UK and their RAE ratings or research productivities in 2001 , 2003, Scientometrics.

[17]  Stephen S. Murray,et al.  The NASA Astrophysics Data System: Obsolescence of Reads and Cites , 2003 .

[18]  J. Johnes,et al.  The 1989 Research Selectivity Exercise: A Statistical Analysis of Differences in Research Rating between Universities at the Cost Centre Level , 1992 .

[19]  Thed N. van Leeuwen,et al.  Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance , 2001, Scientometrics.

[20]  Stevan Harnad,et al.  The self-archiving initiative , 2001, Nature.

[21]  Tom Schuller,et al.  The Future of Higher Education. , 1991 .

[22]  Clifford A. Lynch,et al.  Institutional Repositories: Essential Infrastructure For Scholarship In The Digital Age , 2003 .

[23]  Jack Harris,et al.  The UK research assessment exercise , 1997 .

[24]  Charles Jennings,et al.  Citation Data: The Wrong Impact? * * Reprinted with permission from Nature Neuroscience, 1, December, 1998, 641-643. , 2001, Cortex.

[25]  Stevan Harnad.  Why I think research access, impact and assessment are linked , 2001 .

[26]  Jean King.  A review of bibliometric and other science indicators and their role in research evaluation , 1987, J. Inf. Sci..

[27]  Mike Thelwall,et al.  The connection between the research of a university and counts of links to its web pages: An investigation based upon a classification of the relationships of pages to the research of the host university , 2003, J. Assoc. Inf. Sci. Technol..

[28]  T. Jefferson,et al.  Measuring the quality of editorial peer review. , 2002, JAMA.

[29]  Leo Egghe,et al.  New informetric aspects of the Internet: some reflections - many problems , 2000, J. Inf. Sci..

[30]  Jonathan Furner,et al.  Scholarly communication and bibliometrics , 2005, Annu. Rev. Inf. Sci. Technol..

[31]  David Nicholas,et al.  Cracking the code: web log analysis , 1999, Online Inf. Rev..

[32]  Judy Luther.  White Paper on Electronic Journal Usage Statistics , 2002 .

[33]  Charles Oppenheim,et al.  The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology , 1997, J. Documentation.

[34]  Judith Elkin.  The UK Research Assessment Exercise 2001 , 2002 .

[35]  Jim Taylor.  A statistical analysis of the 1992 research assessment exercise , 1995 .

[36]  Jon M. Kleinberg,et al.  Mining the Web's Link Structure , 1999, Computer.

[37]  R. Merton.  The Matthew Effect in Science , 1968, Science.

[38]  Paul J. Bracke,et al.  Web usage mining at an academic health sciences library: an exploratory study. , 2004, Journal of the Medical Library Association : JMLA.

[39]  Mike Thelwall,et al.  Three target document range metrics for university web sites , 2003, J. Assoc. Inf. Sci. Technol..

[40]  Charles Oppenheim,et al.  Use of citation analysis to predict the outcome of the 2001 Research Assessment Exercise for Unit of Assessment (UoA) 61: Library and Information Management, , 2001, Inf. Res..

[41]  Myra Spiliopoulou,et al.  Web usage mining for Web site evaluation , 2000, CACM.

[42]  M. Eysenck,et al.  The correlation between RAE ratings and citation counts in psychology Technical Report , 2002 .

[43]  Herbert Van de Sompel,et al.  Open Linking in the Scholarly Information Environment Using the OpenURL Framework , 2001, D Lib Mag..

[44]  Alastair Smith,et al.  A Tale of Two Web Spaces: Comparing Sites Using Web Impact Factors. , 1999 .

[45]  Steve Hitchcock,et al.  How Dynamic E-journals can Interconnect Open Access Archives , 2001, ELPUB.

[46]  D. King.  The scientific impact of nations , 2004, Nature.

[47]  E. Garfield.  Fortnightly Review: How can impact factors be improved? , 1996 .

[48]  John R. Doyle,et al.  Judging the quality of research in business schools: The UK as a case study , 1995 .

[49]  R. May.  The Scientific Wealth of Nations , 1997, Science.

[50]  Anthony F. J. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005, Scientometrics.

[51]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[52]  Myra Spiliopoulou,et al.  Data Mining for Measuring and Improving the Success of Web Sites , 2004, Data Mining and Knowledge Discovery.

[53]  Mike Thelwall,et al.  Conceptualizing documentation on the Web: An evaluation of different heuristic-based models for counting links between university Web sites , 2002, J. Assoc. Inf. Sci. Technol..

[54]  E. Garfield.  Journal impact factor: a brief review. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[55]  Mike Thelwall,et al.  Extracting macroscopic information from Web links , 2001, J. Assoc. Inf. Sci. Technol..

[56]  P. Seglen,et al.  Citation rates and journal impact factors are not suitable for evaluation of research. , 1998, Acta orthopaedica Scandinavica.

[57]  R. May.  The Scientific Investments of Nations , 1998, Science.

[58]  Amy Friedlander,et al.  D-Lib Magazine: Publishing as the Honest Broker , 1998 .

[59]  Per O. Seglen,et al.  The Skewness of Science , 1992, J. Am. Soc. Inf. Sci..

[60]  Herbert Van de Sompel,et al.  Reference Linking in a Hybrid Library Environment, Part 1: Frameworks for Linking , 1999, D Lib Mag..

[61]  Julian Warner.  The public reception of the Research Assessment Exercise 1996 , 1998, Inf. Res..

[62]  E Friedman,et al.  Loose ends. , 2020, The Healthcare Forum journal.

[63]  Jon Kleinberg,et al.  The Structure of the Web , 2001, Science.

[64]  Charles Oppenheim,et al.  Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE , 2003, J. Documentation.

[65]  S. McKay.  Quantifying Quality: Can Quantitative Data (“Metrics”) Explain the 2001 RAE Ratings for Social Policy and Administration? , 2003 .

[66]  Les Carr,et al.  Mandated online RAE CVs Linked to University Eprint Archives , 2003 .

[67]  Jeff Butler,et al.  The Implicit use of Business Concepts in the UK Research Assessment Exercise , 2004 .

[68]  Judit Bar-Ilan,et al.  Data collection methods on the Web for infometric purposes — A review and analysis , 2004, Scientometrics.

[69]  S. Strogatz.  Exploring complex networks , 2001, Nature.

[70]  Les Carr,et al.  Open Citation Linking: The Way Forward , 2002, D Lib Mag..

[71]  Guenther Eichhorn,et al.  The NASA Astrophysics Data System : Sociology , Bibliometrics , and Impact , 2022 .

[72]  Les Carr,et al.  Developing services for open eprint archives: globalisation, integration and the impact of links , 2000, DL '00.

[73]  Jill Johnes,et al.  Performance assessment in higher education in Britain , 1996 .

[74]  Raym Crow,et al.  The case for institutional repositories : a SPARC position paper , 2002 .

[75]  Charles Oppenheim.  Do Citations Count? Citation Indexing and the Research Assessment Exercise (RAE) , 1996 .

[76]  John Gennard,et al.  The state of the field in UK management research: reflections of the research assessment exercise (rae) panel , 2003 .

[77]  Henk Voogd,et al.  Reflections on the 2001 United Kingdom Research Assessment Exercise for Town and Country Planning , 2002 .

[78]  Laurie McAulay,et al.  The judge, the model of the judge, and the model of the judged as judge: Analyses of the UK 1992 research assessment exercise data for business and management studies , 1996 .

[79]  Myra Spiliopoulou,et al.  Analysis of navigation behaviour in web sites integrating multiple information systems , 2000, The VLDB Journal.

[80]  Simon Wessely,et al.  Peer review of grant applications: what do we know? , 1998, The Lancet.

[81]  Jonathan Adams.  Research Assessment in the UK , 2002, Science.

[82]  H. Kokko,et al.  What do impact factors tell us? , 1999, Trends in ecology & evolution.

[83]  U. Segerstråle,et al.  Real Science. What it is, and what it means , 2001 .

[84]  S. Tomlinson,et al.  The research assessment exercise and medical research , 2000, BMJ : British Medical Journal.

[85]  Robert D. Cameron,et al.  A Universal Citation Database as a Catalyst for Reform in Scholarly Communication , 1997, First Monday.

[86]  Mike Thelwall,et al.  Do the Web sites of higher rated scholars have significantly more online impact? , 2004, J. Assoc. Inf. Sci. Technol..

[87]  David Adam,et al.  Citation analysis: The counting house , 2002, Nature.

[88]  Karl-Franz Kaltenborn,et al.  Der Impact-Faktor als Parameter zur Evaluation von Forscherinnen/Forschern und Forschung , 2003, Medizinische Klinik.

[89]  D. Horrobin,et al.  Peer review of grant applications: a harbinger for mediocrity in clinical research? , 1996, The Lancet.

[90]  Owen Thomas,et al.  Webometric analysis of departments of librarianship and information science , 2000, J. Inf. Sci..

[91]  Jonathan Adams.  Benchmarking international research , 1998, Nature.

[92]  Mike Thelwall,et al.  Motivations for academic web site interlinking: evidence for the Web as a novel source of information on informal scholarly communication , 2003, J. Inf. Sci..

[93]  Michael D. Gordon,et al.  Web Search---Your Way , 2001, CACM.

[94]  Sean Howard,et al.  The Research Assessment Exercise , 1998 .

[95]  David W.H. Walton,et al.  Scientific Publications – Free For All? , 2004, Antarctic Science.

[96]  E. Garfield.  From Citation Indexes to Informetrics: Is the Tail Now Wagging the Dog ? , 1998 .

[97]  B. Martin,et al.  University Research Evaluation and Funding: An International Comparison , 2003 .