Evidence-based Nomenclature and Taxonomy of Research Impact Indicators

Abstract Purpose This study aims to classify research impact indicators based on their characteristics and scope. A concept of evidence-based nomenclature of research impact (RI) indicator has been introduced for generalization and transformation of scope. Design/methodology/approch Literature was collected related to the research impact assessment. It was categorized in conceptual and applied case studies. One hundred and nineteen indicators were selected to prepare classification and nomenclature. The nomenclature was developed based on the principle—“every indicator is a contextual-function to explain the impact”. Every indicator was disintegrated into three parts, i.e. Function, Domain, and Target Areas. Findings The main functions of research impact indicators express improvement (63%), recognition (23%), and creation/development (14%). The focus of research impact indicators in literature is more towards the academic domain (59%) whereas the environment/sustainability domain is least considered (4%). As a result, research impact related to the research aspects is felt the most (29%). Other target areas include system and services, methods and procedures, networking, planning, policy development, economic aspects and commercialisation, etc. Research limitations This research applied to 119 research impact indicators. However, the inclusion of additional indicators may change the result. Practical implications The plausible effect of nomenclature is a better organization of indicators with appropriate tags of functions, domains, and target areas. This approach also provides a framework of indicator generalization and transformation. Therefore, similar indicators can be applied in other fields and target areas with modifications. Originality/value The development of nomenclature for research impact indicators is a novel approach in scientometrics. It is developed on the same line as presented in other scientific disciplines, where fundamental objects need to classify on common standards such as biology and chemistry.

[1]  John Lavis,et al.  Measuring the impact of health research , 2003, Journal of health services research & policy.

[2]  Inmaculada de Melo-Martín,et al.  Clinical and Translational Science Awards: Can They Increase the Efficiency and Speed of Clinical and Translational Research? , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[3]  Jill A. Engel-Cox,et al.  Conceptual Model of Comprehensive Research Metrics for Improved Human Health and Environment , 2008, Environmental health perspectives.

[4]  Francisco Herrera,et al.  A methodology for Institution-Field ranking based on a bidimensional analysis: the IFQ2A index , 2011, Scientometrics.

[5]  Peter Vinkler,et al.  Indicators are the essence of scientometrics and bibliometrics , 2010, Scientometrics.

[6]  Susan G. Fisher,et al.  A Metric‐Based System for Evaluating the Productivity of Preclinical Faculty at an Academic Medical Center in the Era of Clinical and Translational Science , 2015, Clinical and translational science.

[7]  Olle Persson,et al.  The DCI index: Discounted cumulated impact-based research evaluation , 2008 .

[8]  Sebastian K. Boell,et al.  Journal Impact Factors for evaluating scientific performance: use of h-like indicators , 2010, Scientometrics.

[9]  A. Weiss,et al.  Measuring the impact of medical research: moving from outputs to outcomes. , 2007, The American journal of psychiatry.

[10]  Kristi L Holmes,et al.  Beyond citation analysis: a model for assessment of research impact. , 2010, Journal of the Medical Library Association : JMLA.

[11]  Yang Liu,et al.  AHP Based Quantitative Evaluation Index System of Teacher's Research Performance in the University , 2016, MUE 2016.

[12]  George Maliha.  Obsolete to Useful to Obsolete Once Again: A History of Section 507 of the Food, Drug, and Cosmetic Act , 2018 .

[13]  Bin Zhu,et al.  The Hl-index: improvement of H-index based on quality of citing papers , 2013, Scientometrics.

[14]  Stephen E. Marcus,et al.  The Evaluation of Large Research Initiatives , 2008 .

[15]  Steven Wooding,et al.  Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's 'Arthritis Research Campaign' , 2004, Health research policy and systems.

[16]  N. Mays,et al.  Describing the impact of health research: a Research Impact Framework , 2006, BMC Health Services Research.

[17]  Yuh-Shan Ho,et al.  Top cited articles in adsorption research using Y-index , 2014 .

[18]  Peter Vinkler,et al.  The use of the Percentage Rank Position index for comparative evaluation of journals , 2014, J. Informetrics.

[19]  S. Woolf.  The meaning of translational research and why it matters. , 2008, JAMA.

[20]  Peter Vinkler,et al.  The Evaluation of Research by Scientometric Indicators , 2010 .

[21]  R. Rousseau,et al.  The R- and AR-indices: Complementing the h-index , 2007 .

[22]  Thed N. van Leeuwen,et al.  New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications , 1995, Scientometrics.

[23]  Elisa Eiseman,et al.  Demonstrating and Communicating Research Impact: Preparing NIOSH Programs for External Review , 2009 .

[24]  J. Pober,et al.  Obstacles facing translational research in academic medical centers , 2001, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[25]  Tommy Dreyfus,et al.  IMAGES AND DEFINITIONS FOR THE CONCEPT OF FUNCTION , 1989 .

[26]  E GARFIELD,et al.  Citation indexes for science; a new dimension in documentation through association of ideas. , 2006, Science.

[27]  D. Rutledge,et al.  Outcomes of adoption: measuring evidence uptake by individuals and organizations. , 2004, Worldviews on evidence-based nursing.

[28]  Julia Lane,et al.  Star Metrics and the Science of Science Policy , 2012 .

[29]  Lutz Bornmann,et al.  Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative , 2012, Scientometrics.

[30]  J. Westfall,et al.  Practice-based research--"Blue Highways" on the NIH roadmap. , 2007, JAMA.

[31]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[32]  Ritu Sadana,et al.  Knowledge for better health: a conceptual framework and foundation for health research systems. , 2003, Bulletin of the World Health Organization.

[33]  T. Campbell,et al.  A framework to measure the impact of investments in health research , 2006 .

[34]  James Raftery,et al.  Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme. , 2016, Health technology assessment.

[35]  E. Zerhouni.  Translational Research: Moving Discovery to Practice , 2007, Clinical pharmacology and therapeutics.

[36]  Yannis Manolopoulos,et al.  Generalized Hirsch h-index for disclosing latent facts in citation networks , 2007, Scientometrics.

[37]  klaguia.  Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research , 2009 .

[38]  D. Nathan,et al.  Careers in translational clinical research-historical perspectives, future challenges. , 2002, JAMA.

[39]  Cynthia Fraser,et al.  Alpha-2 agonists for sedation of mechanically ventilated adults in intensive care units: a systematic review. , 2016, Health technology assessment.

[40]  Cathleen Kane,et al.  Evaluating Translational Research: A Process Marker Model , 2011, Clinical and translational science.

[41]  Christian Pieter Hoffmann,et al.  A relational altmetric? Network centrality on ResearchGate as an indicator of scientific impact , 2016, J. Assoc. Inf. Sci. Technol..

[42]  Steve Hanney,et al.  How Can Payback from Health Services Research Be Assessed? , 1996, Journal of health services research & policy.

[43]  Stephen B. Johnson,et al.  Central challenges facing the national clinical research enterprise. , 2003, JAMA.

[44]  P. Conway,et al.  The "3T's" road map to transform US health care: the "how" of high-quality care. , 2008, JAMA.

[45]  Olle Persson,et al.  The DCI index: Discounted cumulated impact-based research evaluation , 2008, J. Assoc. Inf. Sci. Technol..

[46]  N R Aries,et al.  The economic impact of biomedical research: a case study of voluntary institutions in the New York metropolitan region. , 1998, Journal of health politics, policy and law.

[47]  Grant Lewison,et al.  Beyond outputs: new measures of biomedical research impact , 2003, Aslib Proc..

[48]  Vincent Larivière,et al.  History of the journal impact factor: Contingencies and consequences , 2009, Scientometrics.

[49]  Ann E. Williams.  Altmetrics: an overview and evaluation , 2017, Online Inf. Rev..

[50]  Linda Butler,et al.  Capturing Research Impacts: A Review of International Practice. Documented Briefing. , 2010 .

[51]  Ellen McIntyre,et al.  The feasibility of determining the impact of primary health care research projects using the Payback Framework , 2009, Health research policy and systems.

[52]  R. Stout,et al.  Validation of a problem-focused nomenclature. , 1983, Archives of general psychiatry.

[53]  Russell E Glasgow,et al.  A proposal to speed translation of healthcare research into practice: dramatic change is needed. , 2011, American journal of preventive medicine.

[54]  Robert Cottrell,et al.  Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study , 2000, BMJ : British Medical Journal.

[55]  B. Haynes,et al.  Barriers and bridges to evidence based clinical practice , 1998, BMJ.

[56]  R. Scoble,et al.  Assessment, evaluations, and definitions of research impact: A review , 2014 .

[57]  Ingeborg Meijer,et al.  Societal output and use of research performed by health research groups , 2010, Health research policy and systems.

[58]  L. Egghe,et al.  Theory and practise of the g-index , 2006, Scientometrics.

[59]  Soldano Ferrone,et al.  Lost in Translation: Obstacles to Translational Medicine , 2004, Journal of Translational Medicine.

[60]  J Raftery,et al.  An assessment of the impact of the NHS Health Technology Assessment Programme. , 2007, Health technology assessment.

[61]  J. Michael Lindsay,et al.  PlumX from Plum Analytics: Not Just Altmetrics , 2016 .

[62]  J. Engel-Cox,et al.  Conceptual model of comprehensive research metrics for improved human health and environment. , 2009, Ciencia & saude coletiva.