A measurement system for science and engineering research center performance evaluation

This research is focused on gaining deeper insights into US National Science Foundation (NSF) science and engineering research center challenges and motivated to develop a method that effectively measures the performance of these organizations. While research has addressed organizational performance at the micro, or single-actor level for universities or companies and at the regional or national macro level, the middle level where the NSF centers reside is largely missing. The bulk of the cooperative research center studies use either case-based methods or bibliometric data to measure traditional research outputs. Many are excellent studies; however, they only focus on a piece of the performance measurement problem. There is a need for more research to understand how to measure performance and compare performance of cooperative research centers formed in a triple-helix type partnership involving government, industry and academia. This research begins to fill these gaps by examining outputs from a balanced perspective and introducing a hierarchical decision model that uses both quantitative and qualitative metrics for a holistic study. The proposed outcome of this research is a performance measurement scoring system that can be used for science and engineering focused research centers. The method is demonstrated using the NSF IUCRC model.

[1]  Hassan Masum,et al.  Five promising methods for health foresight , 2010 .

[2]  Barry Bozeman,et al.  Technology transfer and public policy: a review of research and theory , 2000 .

[3]  Karen Eden,et al.  Selecting information technology for physicians' practices: a cross-sectional study , 2002, BMC Medical Informatics Decis. Mak..

[4]  A. Link,et al.  Assessing the Impact of Organizational Practices on the Productivity of University Technology Transfer Offices : An Exploratory Study # , 1999 .

[5]  Maryann P. Feldman,et al.  The ex ante assessment of knowledge spillovers: Government R&D policy, economic incentives and private firm behavior , 2006 .

[6]  K. Pavitt,et al.  R&D Patenting and Innovative Activities : A Statistical Exploration : Research Policy , 1982 .

[7]  Eric P. Chiang,et al.  Industry-University Cooperative Research Centers , 2000 .

[8]  Heiko A. von der Gracht,et al.  Technological Forecasting & Social Change Assessing Delphi panel composition for strategic foresight — A comparison of panels based on company-internal and external participants , 2014 .

[9]  J. D. Jennings,et al.  Investigating the Surface Energy and Bond Performance of Compression Densified Wood , 1997 .

[10]  Sally Coleman Selden,et al.  No Longer Unmeasurable? A Multidimensional Integrated Model of Nonprofit Organizational Effectiveness , 2004 .

[11]  Yu Zhou Nanoscale surface modification of wood veneers for adhesion , 2008 .

[12]  L. Davis,et al.  Selection and use of content experts for instrument development. , 1997, Research in nursing & health.

[13]  Luke Georghiou,et al.  Evaluating technology programs: tools and methods , 2000 .

[14]  Ulrich Schmoch,et al.  How to use indicators to measure scientific performance: a balanced approach , 2010 .

[15]  Mark Clayton Delphi: a technique to harness expert opinion for critical decision‐making tasks in education , 1997 .

[16]  Benoît Godin,et al.  The obsession for competitiveness and its impact on statistics: the construction of high-technology indicators , 2004 .

[17]  Sharon L. Nunes Extending the innovation ecosystem , 2004, Proceedings. IEEE International Conference on e-Commerce Technology, 2004. CEC 2004..

[18]  Daniel L. Fay,et al.  Research collaboration in universities and academic entrepreneurship: the-state-of-the-art , 2012, The Journal of Technology Transfer.

[19]  Elizabeth A. Corley,et al.  University research centers and the composition of research collaborations , 2008 .

[20]  Charles E. Frazier,et al.  Improvements In The Fracture Cleavage Testing Of Adhesively-Bonded Wood , 2007 .

[21]  Tugrul U. Daim,et al.  Forecasting emerging technologies: Use of bibliometrics and patent analysis , 2006 .

[22]  George Kozmetsky,et al.  Creating the technopolis: High-technology development in Austin, Texas☆ , 1989 .

[23]  Mark Muro,et al.  Chapter 5 The New “Cluster Moment”: How Regional Innovation Clusters can Foster the Next Economy , 2011 .

[24]  Barry Bozeman,et al.  Academic faculty as intellectual property in university-industry research alliances , 2015 .

[25]  J. Tyler,et al.  Redeploying Bayh-Dole: beyond Merely doing good to optimizing the potential in results of taxpayer-funded research , 2011 .

[26]  Richard K. Crump,et al.  Nonparametric Tests for Treatment Effect Heterogeneity , 2006, The Review of Economics and Statistics.

[27]  Beth Marie Coberly Faculty Satisfaction and Orgnizational Commitment with Industry-University Research Centers , 2004 .

[28]  Alan L. Porter,et al.  Science overlay maps: A new tool for research policy and library management , 2009, J. Assoc. Inf. Sci. Technol..

[29]  R. Rohrbeck Harnessing a Network of Experts for Competitive Advantage: Technology Scouting in the ICT Industry , 2010 .

[30]  R. Srinivasan,et al.  A global sensitivity analysis tool for the parameters of multi-variable catchment models , 2006 .

[31]  Philippe Aghion,et al.  Science, Technology and Innovation for Economic Growth: Linking Policy Research and Practice in "STIG Systems" , 2009 .

[32]  Hariolf Grupp,et al.  National Technology Foresight Activities Around the Globe , 1999 .

[33]  C. Freeman,et al.  Developing science, technology and innovation indicators: What we can learn from the past , 2009 .

[34]  Denis O. Gray,et al.  Research Center Sustainability and Survival: Case Studies of Fidelity, Reinvention and Leadership of Industry/University Cooperative Research Centers , 2012 .

[35]  James A. Cunningham,et al.  The inhibiting factors that principal investigators experience in leading publicly funded research , 2014 .

[36]  F. Geels Ontologies, socio-technical transitions (to sustainability), and the multi-level perspective , 2010 .

[37]  Sudip Chowdhury,et al.  Advancing characterization techniques for structure-property determination of in-situ lignocelluloses , 2011 .

[38]  Richard A. Slaughter,et al.  Foresight beyond strategy: Social initiatives by business and government , 1996 .

[39]  Ian Miles,et al.  Foresight tools for participative policy-making in inter-governmental processes in developing countries: Lessons learned from the eLAC Policy Priorities Delphi , 2009 .

[40]  Denis O. Gray,et al.  Managing the industry/university cooperative research center : a guide for directors and other stakeholders , 1998 .

[41]  Ronald N. Kostoff Science and Technology Metrics , 2005 .

[42]  Kim S. Cameron,et al.  Measuring Organizational Effectiveness in Institutions of Higher Education. , 1978 .

[43]  Heiko A. Gracht,et al.  A Delphi-based risk analysis — Identifying and assessing future challenges for supply chain security in a multi-stakeholder environment , 2013 .

[44]  Myrna Flores,et al.  A balanced scorecard for measuring the impact of industry–university collaboration , 2011 .

[45]  Kumi Okuwada,et al.  Identifying expected areas of future innovation by combining foresight outputs , 2013 .

[46]  Elizabeth A. Corley,et al.  Design and the management of multi-institutional research collaborations: Theoretical implications from two case studies , 2006 .

[47]  P. Craig Boardman,et al.  Government centrality to university–industry interactions: University research centers and the industry involvement of academic researchers , 2009 .

[48]  Equity and the Technology Transfer Strategies of American Research Universities , 2002 .

[49]  Ron Adner,et al.  Value Creation in Innovation Ecosystems: How the Structure of Technological Interdependence Affects Firm Performance in New Technology Generations , 2009 .

[50]  Kenny Phan,et al.  Innovation Measurement: A Decision Framework to Determine Innovativeness of a Company , 2013 .

[51]  Chih-Young Hung,et al.  Strategic foresight using a modified Delphi with end-user participation: A case study of the iPad's impact on Taiwan's PC ecosystem , 2013 .

[52]  Anup Kumar Das,et al.  Webliography of STI Indicator Databases and Related Publications , 2012, J. Sci. Res..

[53]  Y. Lee The Sustainability of University-Industry Research Collaboration: An Empirical Assessment , 2000 .

[54]  Thien A. Tran,et al.  Strategic Evaluation of University Knowledge and Technology Transfer Effectiveness , 2013 .

[55]  Maria Rosaria Della Peruta,et al.  Managing the intellectual capital within government-university-industry R&D partnerships : A framework for the engineering research centers , 2014 .

[56]  M. Balconi,et al.  University–industry interactions in applied research: The case of microelectronics , 2006 .

[57]  Eldon Y. Li,et al.  Co-authorship networks and research impact: A social capital perspective , 2013 .

[58]  E. Geisler Measuring the Contributions of Knowledge Management Systems to the Strategic Competitiveness of Organizations: A Review and a Model , 2010 .

[59]  Marco Sabbadin Links and impacts: the influence of public research on industrial R&D , 2016 .

[60]  Pisek Gerdsri,et al.  A systematic approach to developing national technology policy and strategy for emerging technologies: A case study of nanotechnology for Thailand's agriculture industry , 2009, PICMET '09 - 2009 Portland International Conference on Management of Engineering & Technology.

[61]  Luke Georghiou,et al.  Rising to the challenges—Reflections on Future-oriented Technology Analysis , 2013 .

[62]  Albert N. Link,et al.  University-based technology initiatives: Quantitative and qualitative evidence , 2005 .

[63]  Jill Ann Tarzian Sorensen,et al.  Evaluating academic technology transfer performance by how well access to knowledge is facilitated––defining an access metric , 2008 .

[64]  Barry Bozeman,et al.  The NSF Engineering Research Centers and the University–Industry Research Revolution: A Brief History Featuring an Interview with Erich Bloch , 2004 .

[65]  Andy Neely,et al.  How Should Firms Evaluate Success in University-Industry Alliances? A Performance Measurement System , 2010 .

[66]  Everett M. Rogers,et al.  Technology Transfer from University-Based Research Centers: The University of New Mexico Experience , 1999 .

[67]  Martyn Cordey-Hayes,et al.  Foresight as a core competence , 2001 .

[68]  Branco Ponomariov,et al.  Management knowledge and the organization of team science in university research centers , 2014 .

[69]  Jennifer M. Miller Universities, Industry, and Government in Collaboration: A Review of the Literature on Research Centers , 2009 .

[70]  Melissa H. Cragin,et al.  Measuring science: Emerging tools for analysis of federal R&D investments , 2012, ASIST.

[71]  T. Bresnahan,et al.  ‘Old Economy’ Inputs for ‘New Economy’ Outcomes: Cluster Formation in the New Silicon Valleys , 2001 .

[72]  F. Geels From sectoral systems of innovation to socio-technical systems: Insights about dynamics and change from sociology and institutional theory , 2004 .

[73]  R. Scoble,et al.  Assessment, evaluations, and definitions of research impact: A review , 2014 .

[74]  A. Salter,et al.  Academic Engagement and Commercialisation: A Review of the Literature on University-Industry Relations , 2012 .

[75]  Kalyanmoy Deb,et al.  Multiple Criteria Decision Making, Multiattribute Utility Theory: Recent Accomplishments and What Lies Ahead , 2008, Manag. Sci..

[76]  Kevin W. Boyack,et al.  Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature , 2011, J. Informetrics.

[77]  David Roessner,et al.  The economic impact of engineering research centers: preliminary results of a pilot study , 2010 .

[78]  Hariolf Grupp,et al.  The measurement of technical performance of innovations by technometrics and its impact on established technology indicators , 1994 .

[79]  Ronald N. Kostoff,et al.  Systematic acceleration of radical discovery and innovation in science and technology , 2006 .

[80]  Arijit Sinha,et al.  The effect of elevated temperature on mechanical behavior of structural wood and wood-based composites , 2010 .

[81]  Teresa R. Behrens,et al.  Unintended consequences of cooperative research: impact of industry sponsorship on climate for academic freedom and other graduate student outcome , 2001 .

[82]  Denis O. Gray,et al.  The New Science and Engineering Management: Cooperative Research Centers as Intermediary Organizations for Government Policies and Industry Strategies , 2013 .

[83]  Scott Stern,et al.  Clusters, Convergence, and Economic Performance , 2010 .

[84]  Alan L. Porter,et al.  Forecasting Innovation Pathways (FIP) for new and emerging science and technologies , 2013 .

[85]  Tugrul U. Daim,et al.  Exploring technology acquisition in Oregon, Turkey and in the U.S. electronics manufacturing companies , 2008 .

[86]  Jun Zheng,et al.  Studies of PF Resole / Isocyanate Hybrid Adhesives , 2002 .

[87]  Loet Leydesdorff,et al.  A Triple Helix of University—Industry—Government Relations , 1998, Scientometrics.

[88]  Hongyi Chen,et al.  A sensitivity analysis algorithm for hierarchical decision models , 2008, Eur. J. Oper. Res..

[89]  Elias G. Carayannis,et al.  Architecting gloCal (global–local), real-virtual incubator networks (G-RVINs) as catalysts and accelerators of entrepreneurship in transitioning and developing economies: lessons learned and best practices from current development and business incubation practices , 2005 .

[90]  Bernice B Brown,et al.  DELPHI PROCESS: A METHODOLOGY USED FOR THE ELICITATION OF OPINIONS OF EXPERTS , 1968 .

[91]  M. Öner,et al.  An integrated view of foresight: integrated foresight management model , 2003 .

[92]  Rosalie Ruegg,et al.  A toolkit for evaluating public R&D investment models, methods, and findings from ATP's first decade , 2003 .

[93]  J. Rohrbaugh,et al.  A Competing Values Approach to Organizational Effectiveness , 1981 .

[94]  Lindsey Cain McGowen Program Sustainability For Cooperative Research Centers: A Longitudinal Analysis , 2012 .

[95]  J. Hayton,et al.  Why do firms join consortial research centers? An empirical examination of firm, industry and environmental antecedents , 2010 .

[96]  A. Geuna,et al.  The Governance of University Knowledge Transfer: A Critical Review of the Literature , 2009 .

[97]  H. Grupp,et al.  Indicators for national science and technology policy: how robust are composite indicators? , 2004 .

[98]  Albert N. Link The use of literature-based innovation output indicators for research evaluation , 1995 .

[99]  Ismael Rafols,et al.  Is science becoming more interdisciplinary? Measuring and mapping six research fields over time , 2009, Scientometrics.

[100]  B.M. Aucoin,et al.  Engineering Management , 1997, IEEE Potentials.

[101]  Suzanne D. Pawlowski,et al.  The Delphi method as a research tool: an example, design considerations and applications , 2004, Inf. Manag..

[102]  A. Link,et al.  Toward a model of the effective transfer of scientific knowledge from academicians to practitioners: qualitative evidence from the commercialization of university technologies. , 2004 .

[103]  Benoît Godin,et al.  The emergence of S&T indicators: why did governments supplement statistics with indicators? , 2003 .

[104]  Ben R. Martin,et al.  Technology foresight for wiring up the national innovation system: experiences in Britain, Australia and New Zealand , 1999 .

[105]  Eliezer Geisler,et al.  Benchmarking inter-organisational technology cooperation: the link between infrastructure and sustained performance , 2003, Int. J. Technol. Manag..

[106]  V. A. Chavez Creative enterprise as a solution vector for twenty first century problems , 2013 .

[107]  Denis O. Gray,et al.  Cooperative Research Centers and Technical Innovation , 2013 .

[108]  Elizabeth Gibson,et al.  Supporting sustainable technology cluster development: A performance measurement problem , 2015, 2015 Portland International Conference on Management of Engineering and Technology (PICMET).

[109]  Joseph F. Coates The future of foresight—A US perspective , 2010 .

[110]  Susana Elena-Pérez,et al.  Ready for the future? Universities' capabilities to strategically manage their intellectual capital , 2011 .

[111]  A. Salter,et al.  Investigating the factors that diminish the barriers to university–industry collaboration , 2009 .

[112]  Branco Ponomariov,et al.  A preliminary assessment of the potential for “team science” in DOE Energy Innovation Hubs and Energy Frontier Research Centers , 2011 .

[113]  Scott Shane,et al.  The effects of patent litigation on university licensing efforts , 2007 .

[114]  Denis O. Gray,et al.  Evaluating cooperative research centers: a strategy for assessing proximal and distal outcomes and associated economic impacts , 2013 .

[115]  Kon Shing Kenneth Chung,et al.  Egocentric analysis of co-authorship network structure, position and performance , 2012, Inf. Process. Manag..

[116]  Thomas L. Saaty,et al.  DECISION MAKING WITH THE ANALYTIC HIERARCHY PROCESS , 2008 .

[117]  Richard M. Steers Problems in the Measurement of Organizational Effectiveness , 1975 .

[118]  Ludwig Bstieler,et al.  Bridging the cultural divide: Trust formation in university-industry research collaborations in the US, Japan, and South Korea , 2014 .

[119]  Alan L. Porter,et al.  Assessing research network and disciplinary engagement changes induced by an NSF program , 2012 .

[120]  Robert J. W. Tijssen,et al.  Universities and industrially relevant science: Towards measurement models and indicators of entrepreneurial orientation , 2006 .

[121]  Alfredo Vittorio De Massis,et al.  Entrepreneurship research centers around the world: research orientation, knowledge transfer and performance , 2013 .

[122]  Roald A.A. Suurs,et al.  Patterns of expectations for emerging sustainable technologies , 2012 .

[123]  P. Boardman,et al.  Influencing scientists’ collaboration and productivity patterns through new institutions: University research centers and scientific and technical human capital☆ , 2010 .

[124]  Harm-Jan Steenhuis,et al.  Quantifying the benefits of participating in an industry university research center: An examination of research cost avoidance , 2003, Scientometrics.

[125]  Robert Phaal,et al.  Key principles for developing industrially relevant strategic technology management toolkits , 2013 .

[126]  Ismael Rafols,et al.  Global maps of science based on the new Web-of-Science categories , 2012, Scientometrics.

[127]  John Rohrbaugh,et al.  A Spatial Model of Effectiveness Criteria: Towards a Competing Values Approach to Organizational Analysis , 1983 .

[128]  Morten T. Hansen,et al.  Knowledge Transfer in Intraorganizational Networks : Effects of Network Position and Absorptive Capacity on Business Unit Innovation and Performance , 2007 .

[129]  Jonathan L Calof,et al.  The Integrative Domain of Foresight and Competitive Intelligence and its Impact on R&D Management , 2009 .

[130]  Ibrahim Iskin,et al.  An Assessment Model for Energy Efficiency Program Planning in Electric Utilities: Case of the Pacific of Northwest U.S.A. , 2014 .

[131]  Larry M. Dooley,et al.  The entrepreneurial university: Examining the underlying academic tensions , 2011 .

[132]  T. I. Adams,et al.  Government supported industry-university research centers: Issues for successful technology transfer , 1987 .

[133]  M. Perkmann,et al.  University Industry Relationships and Open Innovation: Towards a Research Agenda , 2007 .

[134]  Ben R. Martin,et al.  The origins of the concept of ‘foresight’ in science and technology: An insider's perspective , 2010 .

[135]  Elizabeth A. Corley,et al.  Universities : The Impacts of University Research Center Affiliation and Gender on Industrial Activities , 2015 .

[136]  S. Nunes Extending the innovation ecosystem , 2004 .

[137]  Claudio Pizzi,et al.  Boundary spanning between industry and university: the role of Technology Transfer Centres , 2012 .

[138]  George Wright,et al.  Assessing the quality of expert judgment: Issues and analysis , 1994, Decis. Support Syst..

[139]  Cristina Rossi-Lamastra,et al.  Systems of indicators to evaluate the performance of university-industry alliances: a review of the literature and directions for future research , 2013 .

[140]  Thomas L. Saaty,et al.  How to Make a Decision: The Analytic Hierarchy Process , 1990 .

[141]  Kerstin Cuhls,et al.  Current Foresight Activities in , 1999 .

[142]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[143]  D. O. Thompson,et al.  Review of Progress in Quantitative Nondestructive Evaluation , 1985 .

[144]  René Rohrbeck,et al.  Collaborative Business Modelling for Systemic and Sustainability Innovations , 2013, Int. J. Technol. Manag..

[145]  Scalable Modeling for Rigorous Software Specification and Testing , 2007 .

[146]  Katy Börner,et al.  A Multi-Level Systems Perspective for the Science of Team Science , 2010, Science Translational Medicine.

[147]  Stephen C. Betts,et al.  Making Industry—University Partnerships Work , 2002 .

[148]  Craig Boardman,et al.  The new science and engineering management: cooperative research centers as government policies, industry strategies, and organizations , 2010 .

[149]  Kun Chang Lee,et al.  KMPI: measuring knowledge management performance , 2005, Inf. Manag..

[150]  Kara L Hall,et al.  Assessing the value of team science: a study comparing center- and investigator-initiated grants. , 2012, American journal of preventive medicine.

[151]  Denis O. Gray,et al.  Making team science better: Applying improvement‐oriented evaluation principles to evaluation of cooperative research centers , 2008 .

[152]  Denis O. Gray,et al.  Industry–University Research Centers: A Multivariate Analysis of Member Retention , 2001 .

[153]  Barry Bozeman,et al.  Academic Faculty in University Research Centers: Neither Capitalism's Slaves nor Teaching Fugitives , 2012 .

[154]  Dundar F. Kocaoglu,et al.  Assessment of solar photovoltaic technologies using multiple perspectives and hierarchical decision modeling: Manufacturers worldview , 2014, Proceedings of PICMET '14 Conference: Portland International Center for Management of Engineering and Technology; Infrastructure and Service Integration.

[155]  Loet Leydesdorff,et al.  Evaluation of research and evolution of science indicators , 2005 .

[156]  Nathasit Gerdsri Strategic Planning: A Quantitative Model for the Strategic Evaluation of Emerging Technologies , 2016 .

[157]  D. Rivers To Join or Not to Join: Individual and Sub-organizational Factors Affecting Industry Membership in University-based Cooperative Research Centers 123 , 2010 .

[158]  Loet Leydesdorff,et al.  An Integrated Impact Indicator (I3): A New Definition of "Impact" with Policy Relevance , 2012, ArXiv.

[159]  Yi-Kuei Lin,et al.  Performance evaluation of extension education centers in universities based on the balanced scorecard. , 2011, Evaluation and program planning.

[160]  Rajiv Ramnath,et al.  An Agile Translation Process for complex innovations: An Industry/University Cooperative Research Center case study , 2013, 2013 IEEE Frontiers in Education Conference (FIE).

[161]  Heiko A. Gracht,et al.  Desirability bias in foresight: Consequences for decision quality based on Delphi results , 2011 .

[162]  Willy C. Shih,et al.  Bridging Science and Technology through Academic-Industry Partnerships , 2014 .

[163]  Eliezer Geisler,et al.  The metrics of knowledge: Mechanisms for preserving the value of managerial knowledge , 2007 .

[164]  Denis O. Gray,et al.  Fosteri ng Team S ci ence: Innovative Leadership Practices in NSF Industry/University Research Cooperative Centers , 2010 .

[165]  H. M. Barnes,et al.  A Preliminary Investigation of the Properties of Engineered Wood Composite Panels Treated with Copper Naphthenate , 2004 .

[166]  Tugrul U. Daim,et al.  An assessment model for energy efficiency program planning in electric utilities: Case of Northwest U.S. , 2016 .

[167]  R. Wigand,et al.  Measuring social capital through network analysis and its influence on individual performance , 2014 .

[168]  Phillip H. Phan,et al.  The Effectiveness of University Technology Transfer , 2006 .

[169]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[170]  Jung-Ki Hong,et al.  Effect of Cellulose Nanocrystals on the Rheology, Curing Behavior, and Fracture Performance of Phenol-Formaldehyde Resol Resin , 2009 .

[171]  Heiko A. von der Gracht,et al.  The influence of information and communication technology (ICT) on future foresight processes — Results from a Delphi survey , 2014 .

[172]  Jörn Altmann,et al.  On the Correlation between Research Performance and Social Network Analysis Measures Applied to Research Collaboration Networks , 2011, 2011 44th Hawaii International Conference on System Sciences.

[173]  Yasuyuki Motoyama,et al.  Think Locally, Act Locally: Building a Robust Entrepreneurial Ecosystem , 2014 .

[174]  H. Etzkowitz The norms of entrepreneurial science: cognitive effects of the new university-industry linkages , 1998 .

[175]  Davinia Palomares Montero,et al.  What are the key indicators for evaluating the activities of universities , 2011 .

[176]  Edoardo Nicoli,et al.  Characterization of Mixed-Mode Fracture Testing of Adhesively Bonded Wood Specimens , 2010 .

[177]  Kara L. Hall,et al.  The science of team science: overview of the field and introduction to the supplement. , 2008, American journal of preventive medicine.

[178]  Elias G. Carayannis,et al.  Technological learning for entrepreneurial development (TL4ED) in the knowledge economy (KE): Case studies and lessons learned , 2006 .

[179]  D. Hicks Performance-based university research funding systems , 2012 .

[180]  Luke Georghiou,et al.  Evaluation of national foresight activities: Assessing rationale, process and impact , 2006 .

[181]  J. S. Katz,et al.  What is research collaboration , 1997 .

[182]  T. Daim,et al.  Measuring the efficiency of university technology transfer , 2007 .

[183]  Michael D. Santoro,et al.  Somewhere between Markets and Hierarchies: Controlling Industry University Relationships for Success , 2011 .

[184]  Mu-Yen Chen,et al.  Measuring knowledge management performance using a competitive perspective: An empirical study , 2009, Expert Syst. Appl..

[185]  Jeryl L. Mumpower,et al.  Expert Judgement and Expert Disagreement , 1996 .

[186]  Elizabeth A. Corley,et al.  Scientists' collaboration strategies: implications for scientific and technical human capital , 2004 .

[187]  Mustafa Sulaiman Abbas,et al.  Consistency Analysis for Judgment Quantification in Hierarchical Decision Model , 2016 .

[188]  Ronald N. Kostoff,et al.  The metrics of science and technology , 2001, Scientometrics.

[189]  Monica R Geist,et al.  Using the Delphi method to engage stakeholders: a comparison of two studies. , 2010, Evaluation and program planning.

[190]  Peter Jones,et al.  Collaborative foresight: : Complementing long-horizon strategic planning , 2014 .

[191]  Jesse Loren Paris Carboxymethylcellulose Acetate Butyrate Water-Dispersions as Renewable Wood Adhesives , 2010 .