Developing a model to measure the effectiveness of research units

Research and development (RD patents; technologies successfully transferred or the external cash flow secured by a R&D organization. These are at times coupled with qualitative indicators such as the impact factor of the journals in which the papers are published and science citation index. However, all these measures often fail to adequately evaluate the effectiveness of research units (RUs) that carry out technological innovation projects. The present study examines this important conceptual question. The study attempts to develop a subjective measure of effectiveness of RU functioning within the laboratories under the Council of Scientific and Industrial Research (CSIR), India, in terms of peer review at four levels – self assessment, head of the RU, external scientific, and external administrative evaluators, using data from 236 RUs working in different laboratories functioning under CSIR. The measurement model component of structural equation modeling methodology using the LISREL 7.16 program has been adopted as the primary methodology for the study. In the measurement model presented, the observed indicators of the different effectiveness measure concepts (R&D effectiveness, user‐oriented effectiveness, administrative effectiveness and recognition) are assumed to possess causal relations with one another. Since subjective measures have not only weak measurement properties, but are also influenced by systematic and random measurement errors, the model's reliability and construct validity – both in terms of convergent validity and discriminate validity – have been ascertained.

[1]  R. Bagozzi,et al.  Representing and testing organizational theories: A holistic construal. , 1982 .

[2]  L. Tyler.  The psychology of human differences. , 1947 .

[3]  Willem E. Saris,et al.  Causal modelling in nonexperimental research : an introduction to the LISREL approach , 1985, American Political Science Review.

[4]  B. Martin,et al.  Some partial indicators of scientific progress in radio astronomy , 1983 .

[5]  Karl G. Jöreskog,et al.  Lisrel 8: Structural Equation Modeling With the Simplis Command Language , 1993 .

[6]  Jeffrey S. Tanaka,et al.  Influence of sample size, estimation method, and model specification on goodness-of-fit assessments in structural equation models. , 1989 .

[7]  William T. Robinson,et al.  Industrial Organization and Consumer Satisfaction-Dissatisfaction , 1983 .

[8]  J. Fox,et al.  Linear Statistical Models and Related Methods: With Applications to Social Research. , 1986 .

[9]  William E. Souder,et al.  Measuring R&D Performance-State of the Art , 1997 .

[10]  Dwight D. Frink,et al.  The Effects of Organizational Restructure on Employee Satisfaction , 1996 .

[11]  Bowon Kim,et al.  An effective R&D performance measurement system: survey of Korean R&D researchers , 2002 .

[12]  John Fox,et al.  Linear Statistical Models and Related Methods; With Applications to Social Research. , 1985 .

[13]  Raynold A. Svenson,et al.  Measuring R&D Productivity , 1988 .

[14]  Madhukar G. Angur,et al.  A causal exploration of `innovatorship' and `lead-usership': a research note , 1998 .

[15]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[16]  Claes Fornell,et al.  Issues in the Application of Covariance Structure Analysis: A Comment , 1983 .

[17]  K. Jöreskog.  A General Method for Estimating a Linear Structural Equation System. , 1970 .

[18]  Leslie A. Hayduk.  Structural equation modeling with LISREL: essentials and advances , 1987 .

[19]  Robert Szakonyi,et al.  Measuring R&D Effectiveness—I , 1994 .

[20]  P. Bentler,et al.  Significance Tests and Goodness of Fit in the Analysis of Covariance Structures , 1980 .

[21]  Bill Nixon,et al.  Productivity and performance measurement in R&D , 1997 .

[22]  I. Feller.  Performance Measurement Redux , 2002 .

[23]  R. Bagozzi,et al.  On the evaluation of structural equation models , 1988 .

[24]  Richard P. Bagozzi,et al.  "Issues in the application of covariance structure analysis": A further comment. , 1983 .

[25]  Sven Hemlin,et al.  Research production in the arts and humanities , 1996, Scientometrics.

[26]  Thomas A. DeCotiis,et al.  Defining and Measuring Project Performance , 1979 .

[27]  H. Moed,et al.  The use of bibliometric data for the measurement of university research performance , 1985 .

[28]  Richard P. Bagozzi,et al.  A Holistic Methodology for Modeling Consumer Response to Innovation , 1983, Oper. Res..

[29]  K. G. Jöreskog,et al.  Efficient estimation in image factor analysis. , 1969 .

[30]  Mushin Lee,et al.  Measuring R&D Effectiveness in Korean Companies , 1996 .

[31]  C. Schriesheim.  Causal Analysis: Assumptions, Models, and Data , 1982 .

[32]  K. Parkes.  Structural Modeling by Example: Field dependence and the differentiation of neurotic syndromes , 1988 .

[33]  Richard L. Daft,et al.  Organization Theory and Design , 1983 .

[34]  S. Reddy.  Effects of Ignoring Correlated Measurement Error in Structural Equation Models , 1992 .

[35]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[36]  James C. Anderson,et al.  The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis , 1984 .

[37]  S. Koreisha,et al.  The impact of measurement errors on ARMA prediction , 1999 .

[38]  Kim S. Cameron,et al.  A Study of Organizational Effectiveness and Its Predictors. ASHE 1983 Annual Meeting Paper. , 1983 .

[39]  K. Jöreskog.  Structural analysis of covariance and correlation matrices , 1978 .

[40]  K. Jöreskog,et al.  Intraclass Reliability Estimates: Testing Structural Assumptions , 1974 .

[41]  J. S. Long,et al.  Estimation and Hypothesis Testing in Linear Models Containing Measurement Error , 1976 .

[42]  L. Hayduk.  Structural equation modeling with LISREL , 1987 .

[43]  A. Basilevsky,et al.  Factor Analysis as a Statistical Method. , 1964 .

[44]  A. V. D. Ven,et al.  Determinants of Coordination Modes within Organizations , 1976 .

[45]  A. E. Maxwell,et al.  Factor Analysis as a Statistical Method. , 1964 .

[46]  M. Afzalur Rahim,et al.  A structural equations model of stress, locus of control, social support, psychiatric symptoms, and propensity to leave a job. , 1996, The Journal of social psychology.

[47]  M. Browne.  Generalized Least Squares Estimators in the Analysis of Covariance Structures. , 1973 .

[48]  Edward G. Carmines,et al.  Reliability and Validity Assessment , 1979 .

[49]  Walter Dick,et al.  Topics in measurement reliability and validity , 1971 .

[50]  M. R. Moser.  Measuring Performance in R&D Settings : Research Management , 1987 .

[51]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[52]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[53]  J. S. Long,et al.  Covariance Structure Models: An Introduction to LISREL , 1983 .

[54]  K. Bollen.  A New Incremental Fit Index for General Structural Equation Models , 1989 .

[55]  Stuart Jay Deutsch,et al.  Organizational Effectiveness: A Multiple-Constituency Approach , 1980 .

[56]  R. Whitley,et al.  The Measurement of Performance in Research , 1971 .

[57]  Kenneth A. Bollen,et al.  Overall Fit in Covariance Structure Models: Two Types of Sample Size Effects , 1990 .

[58]  M. Moser.  Measuring Performance in R&D Settings , 1985 .

[59]  C. P. Hansen,et al.  A causal model of the relationship among accidents, biodata, personality, and cognitive factors. , 1989 .

[60]  N. Toren,et al.  The determinants of the potential effectiveness of government-supported industrial research institutes , 1978 .