Measuring the Performance of Information Systems: A Functional Scorecard

This study develops an instrument that may be used as an information systems (IS) functional scorecard (ISFS). It is based on a theoretical input-output model of the IS function's role in supporting business process effectiveness and organizational performance. The research model consists of three system output dimensions--systems performance, information effectiveness, and service performance. The "updated paradigm" for instrument development was followed to develop and validate the ISFS instrument. Construct validation of the instrument was conducted using responses from 346 systems users in 149 organizations by a combination of exploratory factor analysis and structural equation modeling using LISREL. The process resulted in an instrument that measures 18 unidimensional factors within the three ISFS dimensions. Moreover, a sample of 120 matched-paired responses of separate CIO and user responses was used for nomological validation. The results showed that the ISFS measure reflected by the instrument was positively related to improvements in business processes effectiveness and organizational performance. Consequently, the instrument may be used for assessing IS performance, for guiding information technology investment and sourcing decisions, and as a basis for further research and instrument development.

[1]  James C. Wetherbe,et al.  Key Issues in Information Systems Management: 1994-95 SIM Delphi Results , 1996, MIS Q..

[2]  Arun Rai,et al.  Assessing the Validity of IS Success Models: An Empirical Test and Theoretical Analysis , 2002, Inf. Syst. Res..

[3]  Sammy W. Pearson,et al.  Development of a Tool for Measuring and Analyzing Computer User Satisfaction , 1983 .

[4]  Sara L. Rynes,et al.  Making total quality work: Aligning organizational processes, performance measures, and stakeholders , 1991 .

[5]  Robert J. Kauffman,et al.  Limits to value in electronic commerce-related IT investments , 2000, Proceedings of the 33rd Annual Hawaii International Conference on System Sciences.

[6]  Heeseok Lee,et al.  Knowledge Management Enablers, Processes, and Organizational Performance: An Integrative View and Empirical Examination , 2003, J. Manag. Inf. Syst..

[7]  Leon A. Kappelman,et al.  A Comprehensive Model for Assessing the Quality and Productivity of the Information Systems Function: Toward a Theory for Information Systems Assessment , 1997 .

[8]  Dale Goodhue,et al.  Task-Technology Fit and Individual Performance , 1995, MIS Q..

[9]  Bernard Baruch,et al.  Measuring e-Commerce in Net-Enabled Organizations: An Introduction to the Special Issue , 2002 .

[10]  William R. King,et al.  Integration between Business Planning and Information Systems Planning: An Evolutionary-Contingency Perspective , 1997, J. Manag. Inf. Syst..

[11]  Rashi Glazer.  Measuring the Knower: Towards a Theory of Knowledge Equity , 1998 .

[12]  Heeseok Lee,et al.  DEVELOPING A BUSINESS PERFORMANCE EVALUATION SYSTEM: AN ANALYTIC HIERARCHICAL MODEL , 1995 .

[13]  Erik Brynjolfsson,et al.  The productivity paradox of information technology , 1993, CACM.

[14]  Susan T. Dumais,et al.  Computerization, productivity, and quality of work-life , 1989, CACM.

[15]  C. H. Kriebel.  Management information systems , 1970, ACM '70.

[16]  David W. Gerbing,et al.  An Updated Paradigm for Scale Development Incorporating Unidimensionality and Its Assessment , 1988 .

[17]  James C. Wetherbe,et al.  Key issues in information systems management , 1987 .

[18]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[19]  Yolande E. Chan,et al.  Business Strategic Orientation, Information Systems Strategic Orientation, and Strategic Alignment , 1997, Inf. Syst. Res..

[20]  Akhil Kumar,et al.  Research Commentary: Workflow Management Issues in e-Business , 2002, Inf. Syst. Res..

[21]  Kar Yan Tam,et al.  The Impact of Information Technology Investments on Firm Performance and Evaluation: Evidence from Newly Industrialized Economies , 1998, Inf. Syst. Res..

[22]  William R. King,et al.  The Development of a Measure for End-User Computing Support , 1994 .

[23]  Kenneth L. Kraemer,et al.  The Usefulness of Computer-Based Information to Public Managers , 1993, MIS Q..

[24]  Ephraim R. McLean,et al.  The DeLone and McLean Model of Information Systems Success: A Ten-Year Update , 2003, J. Manag. Inf. Syst..

[25]  Peter B. Seddon.  A Respecification and Extension of the DeLone and McLean Model of IS Success , 1997, Inf. Syst. Res..

[26]  Diane M. Strong,et al.  Beyond Accuracy: What Data Quality Means to Data Consumers , 1996, J. Manag. Inf. Syst..

[27]  James C. Anderson,et al.  An approach for confirmatory measurement and structural equation modeling of organizational properties , 1987 .

[28]  Edward J. Garrity,et al.  Information systems success measurement , 1998 .

[29]  John Hattie,et al.  Methodology Review: Assessing Unidimensionality of Tests and ltenls , 1985 .

[30]  Richard T. Watson,et al.  Service Quality: A Measure of Information System Effectiveness , 1995, MIS Q..

[31]  Stephanie Watts,et al.  Informational Influence in Organizations: An Integrated Approach to Knowledge Adoption , 2003, Inf. Syst. Res..

[32]  Albert H. Segars,et al.  Assessing the unidimensionality of measurement : a paradigm and illustration within the context of information systems research , 1997 .

[33]  Terry S. Overton,et al.  Estimating Nonresponse Bias in Mail Surveys , 1977 .

[34]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1973 .

[35]  Vallabh Sambamurthy,et al.  The Shareholder-Wealth and Trading-Volume Effects of Information-Technology Infrastructure Investments , 2002, J. Manag. Inf. Syst..

[36]  Vallabh Sambamurthy,et al.  Shaping UP for E-Commerce: Institutional Enablers of the Organizational Assimliation of Web Technologies , 2002, MIS Q..

[37]  Suzanne Rivard,et al.  Investigating the Support Role of the Information Center , 1990, MIS Q..

[38]  Jinwoo Kim,et al.  Businesses as Buildings: Metrics for the Architectural Quality of Internet Businesses , 2002, Inf. Syst. Res..

[39]  Albert H. Segars,et al.  Information systems effectiveness: The construct space and patters of application , 1996, Inf. Manag..

[40]  Richard Y. Wang,et al.  Anchoring data quality dimensions in ontological foundations , 1996, CACM.

[41]  Gary F. Templeton,et al.  Development of a Measure for the Organizational Learning Construct , 2002, J. Manag. Inf. Syst..

[42]  Qiang Tu,et al.  Dimensionality of the Strategic Grid Framework: The Construct and Its Measurement , 1999, Inf. Syst. Res..

[43]  P. Weill,et al.  Management by Maxim: How Business and IT Managers Can Create IT Infrastructures , 1997 .

[44]  Carol Stoak Saunders,et al.  Measuring Performance of the Information Systems Function , 1992, J. Manag. Inf. Syst..

[45]  Richard T. Watson,et al.  Measuring Information Systems Service Quality: Lessons From Two Longitudinal Case Studies , 1998, MIS Q..

[46]  James C. Wetherbe,et al.  Key Information Systems Issues for the 1980's , 1984, MIS Q..

[47]  Gary Klein,et al.  Measuring Information System Service Quality: SERVQUAL from the Other Side , 2002, MIS Q..

[48]  D. Whetten,et al.  12 – Some Conclusions about Organizational Effectiveness , 1983 .

[49]  G Premkumar.  Evaluation of strategic information system planning : empirical validation of a conceptual model , 1989 .

[50]  Leslie P. Willcocks,et al.  Global Information Technology Outsourcing: In Search of Business Advantage , 2001 .

[51]  Richard M. Steers.  Problems in the Measurement of Organizational Effectiveness , 1975 .

[52]  D. Sandy Staples,et al.  Dimensions of Information Systems Success , 1999, Commun. Assoc. Inf. Syst..

[53]  Rajiv Kohli,et al.  Measuring Information Technology Payoff: A Meta - Analysis of Structural Variables in Firm - Level Empirical Research , 2003, Inf. Syst. Res..

[54]  Timo Saarinen,et al.  An expanded instrument for evaluating information system success , 1996, Inf. Manag..

[55]  Martin Herbert,et al.  1985 Opinion Survey of MIS Managers: Key Issues , 1986, MIS Q..

[56]  Christopher Lindsey Carr.  Managing service quality at the IS help desk : toward the development and testing of tech-qual, a model of information systems (IS) technical support service quality , 1999 .

[57]  Wanda J. Orlikowski,et al.  A Short Form Measure of User Information Satisfaction: A Psychometric Evaluation and Notes on Use , 1987, J. Manag. Inf. Syst..

[58]  Albert H. Segars,et al.  Strategic Information Systems Planning Success: An Investigation of the Construct and Its Measurement , 1998, MIS Q..

[59]  Leon A. Kappelman,et al.  Measuring Information Systems Service Quality: Concerns on the Use of the SERVQUAL Questionnaire , 1997, MIS Q..

[60]  M. Porter,et al.  How Information Gives You Competitive Advantage , 1985 .

[61]  Izak Benbasat,et al.  Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation , 1991, Inf. Syst. Res..

[62]  Brian J. Reithel,et al.  Beyond the information center: An instrument to measure end-user computing support from multiple sources , 1998, Inf. Manag..

[63]  Steven A. Taylor,et al.  Servperf versus Servqual: Reconciling Performance-Based and Perceptions-Minus-Expectations Measurement of Service Quality , 1994 .

[64]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[65]  David Gefen,et al.  Implementation Team Responsiveness and User Evaluation of Customer Relationship Management: A Quasi-Experimental Design Study of Social Exchange Theory , 2002, J. Manag. Inf. Syst..

[66]  Paul H. Cheney,et al.  Training End Users: An Exploratory Study , 1987, MIS Q..

[67]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[68]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[69]  D. Whetten,et al.  Organizational Effectiveness: A Comparison of Multiple Models , 1984 .

[70]  D. J. Wu,et al.  Investment in Enterprise Resource Planning: Business Impact and Productivity Measures , 2002, J. Manag. Inf. Syst..

[71]  Terry Anthony Byrd,et al.  Measuring the Flexibility of Information Technology Infrastructure: Exploratory Analysis of a Construct , 2000, J. Manag. Inf. Syst..

[72]  Douglas N. Jackson,et al.  Problems in Human Assessment , 1978 .

[73]  James C. Anderson,et al.  STRUCTURAL EQUATION MODELING IN PRACTICE: A REVIEW AND RECOMMENDED TWO-STEP APPROACH , 1988 .

[74]  Gurpreet Dhillon,et al.  TORKZADEH AND DHILLON Measuring Factors that Influence the Success of Internet , 2015 .

[75]  Margi Levy,et al.  Developing a 3-D model of information systems success , 1998 .

[76]  Peter Weill,et al.  Firm Context and Patterns of IT Infrastructure Capability (Best Paper Award) , 1996, ICIS.

[77]  Sherry D. Ryan,et al.  Considering Social Subsystem Costs and Benefits in Information Technology Investment Decisions: A View from the Field on Anticipated Payoffs , 2000, J. Manag. Inf. Syst..

[78]  Michel Benaroch,et al.  Managing Information Technology Investment Risk: A Real Options Perspective , 2002, J. Manag. Inf. Syst..

[79]  R. Johnston,et al.  Performance Measurement in Service Businesses , 1992 .

[80]  Albert H. Segars,et al.  Value, knowledge, and the human equation: , 2000 .

[81]  Peter Weill,et al.  The Relationship Between Investment in Information Technology and Firm Performance: A Study of the Valve Manufacturing Sector , 1992, Inf. Syst. Res..

[82]  W. Doll,et al.  The development of a tool for measuring the perceived impact of information technology on work , 1999 .

[83]  Fred Niederman,et al.  Information Systems Management Issues for the 1990s , 1991, MIS Q..

[84]  Kenneth L. Kraemer,et al.  Executives’ Perceptions of the Business Value of Information Technology: A Process-Oriented Approach , 2000, J. Manag. Inf. Syst..

[85]  Tor Guimaraes,et al.  Exploring the relationship between IC success and company performance , 1994, Inf. Manag..

[86]  Kai R. T. Larsen,et al.  A Taxonomy of Antecedents of Information Systems Success: Variable Analysis Studies , 2003, J. Manag. Inf. Syst..

[87]  Vijay Sethi,et al.  Development of measures to assess the extent to which an information technology application provides competitive advantage , 1994 .

[88]  N. Venkatraman,et al.  Measurement of Business Economic Performance: An Examination of Method Convergence , 1987 .

[89]  Peter Gyngell,et al.  Process Innovation: Reengineering Work through Information Technology , 1994 .

[90]  Rajiv Kohli,et al.  Antecedents of B2C Channel Satisfaction and Preference: Validating e-Commerce Metrics , 2002, Inf. Syst. Res..

[91]  Albert H. Segars,et al.  Knowledge Management: An Organizational Capabilities Perspective , 2001, J. Manag. Inf. Syst..

[92]  Radhika Santhanam,et al.  Issues in Linking Information Technology Capability to Firm Performance , 2003, MIS Q..

[93]  J. S. Long,et al.  Testing Structural Equation Models , 1993 .

[94]  Andrew B. Whinston,et al.  Decentralized Mechanism Design for Supply Chain Organizations Using an Auction Market , 2003, Inf. Syst. Res..

[95]  Hugh J. Watson,et al.  Critical Success Factors for Information Center Managers , 1988, MIS Q..

[96]  Gregory G. Dess,et al.  Measuring organizational performance in the absence of objective measures: The case of the privately-held firm and conglomerate business unit , 1984 .

[97]  K. Cameron,et al.  A Study of Organizational Effectiveness and Its Predictors , 1983 .

[98]  Youngjin Yoo,et al.  A Comparative Study of Distributed Learning Environments on Learning Outcomes , 2002, Inf. Syst. Res..

[99]  A. Parasuraman,et al.  Refinement and reassessment of the SERVQUAL scale. , 1991 .

[100]  William J. Doll,et al.  The Measurement of End-User Computing Satisfaction , 1988, MIS Q..

[101]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[102]  R. Bagozzi.  An Examination Of The Validity Of Two Models Of Attitude. , 1981, Multivariate behavioral research.

[103]  Judy E. Scott.  Facilitating Interorganizational Learning with Information Technology , 2000, J. Manag. Inf. Syst..

[104]  Kay M. Nelson,et al.  International Conference on Information Systems ( ICIS ) 1993 THE CONTRIBUTION OF SHARED KNOWLEDGE TO IS GROUP PERFORMANCE , 2017 .

[105]  Varun Grover,et al.  General Perspectives on Knowledge Management: Fostering a Research Agenda , 2001, J. Manag. Inf. Syst..

[106]  Ravinder Nath,et al.  An empirical examination of the impact of computer information systems on users , 1995, Inf. Manag..

[107]  Detmar W. Straub,et al.  Measuring e-Commerce in Net-Enabled Organizations: An Introduction to the Special Issue , 2002, Inf. Syst. Res..

[108]  Connie Eleanor Wells,et al.  Evaluation of the MIS function in an organization : an exploration of the measurement problem , 1987 .

[109]  Nancy Bogucki Duncan,et al.  Capturing Flexibility of Information Technology Infrastructure: A Study of Resource Characteristics and their Measure , 1995, J. Manag. Inf. Syst..

[110]  Albert H. Segars,et al.  Re-examining perceived ease of use and usefulness , 1993 .