A Comprehensive Model for Assessing the Quality and Productivity of the Information Systems Function: Toward a Theory for Information Systems Assessment

Information Systems IS managers are under increasing pressure to justify the value and contribution of IS expenditures to the productivity, quality, and competitiveness of the organization. This paper examines the need for IS assessment and suggests a comprehensive IS assessment framework linked to organizational performance using existing IS assessment theory as a base and incorporating measurement concepts from other disciplines. The existing models of IS success are updated to include the emerging IS success dimensions of service quality and work group impact and provide a comprehensive method for organizing the various measures of IS success. In addition, many new measures from recent research are presented to supplement the lists supplied by previous research. Additional research is suggested to advance the IS assessment contingency theory. Such a theory has the potential to contribute to the quality and productivity of the IS function and the larger organization by providing feedback to manage and improve the IS function to better meet the needs of the organization.

[1]  A. Parasuraman,et al.  Reassessment of expectations as a comparison standard in measuring service quality: Implications , 1994 .

[2]  Delmar L. Landen,et al.  Understanding—and improving— organizations through cybernetics , 1990 .

[3]  Prashant Palvia,et al.  The PRISM system: a key to organizational effectiveness at Federal Express Corporation , 1992 .

[4]  Gary W. Dickson,et al.  Assessment of the Information Systems Organization: an Empirical Investigation of assessor Perspectives , 1987, ICIS.

[5]  James C. Wetherbe,et al.  Key issues in information systems management , 1987 .

[6]  Richard Scudder,et al.  Productivity measures for information systems , 1991, Inf. Manag..

[7]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[8]  Louis A. Le Blanc,et al.  An Empirical Investigation of the Relationship Between DSS Usage and System Performance: A Case Study of a Navigation Support System , 1990, MIS Q..

[9]  Charles W. Hofer.  Toward a Contingency Theory of Business Strategy , 1975 .

[10]  Maryellen R. Kelley,et al.  Productivity and Information Technology: The Elusive Connection , 1994 .

[11]  P. S. Goodman,et al.  Perspectives and issues: An introduction , 1977 .

[12]  Craig K. Tyran,et al.  The Application of Electronic Meeting Technology to Support Strategic Management , 1992, MIS Q..

[13]  Mo Adam Mahmood,et al.  Effect of Organizational Maturity on End-Users' Satisfaction with Information Systems , 1986, J. Manag. Inf. Syst..

[14]  A. Parasuraman,et al.  The nature and determinants of customer expectations of service , 1993 .

[15]  William J. Doll,et al.  A Confirmatory Factor Analysis of the End-User Computing Satisfaction Instrument , 1994, MIS Q..

[16]  Dennis F. Galletta,et al.  Some Cautions on the Measurement of User Information Satisfaction , 1989 .

[17]  Paul A. Strassmann,et al.  The Business Value of Computers: An Executive's Guide , 1990 .

[18]  A. Lewin,et al.  Determining organizational effectiveness: another look, and an agenda for research , 1986 .

[19]  Sammy W. Pearson,et al.  Development of a Tool for Measuring and Analyzing Computer User Satisfaction , 1983 .

[20]  Jeff Moad.  Successful benchmarking with the team approach , 1994 .

[21]  Jennifer L. Hartnett,et al.  Managing Quality: The Strategic and Competitive Edge , 1988 .

[22]  William J. Kettinger,et al.  Perceived Service Quality and User Satisfaction with the Information Services Function , 1994 .

[23]  James C. Wetherbe,et al.  Key Information Systems Issues for the 1980's , 1984, MIS Q..

[24]  Kim S. Cameron,et al.  A Study of Organizational Effectiveness and Its Predictors. ASHE 1983 Annual Meeting Paper. , 1983 .

[25]  Ronald B. Wilkes,et al.  Assessment of the information systems function: a study of the is/organizational interface , 1989 .

[26]  Jay F. Nunamaker,et al.  Information Technology to Support Electronic Meetings , 1988, MIS Q..

[27]  A. Parasuraman,et al.  More on improving service quality measurement , 1993 .

[28]  A. Parasuraman,et al.  A Conceptual Model of Service Quality and Its Implications for Future Research , 1985 .

[29]  Dan Remenyi,et al.  Service quality and correspondence analysis in determining problems with the effective use of computer services , 1993, ECIS.

[30]  Peter Weill,et al.  An Assessment of the Contingency Theory of Management Information Systems , 1989, J. Manag. Inf. Syst..

[31]  Ravinder Nath.  A framework to identify applications of information technology to improve service quality , 1992 .

[32]  D. Peppers,et al.  A new marketing paradigm: Share of customer, not market share , 1995 .

[33]  Yannis Bakos,et al.  Dependent Variables for the Study of Firm and Industry-Level Impacts of Information Technology , 2013, ICIS.

[34]  A. Parasuraman,et al.  Prescriptions for a service quality revolution in America , 1992 .

[35]  Harry C. Triandis,et al.  The Human Organization: Its Management and Value. , 1967 .

[36]  Gerardine DeSanctis,et al.  Computer-Based Support for Group Problem-Finding: An Experimental Investigation , 1988, MIS Q..

[37]  Douglas R. Vogel,et al.  Technological Support for Group Process Modeling , 1994, J. Manag. Inf. Syst..

[38]  Carol Stoak Saunders,et al.  Measuring Performance of the Information Systems Function , 1992, J. Manag. Inf. Syst..

[39]  Peter B. Seddon,et al.  A Partial Test and Development of the DeLone and McLean Model of IS Success , 1994, ICIS.

[40]  Richard M. Hodgetts,et al.  Blueprints for Continuous Improvement: Lessons from the Baldrige Winners , 1993 .

[41]  Kim S. Cameron,et al.  Critical questions in assessing organizational effectiveness , 1980 .

[42]  Louis Raymond,et al.  Validating and applying user satisfaction as a measure of MIS success in small organizations , 1987, Inf. Manag..

[43]  V. Zeithaml,et al.  The Contingency Approach: Its Foundations and Relevance to Theory Building and Research in Marketing , 1988 .

[44]  R. D. Buzzell,et al.  The PIMS Principles: Linking Strategy to Performance , 1987 .

[45]  James A. Senn,et al.  Research in Management Information Systems: The Minnesota Experiments , 1977 .

[46]  A. Parasuraman,et al.  Alternative scales for measuring service quality: A comparative assessment based on psychometric and diagnostic criteria , 1994 .

[47]  Robert J. Thierauf.  Effective Information Centers: Guidelines for MIS and IC Managers , 1988 .

[48]  A. Parasuraman,et al.  Communication and Control Processes in the Delivery of Service Quality , 1988, Journal of Marketing.

[49]  H. Thaine Lyman,et al.  Keeping the customer satisfied. The new CIO mandate , 1992 .

[50]  Mark T. Norris,et al.  Software process modelling and measurement: a QMS case study , 1993, Inf. Softw. Technol..

[51]  R. Zmud.  INDIVIDUAL DIFFERENCES AND MIS SUCCESS: A REVIEW OF THE EMPIRICAL LITERATURE* , 1979 .

[52]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[53]  Blake Ives,et al.  The measurement of user information satisfaction , 1983, CACM.

[54]  Fred Niederman,et al.  Information Systems Management Issues for the 1990s , 1991, MIS Q..

[55]  L Thornburg.  Performance measures that work. , 1991, HRMagazine : on human resource management.

[56]  Kevin Crowston,et al.  Assessing the Impact of Information Technology on Enterprise Level Performance , 1986, ICIS.

[57]  Vijay Sethi,et al.  Development of measures to assess the extent to which an information technology application provides competitive advantage , 1994 .

[58]  Sidney E. Harris,et al.  Firm Size and the Information Technology Investment Intensity of LIfe Insurers , 1991, MIS Q..

[59]  Niv Ahituv,et al.  Procedures and Practices for Conducting Postevaluation of Information Systems , 1986 .

[60]  Gordon B. Davis,et al.  A Framework for Research in Computer-Based Management Information Systems , 1980 .

[61]  Valarie A. Zeithaml,et al.  The service-quality puzzle , 1988 .

[62]  Leonard L. Berry,et al.  Five imperatives for improving service quality , 1991 .

[63]  Donald H. Bender,et al.  Financial Impact of Information Processing , 1986, J. Manag. Inf. Syst..

[64]  D. Hambrick.  Environment, strategy, and power within top management teams. , 1981, Administrative science quarterly.

[65]  R. Kaplan,et al.  The balanced scorecard--measures that drive performance. , 2015, Harvard business review.

[66]  Thomas D. Clark,et al.  Corporate systems management: an overview and research perspective , 1992, CACM.

[67]  Charles H. Kriebel,et al.  A Method for Assessing the Economic Impact of Information Systems Technology on Organizations , 1985, ICIS.

[68]  Jeff Moad.  New rules, new ratings as IS reengineers , 1993 .

[69]  William J. Doll,et al.  The place and value of documentation in end-user computing , 1993, Inf. Manag..

[70]  John Leslie King,et al.  Cost-Benefit Analysis in Information Systems Development and Operation , 1978, CSUR.

[71]  Armand V. Feigenbaum,et al.  Total quality control , 1961 .

[72]  Thompson Gb,et al.  Hospitals, like industry, must apply cost accounting techniques. , 1978 .

[73]  S. Harris,et al.  Organizational Performance and Information Technology Investment Intensity in the Insurance Industry , 1991 .

[74]  Richard O. Mason,et al.  Measuring information output: A communication systems approach , 1978, Inf. Manag..

[75]  Kim S. Cameron,et al.  Measuring Organizational Effectiveness in Institutions of Higher Education. , 1978 .

[76]  D. Seymour.  On Q: Causing Quality In Higher Education , 1992 .

[77]  Domenico Ferrari,et al.  Computer Systems Performance Evaluation , 1978 .

[78]  A. Parasuraman,et al.  Delivering quality service : balancing customer perceptions and expectations , 1990 .

[79]  Moshe Zviran,et al.  A measure for determining the strategic relevance of IS to the organization , 1992, Inf. Manag..

[80]  Jay F. Nunamaker,et al.  Implementing Electronic Meeting Systems at IBM: Lessons Learned and Success Factors , 1990, MIS Q..

[81]  R. Eccles.  The performance measurement manifesto. , 1991, Harvard business review.

[82]  David J. Miller,et al.  The Architecture of Simplicity , 1993 .

[83]  A. Parasuraman,et al.  Refinement and reassessment of the SERVQUAL scale. , 1991 .

[84]  William R. King,et al.  Evaluating Management Informatoin Systems , 1978, MIS Q..

[85]  Erik Brynjolfsson,et al.  The productivity paradox of information technology , 1993, CACM.

[86]  Jari M. Talvinen,et al.  Integration and Effectiveness of Marketing Information Systems , 1993 .

[87]  A. Parasuraman,et al.  SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. , 1988 .

[88]  Carl R. Jones,et al.  Information Systems Management: Analytical Tools and Techniques , 1985 .

[89]  F. F. Reichheld,et al.  Zero defections: quality comes to services. , 1990, Harvard business review.

[90]  David Lei,et al.  Toward an Empirical Prioritization of Contingency Variables for Business Strategy , 1985 .

[91]  J. William Semich.  Here's how to quantify IT investment benefits , 1994 .

[92]  Richard M. Steers.  Problems in the Measurement of Organizational Effectiveness , 1975 .

[93]  William R. King,et al.  Organizational Characteristics and Information Systems Planning: An Empirical Study , 1994, Inf. Syst. Res..

[94]  Jay F. Nunamaker,et al.  Electronic meeting systems , 1991, CACM.

[95]  Richard Harris,et al.  SMIS Members: A Membership Analysis , 1982, MIS Q..

[96]  Tor Guimaraes,et al.  The Relationship Between User Participation and User Satisfaction: An Investigation of Four Contingency Factors , 1994, MIS Q..

[97]  Scott Hamilton,et al.  Evaluating Information Systems Effectiveness - Part I: Comparing Evaluation Approaches , 1981, MIS Q..

[98]  Ephraim R. McLean,et al.  SIM Competition Paper: Measuring Information Systems Performance: Experience with the Management By Results System at Security Pacific Bank , 1988, MIS Q..

[99]  Scott Hamilton,et al.  Evaluating Information Systems Effectiveness - Part II: Comparing Evaluator Viewpoints , 1981, MIS Q..

[100]  Harlow B. Cohen.  The performance paradox , 1998 .

[101]  Peter F. Drucker,et al.  The New Realities , 1989 .

[102]  David W. Conrath,et al.  What is being done to measure user satisfaction with EDP/MIS , 1990, Inf. Manag..

[103]  William J. Doll,et al.  Documenting information systems for management: A key to maintaining user satisfaction , 1985, Inf. Manag..

[104]  James C. Wetherbe,et al.  A Contingency View of Managing the Data Processing Organization , 1977, MIS Q..

[105]  Neil D. Burns,et al.  Performance Measurement Revisited , 1992 .

[106]  A. Parasuraman,et al.  Perceived service quality as a customer‐based performance measure: An empirical examination of organizational barriers using an extended service quality model , 1991 .

[107]  Gerald L. Matlin,et al.  SIM Competition Paper: What Is the Value of Investment in Information Systems? , 1979, MIS Q..

[108]  Gerald L. Matlin.  How to Survive a Management Assessment , 1977, MIS Q..

[109]  Thomas A. Mahoney,et al.  Managerial Models of Organizational Effectiveness. , 1969 .

[110]  A. Parasuraman,et al.  Improving service quality in America: Lessons learned , 1994 .

[111]  Jay F. Nunamaker,et al.  Bringing automated support to large groups: The Burr-Brown experience , 1990, Inf. Manag..

[112]  A. V. D. Ven,et al.  Measuring And Assessing Organizations , 1980 .

[113]  A. L. Mendelow,et al.  Setting corporate goals and measuring organizational effectiveness--a practical approach. , 1983, Long range planning.

[114]  T. R. Davis.  Information technology and white-collar productivity , 1991 .

[115]  Carol V. Brown,et al.  Alignment of the IS Functions With the Enterprise: Toward a Model of Antecedents , 1994, MIS Q..

[116]  Avi Rushinek,et al.  What makes users happy? , 1986, CACM.

[117]  K. Cameron.  Effectiveness as Paradox: Consensus and Conflict in Conceptions of Organizational Effectiveness , 1986 .

[118]  N. Melone.  A theoretical assessment of the user-satisfaction construct in information systems research , 1990 .

[119]  Avi Rushinek,et al.  The Influence of Troubleshooting, Education, and Documentation on Computer User Satisfaction , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[120]  P. Strassmann.  Information Payoff: The Transformation of Work in the Electronic Age , 1985 .

[121]  Peter G. W. Keen,et al.  Business Teams in an Information Age , 1989, Inf. Soc..

[122]  P. Ein-Dor,et al.  Organizational Context and the Success of Management Information Systems , 1978 .

[123]  Mo Adam Mahmood,et al.  Effect of the quality of user documentation on user satisfaction with information systems , 1990, Inf. Manag..

[124]  Robert M. Mason,et al.  A lower bound cost benefit model for information services , 1978, Inf. Process. Manag..

[125]  Leon A. Kappelman,et al.  Converging end-user and corporate computing , 1993, CACM.

[126]  Robert C. Camp,et al.  Benchmarking: The Search for Industry Best Practices That Lead to Superior Performance , 1989 .

[127]  Amy W. Gatian.  Is user satisfaction a valid measure of system effectiveness? , 1994, Inf. Manag..

[128]  D. Whetten,et al.  12 – Some Conclusions about Organizational Effectiveness , 1983 .

[129]  Danny Miller,et al.  The Icarus Paradox: How Exceptional Companies Bring About Their Own Downfall , 1990 .

[130]  R. Kelly Rainer,et al.  The Keys to Executive Information Systems Success , 1995, J. Manag. Inf. Syst..

[131]  Charles H. Kriebel,et al.  Information Technologies and Business Value: An Analytic and Empirical Investigation , 1995, Inf. Syst. Res..

[132]  Richard L. Nolan,et al.  Toward a comprehensive framework for MIS research , 1980 .

[133]  T.C.E. Cheng,et al.  Management Support Systems for Service Quality Management: A Research Direction , 1994 .

[134]  Tor Guimaraes,et al.  Measuring top management satisfaction with the MIS department , 1988 .

[135]  Martin Herbert,et al.  1985 Opinion Survey of MIS Managers: Key Issues , 1986, MIS Q..

[136]  Wanda J. Orlikowski,et al.  A Short Form Measure of User Information Satisfaction: A Psychometric Evaluation and Notes on Use , 1987, J. Manag. Inf. Syst..

[137]  Christine B. Tayntor.  Partners in excellence. Metrics and productivity programs , 1994 .

[138]  Sunil Babbar,et al.  A Dynamic Model for Continuous Improvement in the Management of Service Quality , 1992 .

[139]  Niv Ahituv,et al.  A Systematic Approach Toward Assessing the Value of an Information System , 1980, MIS Q..

[140]  Richard T. Watson,et al.  Service Quality: A Measure of Information System Effectiveness , 1995, MIS Q..

[141]  R. Rust,et al.  Return on Quality (ROQ): Making Service Quality Financially Accountable , 1995 .

[142]  Lorne Olfman,et al.  Computer Support for Group Work: Perceptions of the Usefulness of Support Scenarios and End-User Tools , 1995, J. Manag. Inf. Syst..

[143]  J. B. Quinn,et al.  Information technology: Increasing productivity in services , 1994 .

[144]  L ChervanyNorman,et al.  Evaluating information systems effectiveness - part ii: , 1981 .

[145]  Mo Adam Mahmood,et al.  Measuring the Organizational Impact of Information Technology Investment: An Exploratory Study , 1993, J. Manag. Inf. Syst..

[146]  Sunder Kekre,et al.  Business Value of Information Technology: A Study of Electronic Data Interchange , 1995, MIS Q..

[147]  Sirkka L. Jarvenpaa,et al.  Computer Support for Meetings of Groups Working on Unstructured Problems: A Field Experiment , 1988, MIS Q..

[148]  R. Anthony,et al.  Planning and Control Systems: A Framework for Analysis , 1965 .

[149]  Jay F. Nunamaker,et al.  Information technology for negotiating groups: generating options for mutual gain , 1991 .