Survey design: Insights from a public sector-ERP success study

Research on the impact of Information Systems (IS) reported in both academic literature and popular press has reported confounding results. Some studies have reported encouraging results of IS, while others have reported nil or detrimental results. The contradictory results of these research studies can be partially attributed to the weaknesses in survey instruments. In an attempt to increase the validity of conclusions of IS assessment studies, survey instrument design should follow a rigorous and scientific procedure. This paper illustrates key validity and reliability issues in measuring Information Systems performance, using examples from a study designed to assess Enterprise Resource Planning systems success. The article emphasizes on the importance of the survey method and the theoretical considerations of item derivation, scale development and item evaluation. Examples are provided from the ERP assessment study to supplement the readers understanding of the theoretical concepts of survey design.

[1]  Henri Barki,et al.  Explaining the Role of User Participation in Information System Use , 1994 .

[2]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[3]  Bill Lehr,et al.  Information technology and its impact on firm-level productivity: evidence from government and private data sources, 1977-1993 , 1999 .

[4]  Kenneth L. Kraemer,et al.  The Usefulness of Computer-Based Information to Public Managers , 1993, MIS Q..

[5]  Diane M. Strong,et al.  Beyond Accuracy: What Data Quality Means to Data Consumers , 1996, J. Manag. Inf. Syst..

[6]  Jane M. Howell,et al.  Personal Computing: Toward a Conceptual Model of Utilization , 1991, MIS Q..

[7]  Maarten Gelderman,et al.  The relation between user satisfaction, usage of information systems and performance , 1998, Inf. Manag..

[8]  Peter B. Seddon,et al.  A Partial Test and Development of the DeLone and McLean Model of IS Success , 1994, ICIS.

[9]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1973 .

[10]  Richard Harris,et al.  SMIS Members: A Membership Analysis , 1982, MIS Q..

[11]  Ingoo Han,et al.  Performance measure of information systems (IS) in evolving computing environments: an empirical investigation , 2003, Inf. Manag..

[12]  Erik Brynjolfsson,et al.  The productivity paradox of information technology , 1993, CACM.

[13]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[14]  Peter B. Seddon,et al.  A Comprehensive Framework for Classifying the Benefits of ERP Systems , 2000 .

[15]  Michael Rosemann,et al.  Australasian ( ACIS ) 2001 A Balanced Scorecard Approach to Enterprise Systems Performance Measurement , 2001 .

[16]  William J. Doll,et al.  A Confirmatory Factor Analysis of the End-User Computing Satisfaction Instrument , 1994, MIS Q..

[17]  R. Kelly Rainer,et al.  The Keys to Executive Information Systems Success , 1995, J. Manag. Inf. Syst..

[18]  Charles H. Kriebel,et al.  Information Technologies and Business Value: An Analytic and Empirical Investigation , 1995, Inf. Syst. Res..

[19]  K. Cameron,et al.  Organizational paradox and transformation. , 1988 .

[20]  James C. Wetherbe,et al.  Key issues in information systems management , 1987 .

[21]  Ali F. Farhoomand,et al.  A structural model of end user computing satisfaction and user performance , 1996, Inf. Manag..

[22]  J. Miller,et al.  Measuring the Effectiveness of Computer-Based Information Systems in the Financial Services Sector , 1987, MIS Q..

[23]  Timo Saarinen,et al.  An expanded instrument for evaluating information system success , 1996, Inf. Manag..

[24]  Annette L. du Plessis,et al.  A method for CASE tool evaluation , 1993, Inf. Manag..

[25]  Carol Stoak Saunders,et al.  Measuring Performance of the Information Systems Function , 1992, J. Manag. Inf. Syst..

[26]  Detmar W. Straub,et al.  Measuring System Usage: Implications for IS Theory Testing , 1995 .

[27]  Houston H. Carr,et al.  Factors that affect user-friendliness in interactive computer programs , 1992, Inf. Manag..

[28]  Justus D. Naumann,et al.  A Selection Model for Systems Development Tools , 1982, MIS Q..

[29]  Leon A. Kappelman,et al.  A Comprehensive Model for Assessing the Quality and Productivity of the Information Systems Function: Toward a Theory for Information Systems Assessment , 1997 .

[30]  Dale Goodhue,et al.  Task-Technology Fit and Individual Performance , 1995, MIS Q..

[31]  Peter Weill,et al.  Assessing The Health of An Information Systems Applications Portfolio: An Example From Process Manufacturing , 1999, MIS Q..

[32]  Gerald V. Post,et al.  A Structural Equation Evaluation of CASE Tools Attributes , 1999, J. Manag. Inf. Syst..

[33]  D. Sandy Staples,et al.  Having expectations of information systems benefits that match received benefits: does it really matter? , 2002, Inf. Manag..

[34]  Michael Rosemann,et al.  Using performance measurement models for benefit realization with enterprise systems the Queensland government approach , 2001, ECIS.

[35]  R. Kaplan,et al.  Having trouble with your strategy? Then map it. , 2000, Harvard business review.

[36]  S. Miller,et al.  Productivity growth in large US commercial banks: The initial post-deregulation experience , 2001 .

[37]  M. Morton,et al.  The corporation of the 1990s: Information technology and organizational transformation , 1993 .

[38]  Howard B. Lee,et al.  Foundations of Behavioral Research , 1965 .

[39]  Diane Wilson.  Assessing the impact of information technology on organizational performance , 1993 .

[40]  Michael Ishman,et al.  Measuring Information Success at the Individual Level in Cross-Cultural Environments , 1996 .

[41]  Erik Brynjolfsson,et al.  Information Technology and Productivity: A Review of the Literature , 1996, Adv. Comput..

[42]  Vijay Sethi,et al.  Development of measures to assess the extent to which an information technology application provides competitive advantage , 1994 .

[43]  Prashant C. Palvia,et al.  A model and instrument for measuring small business user satisfaction with information technology , 1996, Inf. Manag..

[44]  William J. Doll,et al.  The Measurement of End-User Computing Satisfaction , 1988, MIS Q..

[45]  John Rohrbaugh,et al.  A Spatial Model of Effectiveness Criteria: Towards a Competing Values Approach to Organizational Analysis , 1983 .

[46]  E. Brynjolfsson,et al.  Paradox Lost? Firm-Level Evidence on the Returns to Information Systems Spending , 1996 .

[47]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[48]  Detmar W. Straub,et al.  Validating Instruments in MIS Research , 1989, MIS Q..

[49]  A. Barua,et al.  The Information Technology Productivity Paradox Revisited: A Theoretical and Empirical Investigation in the Manufacturing Sector , 1997 .

[50]  Richard O. Mason,et al.  Measuring information output: A communication systems approach , 1978, Inf. Manag..

[51]  K. Cameron,et al.  Paradox and transformation : toward a theory of change in organization and management , 1990 .

[52]  C. R. Franz,et al.  ORGANIZATIONAL CONTEXT, USER INVOLVEMENT, AND THE USEFULNESS OF INFORMATION SYSTEMS* , 1986 .

[53]  James C. Wetherbe,et al.  Key Information Systems Issues for the 1980's , 1984, MIS Q..

[54]  Sammy W. Pearson,et al.  Development of a Tool for Measuring and Analyzing Computer User Satisfaction , 1983 .

[55]  R. Kaplan,et al.  The balanced scorecard--measures that drive performance. , 2015, Harvard business review.

[56]  Magid Igbaria,et al.  Analysis of Information Technology Success in Small Firms in New Zealand , 1998 .

[57]  Rick H. Hoyle,et al.  Confirmatory Factor Analysis , 1983 .