IT Project Evaluation: Is More Formal Evaluation Necessarily Better?

This paper explores effective IT project evaluation practices in Australia among 36 companies operating in three industry sectors: Finance and Insurance; Mining; and Electricity, Gas and Water Supply. Qualitative analysis of 78 interviews was used to determine characteristics of effective evaluation practices. The study suggests that more formal and sophisticated evaluation methods do not necessarily result in more effective evaluation or better IT project outcomes. Rather, effective IT project evaluation outcomes are more closely related to the drivers of behaviour in organisations such as accountability for results, top-leadership commitment, a strong IT-Business relationship, alignment to strategy, an effective measurement regime, and a willingness to take action. Simplicity and flexibility were also key characteristics underlying effective practices. These conclusions provide both theoretical and practical insights into the evaluation of IT projects in complex environments.

[1]  Rudy Hirschheim,et al.  Analysing information systems evaluation: another look at an old problem , 1998 .

[2]  Rens Scheepers,et al.  Contexts of Relevance in Explanatory Case Studies in Information Systems: Ubiquitous Information Technology Implementation in Organizations , 2003, ICIS.

[3]  A. Strauss,et al.  The Discovery of Grounded Theory , 1967 .

[4]  Michael Sherwood-Smith,et al.  Maximise information systems value by continuous participative evaluation , 1999 .

[5]  Zahir Irani,et al.  Linking knowledge transformation to Information Systems evaluation , 2005, Eur. J. Inf. Syst..

[6]  M. Patton,et al.  Qualitative evaluation and research methods , 1992 .

[7]  Judy McKay,et al.  Strategic Management of E-Business , 2004 .

[8]  Alan Blair.  Social Research , 1935, Nature.

[9]  Zahir Irani,et al.  Developing a frame of reference for ex-ante IT/IS investment evaluation , 2002, Eur. J. Inf. Syst..

[10]  Frank Bannister,et al.  The effective measurement and management of ICT costs and benefits. 3rd edition. , 2007 .

[11]  Chad Lin,et al.  The practice of IS/IT benefits management in large Australian organizations , 2003, Inf. Manag..

[12]  M. Patton.  Qualitative evaluation and research methods, 2nd ed. , 1990 .

[13]  Leslie P. Willcocks,et al.  Measuring organizational IS effectiveness: an overview and update of senior management perspectives , 2002, DATB.

[14]  Quinn McNemar,et al.  Social research (2nd Ed.). , 1943 .

[15]  Joan A. Ballantine,et al.  Information systems/technology evaluation practices: evidence from UK organizations , 1996, J. Inf. Technol..

[16]  Joan A. Ballantine,et al.  Financial appraisal and the IS/IT investment decision making process , 1998, J. Inf. Technol..

[17]  Zahir Irani,et al.  Information technology evaluation: classifying indirect costs using the structured case method , 2004, J. Enterp. Inf. Manag..

[18]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[19]  Leslie P. Willcocks,et al.  Evaluating Information Technology investments: research findings and reappraisal , 1992, Inf. Syst. J..

[20]  Chad Lin,et al.  IS/IT Investment Evaluation and Benefits Realization Issues in Australia , 2005, J. Res. Pract. Inf. Technol..

[21]  B. Hochstrasser.  Justifying IT investments , 1994 .

[22]  Dan Remenyi,et al.  Management of information technology evaluation ‐ the development of a managerial thesis , 1999 .

[23]  Zahir Irani,et al.  Information systems evaluation: past, present and future , 2001, Eur. J. Inf. Syst..

[24]  Chad Lin,et al.  The enigma of evaluation: benefits, costs and risks of IT in Australian small-medium-sized enterprises , 2005, Inf. Manag..

[25]  John M. Ward,et al.  A portfolio approach to evaluating information systems investments and setting priorities , 1990, J. Inf. Technol..

[26]  Zahir Irani,et al.  Evaluating information technology in dynamic environments: a focus on tailorable information systems , 1999 .

[27]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[28]  KA Thleen,et al.  Building Theories from Case Study , 2007 .

[29]  Z. Irani,et al.  Benchmarking information technology investment and benefits extraction , 2003 .

[30]  Frank F. Land,et al.  Evaluating investments in IT , 1992, J. Inf. Technol..

[31]  Amrik S Sohal,et al.  The role and impact of information technology in Australian business , 1998, J. Inf. Technol..

[32]  Kenneth L. Kraemer,et al.  Executives’ Perceptions of the Business Value of Information Technology: A Process-Oriented Approach , 2000, J. Manag. Inf. Syst..