Pilot Implementation: Learning from Field Tests in IS Development

A recurrent problem in information-systems development (ISD) is that many design shortcomings are not detected during development, but first after the system has been delivered and implemented in its intended environment. Pilot implementations appear to promise a way to extend prototyping from the laboratory to the field, thereby allowing users to experience a system design under realistic conditions and developers to get feedback from realistic use while the design is still malleable. We characterize pilot implementation, contrast it with prototyping, propose a five-element model of pilot implementation, and provide three empirical illustrations of our model. We conclude that pilot implementation has much merit as an ISD technique when system performance is contingent on context. But we also warn developers that, despite their seductive conceptual simplicity, pilot implementations can be difficult to plan and conduct. It is sometimes assumed that pilot implementations are less complicated and risky than ordinary implementations. Pilot implementations are, however, neither prototyping nor small-scale versions of full-scale implementations; they are fundamentally different and have their own challenges, which will be enumerated and discussed in this article.

[1]  Steven L. Alter Which Life Cycle - Work System, Information System, or Software? , 2001, Commun. Assoc. Inf. Syst..

[2]  Morten Hertzum,et al.  Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design , 2006, Int. J. Hum. Comput. Interact..

[3]  Claus Bossen,et al.  Test the artefact - Develop the organization: The implementation of an electronic medication plan , 2007, Int. J. Medical Informatics.

[4]  Pekka Abrahamsson,et al.  New directions on agile methods: a comparative analysis , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[5]  C. Floyd A Systematic Look at Prototyping , 1984 .

[6]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[7]  Monique W. M. Jaspers,et al.  The significance of a usability evaluation of an emerging laboratory order entry system , 2007, Int. J. Medical Informatics.

[8]  Arijit Sengupta,et al.  Role of Pilot Study in Assessing Viability of New Technology Projects: The Case of RFID in Parking Operations , 2008, Commun. Assoc. Inf. Syst..

[9]  Jorgen P. Bansler,et al.  Pilot Implementation of Health Information Systems: Issues and Challenges , 2010, AMCIS.

[10]  J. Rodney Turner,et al.  The role of pilot studies in reducing risk on projects and programmes , 2005 .

[11]  Valdis Berzins,et al.  Software prototyping , 2003 .

[12]  Jesper Simonsen,et al.  Positive effects of electronic patient records on three clinical activities , 2008, Int. J. Medical Informatics.

[13]  Mark T True,et al.  Software Requirements , 2005 .

[14]  Frederick P. Brooks,et al.  No Silver Bullet: Essence and Accidents of Software Engineering , 1987 .

[15]  Wanda J. Orlikowski,et al.  Improvising Organizational Transformation Over Time: A Situated Change Perspective , 1996, Inf. Syst. Res..

[16]  Barry W. Boehm,et al.  Get Ready for Agile Methods, with Care , 2002, Computer.

[17]  Jonathon Gray,et al.  Telegenetics: A Pilot Study of Video-Mediated Genetic Consultations in Wales: , 2002 .

[18]  Daniel D. McCracken,et al.  Life cycle concept considered harmful , 1982, ACM SIGSOFT Softw. Eng. Notes.

[19]  George Rzevski,et al.  Prototypes Versus Pilot Systems: Strategies for Evolutionary Information System Development , 1984 .

[20]  Chad Lin,et al.  The practice of IS/IT benefits management in large Australian organizations , 2003, Inf. Manag..

[21]  Jesper Simonsen,et al.  Iterative participatory design , 2010 .

[22]  Reed M. Gardner,et al.  Strategic Information Management in Hospitals: An Introduction to Hospital Information Systems , 2004 .

[23]  Marius A. Janson,et al.  Applying a pilot system and prototyping approach to systems development and implementation , 1986, Inf. Manag..

[24]  Alan M. Davis,et al.  Software Prototyping , 1995, Adv. Comput..

[25]  Lynda M. Applegate,et al.  Corporate information strategy and management , 2003 .

[26]  Erik Stolterman,et al.  The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas , 2008, TCHI.

[27]  Jaak Jurison,et al.  The temporal nature of IS benefits: A longitudinal study , 1996, Inf. Manag..

[28]  Volker Wulf,et al.  Towards an integrated organization and technology development , 1995, Symposium on Designing Interactive Systems.

[29]  Victor R. Basili,et al.  Iterative enhancement: A practical technique for software development , 1975, IEEE Transactions on Software Engineering.

[30]  Andraž Cej,et al.  Agile software development with Scrum , 2010 .

[31]  Robert L. Glass,et al.  Pilot Studies: What, Why, and How , 1997, J. Syst. Softw..

[32]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[33]  Reinhard Budde,et al.  Prototyping: An Approach to Evolutionary System Development , 1992, Springer Berlin Heidelberg.

[34]  Elizabeth Daniel,et al.  Benefits management : delivering value from IS & IT investments , 2005 .

[35]  Peter Constable,et al.  DSDM: Dynamic Systems Development Method: The Method in Practice , 1997 .

[36]  Barry W. Boehm,et al.  Requirements that Handle IKIWISI, COTS, and Rapid Change , 2000, Computer.

[37]  P. Williamson,et al.  Design and analysis of pilot studies: recommendations for good practice. , 2004, Journal of evaluation in clinical practice.

[38]  Karl E. Wiegers,et al.  Software Requirements , 1999 .

[39]  David R. Firth,et al.  Communications of the Association for Information Systems , 2011 .

[40]  M. Sobol,et al.  Barriers to the adoption of computerized technology in health care systems. , 1999, Topics in health information management.

[41]  Michael L. Brodie,et al.  Final report of the ANSI/X3/SPARC DBS-SG relational database task group , 1982, SGMD.

[42]  M. A. Janson,et al.  Managing the information systems development process: the case for prototype and pilot systems , 1990 .

[43]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[44]  M. Lynne Markus,et al.  Technochange management: using IT to drive organizational change , 2004, J. Inf. Technol..

[45]  JurisonJaak The temporal nature of IS benefits , 1996 .

[46]  Horst Lichter,et al.  Prototyping in industrial software projects—bridging the gap between theory and practice , 1994, ICSE '93.

[47]  A. Adam Whatever happened to information systems ethics? Caught between the devil and the deep blue sea , 2004 .

[48]  Laurie A. Williams,et al.  Agile software development: it's about feedback and change , 2003, Computer.

[49]  Victor R. Basili,et al.  Iterative and incremental developments. a brief history , 2003, Computer.

[50]  Gary E. Reed,et al.  Information technology: changing the face of automotive retailing? , 2004 .

[51]  Cynthia LeRouge,et al.  Healthcare information systems research, revelations and visions , 2007 .

[52]  Judy E. Scott,et al.  Implementing Enterprise Resource Planning Systems: The Role of Learning from Failure , 2000, Inf. Syst. Frontiers.

[53]  A. Boonstra Information systems and organizational change , 2002 .

[54]  J. Ward,et al.  Evaluation and realisation of IS/IT benefits: an empirical study of current practice , 1996 .

[55]  Liming Zhu,et al.  An empirical study of groupware support for distributed software architecture evaluation process , 2004, J. Syst. Softw..

[56]  Vanora Hundley,et al.  Pilot studies in family planning and reproductive health care , 2005, Journal of Family Planning and Reproductive Health Care.

[57]  Yajiong Xue,et al.  Web-based intervention support system for health promotion , 2006, Decis. Support Syst..

[58]  Mark Keil,et al.  The user–developer communication process: a critical case study , 2003, Inf. Syst. J..

[59]  Paul Beynon-Davies,et al.  Information systems prototyping in practice , 1999, J. Inf. Technol..

[60]  S. C. Holmberg,et al.  Effect specifications as an Alternative to Use Cases , 2008 .