Strengthening ICT4D evaluation: lessons from the fields of program evaluation, IS/IT evaluation, and aid/development evaluation

ABSTRACT This study suggests ways to strengthen and clarify conceptual underpinnings of information and communication technology for development (ICT4D) evaluation by exploring its associations with other related evaluation fields – program evaluation, information system/information technology (IS/IT) evaluation, and aid/development evaluation. These fields have developed long and rich theoretical discourses, models, and approaches over the years, which are of interest to strengthen evaluation practices in ICT4D. Through detailed content analyses and expert interviews, we find that ICT4D evaluation shows significant parallels with the three fields in theory and practice. Also, we see that ICT4D evaluation has developed its own discourse – in particular, ICT4D researchers have valued capturing the situated development context of ICT. We argue that ICT4D evaluation can learn more from other evaluation traditions and disciplines, in order to help move the field to a more ‘mature’ stage.

[1]  Rudy Hirschheim,et al.  Analysing information systems evaluation: another look at an old problem , 1998 .

[2]  B. Peat,et al.  Process Evaluation of a County Drug Court , 2008 .

[3]  G. Kituyi,et al.  Antecedents and Dynamics for Strategic Alignment of Health Information Systems in Uganda , 2014, Electron. J. Inf. Syst. Dev. Ctries..

[4]  Roope Raisamo,et al.  A Model for Assessing Learning Management System Success in Higher Education in Sub‐Saharan Countries , 2014, Electron. J. Inf. Syst. Dev. Ctries..

[5]  C. D. Villiers,et al.  A Framework to Define the Impact of Sustainable ICT for Agriculture Projects: The Namibian Livestock Traceability System , 2017, Electron. J. Inf. Syst. Dev. Ctries..

[6]  Chrisanthi Avgerou,et al.  Theoretical Framing of ICT4D Research , 2017, ICT4D.

[7]  Staffan Jacobsson,et al.  Analyzing the functional dynamics of technological innovation systems: A scheme of analysis , 2008 .

[8]  Graham Pervan,et al.  A review of IS/IT investment evaluation and benefits management issues, problems, and processes , 2001 .

[9]  Bidit Lal Dey,et al.  Ethnographic Approach to User-Centred Evaluation of Telecentres , 2010, Int. J. Innov. Digit. Econ..

[10]  Peggy Shannon-Baker,et al.  Making Paradigms Meaningful in Mixed Methods Research , 2016 .

[11]  Sevgi Özkan,et al.  Evaluation of a Hospital Information System in an International Context: Towards Implementing PB‐ISM in Turkey , 2006, Electron. J. Inf. Syst. Dev. Ctries..

[12]  Jon M. Corbett,et al.  Empowerment and Participatory Geographic Information and Multimedia Systems: Observations from Two Communities in Indonesia , 2004 .

[13]  Subhash Bhatnagar,et al.  Assessing the Impact of E-government: A Study of Projects in India , 2010 .

[14]  Petri Hallikainen,et al.  A Holistic Framework on Information Systems Evaluation with a Case Analysis , 2007 .

[15]  G. Laverack,et al.  Theory of Change , 2019, Culturally Responsive Education in the Classroom.

[16]  Björn-Sören Gigler,et al.  Including the Excluded - Can ICTs Empower Poor Communities? Towards an Alternative Evaluation Framework Based on the Capability Approach , 2004 .

[17]  Sajda Qureshi,et al.  Field experiences with collaboration technology: A comparative study in Tanzania and South Africa , 2003 .

[18]  Mario Marais,et al.  Evaluating Users’ Perceptions of the Digital Doorway: A Narrative Analysis , 2015, Inf. Technol. Dev..

[19]  Melissa Tully,et al.  Investigating the Role of Innovation Attributes in the Adoption, Rejection, and Discontinued Use of Open Source Software for Development , 2015 .

[20]  Daniel Azerikatoa Ayoung,et al.  The Influence of Intangible (‘Soft’) Constructs on the Outcome of Community ICT Initiatives in Ghana: A Gap Archetype Analysis , 2016, Electron. J. Inf. Syst. Dev. Ctries..

[21]  David J. Grimshaw,et al.  Local Voices Enhance Knowledge Uptake: Sharing Local Content in Local Voices , 2010, Electron. J. Inf. Syst. Dev. Ctries..

[22]  Ricardo Gomez,et al.  Assessment of community wellness outcomes to measure ICT impact , 2013, ICTD.

[23]  Yingqin Zheng,et al.  Conceptualizing development in information and communication technology for development (ICT4D) , 2018, Inf. Technol. Dev..

[24]  Madhabi Chatterji.  Foundations of empowerment evaluation , 2001 .

[25]  Geoff Walsham,et al.  ICT4D research: reflections on history and future agenda , 2017, Inf. Technol. Dev..

[26]  S. Madon.  Evaluating the Developmental Impact of E‐governance Initiatives: An Exploratory Framework , 2004, Electron. J. Inf. Syst. Dev. Ctries..

[27]  Solomon Negash,et al.  Learning assessment of a videoconference-based training: lessons from medical training between USA and Ethiopia , 2010, Inf. Technol. Dev..

[28]  R. Gómez.  The Changing Field of ICTD: Growth and Maturation of the Field, 2000–2010 , 2013, Electron. J. Inf. Syst. Dev. Ctries..

[29]  Keng Siau,et al.  A survey of online e-banking retail initiatives , 2004, CACM.

[30]  Barbara S. Chaparro,et al.  A Balanced Usability Checklist Approach to Evaluate Palestinian Hotel Websites , 2007 .

[31]  Julian M. Bass,et al.  A Framework Using Institutional Analysis and the Capability Approach in ICT4D , 2013 .

[32]  Kenneth H. Blanchard,et al.  Great ideas revisited , 1996 .

[33]  M. Turshen.  Development as Freedom , 2001 .

[34]  Alemayehu Molla,et al.  ERP and Competitive Advantage in Developing Countries: The Case of an Asian Company , 2006, Electron. J. Inf. Syst. Dev. Ctries..

[35]  Dennis M. Ochieng,et al.  Exploring Digital Archive System to Develop Digitally Resilient Youths in Marginalised Communities in South Africa , 2017, Electron. J. Inf. Syst. Dev. Ctries..

[36]  Babajide Osatuyi,et al.  Examining Online Banking Initiatives in Nigeria: A Value Network Approach , 2009, Electron. J. Inf. Syst. Dev. Ctries..

[37]  Hosian Akeel,et al.  Information Systems Deployment in Libyan Oil Companies: Two Case Studies , 2013, Electron. J. Inf. Syst. Dev. Ctries..

[38]  Shweta Chopra,et al.  Modeling Intermediary Satisfaction with Mandatory Adoption of E-government Technologies for Food Distribution , 2016 .

[39]  Trina Gorman,et al.  Impact of Low-Cost, On-demand, Information Access in a Remote Ghanaian Village , 2010, 2011 IEEE Global Humanitarian Technology Conference.

[40]  John Yen,et al.  Introduction , 2004, CACM.

[41]  Mathias Hatakka,et al.  Increasing Interactivity in Distance Educations: Case Studies Bangladesh and Sri Lanka , 2010, Inf. Technol. Dev..

[42]  Mats Danielson,et al.  Using a Structured Approach to Evaluate ICT4D: Healthcare Delivery in Uganda , 2015, Electron. J. Inf. Syst. Dev. Ctries..

[43]  S. Revi Sterling,et al.  Considering failure: eight years of ITID research , 2012, ICTD.

[44]  Ephraim R. McLean,et al.  The DeLone and McLean Model of Information Systems Success: A Ten-Year Update , 2003, J. Manag. Inf. Syst..

[45]  Rob McLean,et al.  Alignment: Using the Balanced Scorecard to Create Corporate Synergies , 2006 .

[46]  Wanda J. Orlikowski,et al.  Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations , 2000, Theory in CSCW.

[47]  Gianluca Miscione,et al.  Telemedicine in the Upper Amazon: Interplay with Local Health Care Practices , 2007, MIS Q..

[48]  J. R. Ndiege,et al.  Quality Assessment of Information Systems in SMEs: A Study of Eldoret, Kenya , 2012, Electron. J. Inf. Syst. Dev. Ctries..

[49]  June Lennie,et al.  Evaluating Communication for Development: A Framework for Social Change , 2013 .

[50]  Sajda Qureshi.  The forgotten awaken: ICT’s evolving role in the roots of mass discontent , 2017, Inf. Technol. Dev..

[51]  V. J. Symons,et al.  A review of information systems evaluation: content, context and process , 1991 .

[52]  William Jobe,et al.  Putting a MOOC for Human Rights in the Hands of Kenyans , 2014, Electron. J. Inf. Syst. Dev. Ctries..

[53]  B. Mooketsi,et al.  Factors Influencing The Usage Of The Tribal Land Information Management System For Land Management And Administration: The Case Of Mogoditshane Subordinate Land‐Board , 2013, Electron. J. Inf. Syst. Dev. Ctries..

[54]  Donald L. Kirkpatrick,et al.  Seven keys to unlock the four levels of evaluation , 2006 .

[55]  Sharon F. Rallis,et al.  Qualitative and quantitative inquiries are not incompatible: A call for a new partnership , 1994 .

[56]  G. Vyver,et al.  An Electronic Commerce Initiative in Regional Sri Lanka: The Vision for the Central Province Electronic Commerce Portal , 2004, Electron. J. Inf. Syst. Dev. Ctries..

[57]  Steve Smithson,et al.  Information systems evaluation in practice: a case study of organizational change , 2000, J. Inf. Technol..

[58]  I. Brown,et al.  Introducing a Mobile Payment System to an Emerging Economy's Mobile Phone Subscriber Market. An Actor Network Perspective , 2014, Electron. J. Inf. Syst. Dev. Ctries..

[59]  K. Milis,et al.  The use of the balanced scorecard for the evaluation of Information and Communication Technology projects , 2004 .

[60]  Vassilis Serafeimidis.  A review of research issues in evaluation of information systems , 2001 .

[61]  Solomon Negash,et al.  IT Infrastructure Role in the Success of a Banking System: The Case of Limited Broadband Access , 2016, Inf. Technol. Dev..

[62]  Roger C. Riddell,et al.  Does Foreign Aid Really Work , 2007 .

[63]  William D. Tucker,et al.  Moving ICTD Research Beyond Bungee Jumping: Practical Case Studies and Recommendations , 2016, IEEE Technology and Society Magazine.

[64]  A.K.M Ganiul Zadid.  Information and communication technology for development (ICT4D) , 2005 .

[65]  L. Willcocks,et al.  Beyond the It Productivity Paradox , 1999 .

[66]  Paul R. Niven.  Balanced Scorecard Step-by-Step for Government and Nonprofit Agencies , 2003 .

[67]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[68]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[69]  K. Toyama,et al.  What Constitutes Good ICTD Research , 2009 .

[70]  Stefan Kuhlmann,et al.  Functions of innovation systems: A new approach for analysing technological change , 2007 .

[71]  Maria Elena Figueroa,et al.  Communication for social change: an integrated model for measuring the process and its outcomes. , 2002 .

[72]  Richard Heeks,et al.  Can a Process Approach Improve ICT4D Project Success , 2011 .

[73]  Frank F. Land,et al.  Moving IS evaluation forward: learning themes and research issues , 1999, J. Strateg. Inf. Syst..

[74]  Xingchen Song,et al.  Research on IT/IS Evaluation: A 25 Year Review , 2012 .

[75]  R. Gómez,et al.  Comparing Approaches: Telecentre Evaluation Experiences in Asia and Latin America , 2001, Electron. J. Inf. Syst. Dev. Ctries..

[76]  L. Linnan,et al.  Y CHAPTER ONE Process Evaluation for Public Health Interventions and Research An Overview , 2002 .

[77]  Stephen Flowerday,et al.  The Usability of IVRs for Smart City Crowdsourcing in Developing Cities , 2016, Electron. J. Inf. Syst. Dev. Ctries..

[78]  Dale Goodhue,et al.  Understanding user evaluations of information systems , 1995 .

[79]  Niels Dabelstein.  The Paris Declaration on aid Effectiveness: History and Significance , 2012, Canadian Journal of Program Evaluation.

[80]  Zahir Irani,et al.  Information systems evaluation: past, present and future , 2001, Eur. J. Inf. Syst..

[81]  Zahir Irani,et al.  Developing a frame of reference for ex-ante IT/IS investment evaluation , 2002, Eur. J. Inf. Syst..

[82]  Shoshana Zuboff,et al.  In the Age of the Smart Machine: The Future of Work and Power , 1989 .

[83]  Informing and Interacting with Citizens: A Strategic Communication Review of the Websites of the ECOWAS Parliaments , 2016 .

[84]  Salihu Ibrahim Dasuki,et al.  An Evaluation of Information Systems Students Internship Programs in Nigeria: A Capability Perspective , 2017, Electron. J. Inf. Syst. Dev. Ctries..

[85]  W. Chigona,et al.  Different Shades of Success: Educator Perceptions of Government Strategy on E‐Education in South Africa , 2014, Electron. J. Inf. Syst. Dev. Ctries..

[86]  สาธิตา สกุลรัตนกุลชัย,et al.  Empowerment Evaluation Principles in Practice , 2015 .

[87]  John C. Henderson,et al.  Strategic Alignment: Leveraging Information Technology for Transforming Organizations , 1999, IBM Syst. J..

[88]  Murodillo Abdusamadovich Latifov,et al.  Data Warehouse Approach to Strengthen Actionability of Health Information Systems: Experiences from Tajikistan , 2012, Electron. J. Inf. Syst. Dev. Ctries..

[89]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[90]  Julian M. Bass,et al.  Changing Computing Curricula in African Universities: Evaluating Progress and Challenges via Design‐Reality Gap Analysis , 2011, Electron. J. Inf. Syst. Dev. Ctries..

[91]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[92]  S. Jones.  Chapter 12 – Social dimension of IT/IS evaluation: Views from the public sector , 2008 .

[93]  Sarmad Alshawi,et al.  E-government evaluation: Citizen's perspective in developing countries , 2009, Inf. Technol. Dev..

[94]  J. Ward,et al.  Evaluation and realisation of IS/IT benefits: an empirical study of current practice , 1996 .

[95]  Semih Bilgen,et al.  Modelling and Assessment of the Effectiveness of Government Information Technologies Value Space Approach with a Public Sector Case Study in Turkey , 2011, Electron. J. Inf. Syst. Dev. Ctries..

[96]  Edim Azom Emmanuel,et al.  A User Interface for Micro‐Entrepreneurs in a Rural Community , 2010, Electron. J. Inf. Syst. Dev. Ctries..

[97]  H. White,et al.  A Contribution to Current Debates in Impact Evaluation , 2010 .

[98]  Jean-Paul Van Belle,et al.  A Critical Evaluation of Past IDIA Research: Lessons Learnt for IDIA and ICT4D Researchers , 2015 .

[99]  Christian Wagner,et al.  The Relative Importance of Monetary and Non-Monetary Drivers for Information and Communication Technology Acceptance in Rural Agribusiness† , 2016, Inf. Technol. Dev..

[100]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[101]  Caroline Pade-Khene,et al.  Critical Themes of Process Assessment in Rural ICT4D Projects: An Analysis of Assessment Approaches , 2014, Electron. J. Inf. Syst. Dev. Ctries..

[102]  Donna M. Mertens,et al.  Program Evaluation Theory and Practice: A Comprehensive Guide , 2012 .

[103]  G. Walsham,et al.  The evaluation of information systems: a critique , 1991 .

[104]  Øystein Sæbø,et al.  A Framework for Understanding the Link Between ICT and Development: How Affordances Influence Capabilities , 2016 .

[105]  Chris Westrup,et al.  Context and the processes of ICT for development , 2012, Inf. Organ..

[106]  Annastellah Sigwejo,et al.  A Citizen‐Centric Framework For Assessing E‐Government Effectiveness , 2016, Electron. J. Inf. Syst. Dev. Ctries..

[107]  Asmae El Mahdi,et al.  E‐Government and local Good Governance: A Pilot Project in Fez, Morocco , 2008, Electron. J. Inf. Syst. Dev. Ctries..

[108]  Sabine Moisan,et al.  Generating Knowledge-Based System Generators: A Software Engineering Approach , 2010, Int. J. Intell. Inf. Technol..

[109]  Francesca De Chiara,et al.  ICTs and monitoring of MDGs: a case study of Kenya HIV/AIDS monitoring and evaluation in a donor multi-agency context , 2011, Inf. Technol. Dev..

[110]  Paz Olivera,et al.  Institutional Barriers to Development Innovation: Assessing the Implementation of XO-1 Computers in Two Peri-Urban Schools in Peru (English) , 2012 .

[111]  K. Vincent,et al.  “Ten Seeds”: How Mobiles Have Contributed to Development in Women-Led Farming Cooperatives in Lesotho , 2013 .

[112]  Jaime H. Nunez,et al.  On the applicability of a computer model for business performance analysis in SMEs: A case study from Chile , 2000 .

[113]  R. Kaplan,et al.  Using the balanced scorecard as a strategic management system , 1996 .

[114]  Egon Berghout,et al.  Methodologies for investment evaluation: a review and assessment , 2001 .

[115]  J. Tacchi,et al.  A holistic, learning-centred approach to building evaluation capacity in development organizations , 2015 .

[116]  Julian C. Jamison,et al.  Mixed Method Evaluation of a Passive Mhealth Sexual Information Texting Service in Uganda , 2013 .

[117]  Leslie P. Willcocks,et al.  Measuring organizational IS effectiveness: an overview and update of senior management perspectives , 2002, DATB.

[118]  K. Hupfeld,et al.  Program evaluation. , 1989, NLN publications.

[119]  W. V. Grembergen.  Information Technology Evaluation Methods and Management , 2000 .

[120]  R. Gómez,et al.  ICT Evaluation: Are We Asking the Right Questions? , 2012 .

[121]  S. Bailur.  Using Stakeholder Theory to Analyze Telecenter Projects , 2007 .

[122]  Sumeet Gupta,et al.  A comprehensive four-stage framework for evaluation of Information Communication Technologies for Development interventions , 2018, Inf. Technol. Dev..

[123]  Kelvin Joseph Bwalya,et al.  Factors Affecting Adoption of e‐Government in Zambia , 2009, Electron. J. Inf. Syst. Dev. Ctries..

[124]  Donald L. Kirkpatrick,et al.  Great Ideas Revisited. Techniques for Evaluating Training Programs. Revisiting Kirkpatrick's Four-Level Model. , 1996 .

[125]  Stephen R. Ruth,et al.  Measuring long term effects of technology transfer in developing nations: The case of internet training at the Romanian academy of science , 2000 .

[126]  E. House,et al.  Deliberative Democratic Evaluation in Practice , 2000 .

[127]  Lynn Westbrook,et al.  Utilization-focused evaluation , 1998 .

[128]  Fred Carden,et al.  Outcome mapping : building learning and reflection into development programs [Arabic version] , 2002 .

[129]  J. Soar,et al.  Evaluating the Anti‐Corruption Capabilities of Public e‐Procurement in a Developing Country , 2012, Electron. J. Inf. Syst. Dev. Ctries..

[130]  Hafeni Mthoko,et al.  Building theory in ICT4D evaluation: a comprehensive approach to assessing outcome and impact* , 2018, Inf. Technol. Dev..

[131]  S. Verkijika,et al.  Quality assessment of e‐government websites in Sub‐Saharan Africa: A public values perspective , 2018, Electron. J. Inf. Syst. Dev. Ctries..

[132]  R. Picciotto.  Experimentalism and development evaluation: Will the bubble burst? , 2012 .

[133]  Richard Heeks,et al.  Impact Assessment of ICT-for-Development Projects: A Compendium of Approaches , 2009 .

[134]  Erwin Fielt,et al.  A Comparative Study of Governmental One-Stop Portals for Public Service Delivery , 2013, Int. J. Intell. Inf. Technol..

[135]  Naomi Isabirye,et al.  A Framework to Guide Development Through ICTs in Rural Areas in South Africa , 2015, Inf. Technol. Dev..

[136]  Kentaro Toyama,et al.  From needs to aspirations in information technology for development , 2018, Inf. Technol. Dev..

[137]  Alta de Waal,et al.  Morphological analysis: a method for selecting ICT applications in South African government service delivery , 2010 .

[138]  Wallace Chigona,et al.  mHealth outcomes for pregnant mothers in Malawi: a capability perspective , 2018, Inf. Technol. Dev..

[139]  Donovan A. McFarlane,et al.  Evaluating Training Programs: The Four Levels , 2006 .

[140]  Michael Scriven,et al.  Evaluation Models: Viewpoints on Educational and Human Services Evaluation , 2000 .

[141]  Sebastián Benítez Larghi,et al.  Digital and Social Inequalities: A Qualitative Assessment of the Impact of the Connecting Equality Program on Argentinean Youth , 2015, Electron. J. Inf. Syst. Dev. Ctries..

[142]  Caroline Pade-Khene,et al.  The Rural ICT Comprehensive Evaluation Framework: Implementing the First Domain, the Baseline Study Process , 2012, Electron. J. Inf. Syst. Dev. Ctries..

[143]  Jeanne Century,et al.  A Framework for Measuring Fidelity of Implementation: A Foundation for Shared Language and Accumulation of Knowledge , 2010 .

[144]  Maung K. Sein,et al.  Conceptualizing the ICT Artifact: Toward Understanding the Role of ICT in National Development , 2004, Inf. Soc..

[145]  S. Zionts,et al.  An Interactive Programming Method for Solving the Multiple Criteria Problem , 1976 .

[146]  Akemi Takeoka Chatfield,et al.  A cross-country comparative analysis of e-government service delivery among Arab countries , 2009, Inf. Technol. Dev..

[147]  David A. Sewry,et al.  An Elaboration of Critical Success Factors for Rural ICT Project Sustainability in Developing Countries: Exploring the DWESA Case , 2008 .

[148]  Frances Wood,et al.  An investigation of the impact of information and communication technologies in sub-Saharan Africa , 1999, J. Inf. Sci..

[149]  Julia Bello-Bravo,et al.  An assessment of learning gains from educational animated videos versus traditional extension presentations among farmers in Benin* , 2018, Inf. Technol. Dev..

[150]  H. Canton.  United Nations Development Programme—UNDP , 2018, The Europa Directory of International Organizations 2021.

[151]  Pamela Y. Abbott,et al.  A Socio‐Technical Analysis of ICT Investments in Developing Countries: A Capability Perspective , 2015, Electron. J. Inf. Syst. Dev. Ctries..

[152]  Dorothea Kleine,et al.  Technologies of Choice? ICTs, Development and the Capabilities Approach , 2014 .

[153]  Simon Batchelor,et al.  Framework For The Assessment Of ICT Pilot Projects Beyond Monitoring and Evaluation To Applied Research , 2007 .

[154]  P. M. Alexander,et al.  A scheme of analysis for eVoting as a technological innovation system , 2018, Electron. J. Inf. Syst. Dev. Ctries..

[155]  R. Picciotto.  Introduction: Evaluation and development , 1995 .

[156]  Narasimhaiah Gorla.  A Survey of rural e-Government projects in India: Status and benefits , 2009, Inf. Technol. Dev..

[157]  Richard Heeks,et al.  An ICT4D Journal Ranking Table , 2010 .

[158]  W. Orlikowski.  Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations , 2000 .

[159]  Darshana Sedera,et al.  Re-conceptualizing Information System Success: The IS-Impact Measurement Model , 2008, J. Assoc. Inf. Syst..

[160]  S. Bhatnagar,et al.  Social implications of computers in developing countries , 1992 .