A Monitoring and Evaluation Scheme for an ICT-Supported Education Program in Schools

More than 20 years after ICTs were introduced in schools, solid evidence of their impact on student attainment is still lacking. Reasons for this include the mismatch between the methods used to measure the effects and the type of learning promoted, the absence of information regarding the specific types of ICT used, and the scarce attention paid to the monitoring and evaluation of ICT for Education (ICT4E) programs. A monitoring and evaluation scheme would provide qualitative and quantitative data to refine, adjust and improve an ICT4E project, to learn from the experience gained, and to determine whether the program has served its client communities and how it might be replicated. In this paper we present a monitoring and evaluation (M&E) scheme for a specific ICT4E program that supports teaching and learning using mobile computer supported collaborative learning (MCSCL). Using the information provided by the scheme, we analyze the program’s impact on student attainment in terms of teacher adoption of innovation. It was found that there were statistically significant positive differences in students whose teachers showed higher adoption levels when compared both to lower adoption cases and other defined control groups. We conclude that an M&E scheme supports the intervention process by providing real-time information for decision making through the application of assessment instruments according to a monitoring plan. This enables intervention activities to be adjusted so as to ensure an adequate level of adoption.

[1]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[2]  Camila Cortez,et al.  Teachers' support with ad-hoc collaborative networks , 2005, J. Comput. Assist. Learn..

[3]  Chris L. S. Coryn,et al.  Evaluation Theory, Models, and Applications , 2007 .

[4]  John Robertson,et al.  The ambiguous embrace: twenty years of IT (ICT) in UK primary schools , 2002, Br. J. Educ. Technol..

[5]  Roberto Agodini,et al.  The Effectiveness of Educational Technology: Issues and Recommendations for the National Study. , 2003 .

[6]  Michael Trucano,et al.  Knowledge Maps: ICTs in Education , 2005 .

[7]  Ragnar Andreas Audunson,et al.  Technology and Social Inclusion. Rethinking the Digital Divide , 2006, J. Documentation.

[8]  Debra N. A. Hayes,et al.  ICT and learning: Lessons from Australian classrooms , 2007, Comput. Educ..

[9]  William R. Penuel,et al.  Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort , 2007 .

[10]  Miguel Nussbaum,et al.  From handheld collaborative tool to effective classroom module: embedding CSCL in a broader design framework , 2009 .

[11]  Herbert J. Walberg,et al.  Pretest and Sensitization Effects in Curriculum Evaluation , 1970 .

[12]  Willem J. Pelgrum,et al.  Obstacles to the integration of ICT in education: results from a worldwide educational assessment , 2001, Comput. Educ..

[13]  J. Voogt,et al.  International handbook of information technology in primary and secondary education , 2008 .

[14]  Rae Condie,et al.  The impact of ICT in schools - a landscape review , 2007 .

[15]  Margaret J. Cox,et al.  Effects of ICT: Do we know what we should know? , 2007, Education and Information Technologies.

[16]  Miguel Nussbaum,et al.  Dynamic Grouping in Collaborative Learning Supported by Wireless Handhelds , 2005, J. Educ. Technol. Soc..

[17]  José Jorge Gil Pérez,et al.  Las Tecnologías de la Información y la Comunicación en la Educación , 2007 .

[18]  Mary Simpson,et al.  Silicon Valley versus Silicon Glen: the impact of computers upon teaching and learning: a comparative study , 2003, Br. J. Educ. Technol..

[19]  Roberto Agodini,et al.  Effectiveness of Reading and Mathematics Software Products: Findings from Two Student Cohorts. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education , 2009 .

[20]  Miguel Nussbaum,et al.  Using multiple choice questions as a pedagogic model for face‐to‐face CSCL , 2009, Comput. Appl. Eng. Educ..

[21]  Roberto Agodini,et al.  Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts. NCEE 2009-4041. , 2009 .

[22]  Miguel Nussbaum,et al.  Technology as small group face-to-face Collaborative Scaffolding , 2009, Comput. Educ..

[23]  Daniel Light,et al.  Evaluating Educational Technology Interventions : How do we know its working , 2008 .

[24]  Tony Fisher,et al.  ImpaCT2: the impact of information and communication technologies on pupil learning and attainment , 2002 .

[25]  David Reynolds,et al.  ICT - the hopes and the reality , 2003, Br. J. Educ. Technol..

[26]  Miguel Nussbaum,et al.  A conceptual framework based on Activity Theory for mobile CSCL , 2007, Br. J. Educ. Technol..

[27]  Alfred P. Rovai,et al.  A practical framework for evaluating online distance education programs , 2003, Internet High. Educ..

[28]  Jonathan Miller,et al.  Monitoring and Evaluation of ICT in Education Projects A Handbook for Developing Countries , 2005 .

[29]  Miguel Nussbaum,et al.  Computer supported collaborative learning using wirelessly interconnected handheld computers , 2004, Comput. Educ..

[30]  R. Kozma.  Comparative Analysis of Policies for ICT in Education , 2008 .

[31]  Simon Batchelor,et al.  Framework For The Assessment Of ICT Pilot Projects Beyond Monitoring and Evaluation To Applied Research , 2007 .

[32]  Guillermo Sunkel,et al.  Las tecnologías de la información y la comunicación (TIC) en educación en América Latina: una exploración de indicadores , 2006 .

[33]  Miguel Nussbaum,et al.  An experimental study of the inclusion of technology in higher education , 2009, Comput. Appl. Eng. Educ..

[34]  Ina V. S. Mullis,et al.  Findings from IEA's Trends in International Mathematics and Science Study at the Fourth and Eighth Grades. TIMSS 2003 International Science Report. , 2004 .

[35]  Miguel Nussbaum,et al.  A constructivist mobile learning environment supported by a wireless handheld network , 2004, J. Comput. Assist. Learn..