Monitoring and evaluating astronomy outreach programmes: Challenges and solutions

A number of tools exist to guide the monitoring and evaluation of science, technology, engineering and mathematics (STEM) education and outreach programmes. Fewer tools exist for evaluating astronomy outreach programmes. In this paper we try to overcome this limitation by presenting a monitoring and evaluation framework developed for the International Astronomical Union's Office of Astronomy for Development (OAD). The mandate of the OAD is to stimulate sustainable development at an international level and to expand astronomy education and outreach globally. The broad assumptions of this programme are that astronomy has the potential to contribute to human development by means of the transferable nature of its science discoveries, as well as its potential to activate feelings of wonderment, inspiration and awareness of the universe. As a result, the programme potentially embodies a far broader mix of outcomes than conventionally considered in STEM evaluation approaches. Towards this aim, we operationalise our monitoring and evaluation approach by first outlining programme theories for three key OAD programmes: a programme for universities and research, another one for schools, and one for public outreach. We then identify outcomes, indicators and measures for each one of these programmes. We conclude with suggestions for evaluating the global impact of astronomy for development.

[1]  Candace Schau,et al.  Conceptual astronomy. II. Replicating conceptual gains, probing attitude changes across three semesters , 1999 .

[2]  Ömer Faruk Özdemir,et al.  The Effect of Science Centres on Students' Attitudes Towards Science , 2014 .

[3]  M. Patton,et al.  Evaluation: A Systematic Approach , 1980 .

[4]  M. Rollnick,et al.  Big Ideas: A review of astronomy education research 1974–2008 , 2010 .

[5]  R. Jarman,et al.  Physics for the star-gazer: pupils' attitudes to astronomy in the Northern Ireland Science Curriculum , 1996 .

[6]  L. Bickman.  Using program theory in evaluation , 1987 .

[7]  Carol H. Weiss,et al.  Theory‐based evaluation: Past, present, and future , 1997 .

[8]  Edward E. Prather,et al.  Measuring the conceptual understandings of citizen scientists participating in zooniverse projects: A first approach , 2013 .

[9]  Changes in Preservice Teacher Attitudes Toward Astronomy Within a Semester-Long Astronomy Instruction and Four-Year-Long Teacher Training Programme , 2011 .

[10]  Timothy F. Slater,et al.  Research on a Lecture-Tutorial Approach to Teaching Introductory Astronomy for Non-Science Majors , 2004 .

[11]  Beatriz Chu Clewell,et al.  Revitalizing the Nation's Talent Pool in STEM: Science, Technology, Engineering and Math , 2006 .

[12]  Eve Kikas,et al.  Children's knowledge of astronomy and its change in the course of learning , 2007 .

[13]  K. Sheppard,et al.  Toward a Methodology for Informal Astronomy Education Research , 2006 .

[14]  Virginia A. Atwood,et al.  Preservice Elementary Teachers' Conceptions of What Causes Night and Day , 1995 .

[15]  B. Bektasli.  The Effect of Media on Preservice Science Teachers' Attitudes toward Astronomy and Achievement in Astronomy Class. , 2013 .

[16]  Chris L. S. Coryn,et al.  A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009 , 2011 .

[17]  Floraline I. Stevens.  User-friendly handbook for project evaluation , 1994 .

[18]  Karen Jones,et al.  Developing Attitudes towards Science Measures , 2007 .

[19]  Tina Jarvis,et al.  Effect of the challenger experience on elementary children's attitudes to science , 2002 .

[20]  J. Ciotti.  Museums and Planetariums: Bridging the Gap between Hawaiian Culture and Astronomy through Informal Education-A Case Study , 2010 .

[21]  Tina Jarvis,et al.  Factors influencing elementary school children's attitudes toward science before, during, and after a visit to the UK National Space Centre , 2005 .

[22]  J. Mant.  A survey of British primary school teachers’ understanding of the Earth's place in the universe , 1995 .

[23]  Huey-tsyh Chen.  Theory-driven evaluations , 1990 .

[24]  Jennifer Caroline Greene,et al.  An educative, values‐engaged approach to evaluating STEM educational programs , 2006 .

[25]  Lisa Tsui,et al.  Effective Strategies to Increase Diversity in STEM Fields: A Review of the Research Literature , 2007 .

[26]  Paul J. Germann,et al.  Development of the attitude toward science in school assessment and its use to investigate the relationship between science achievement and attitude toward science in school , 1988 .

[27]  Floraline Stevens,et al.  User-Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering, and Technology Education. , 1993 .

[28]  Ayla Çetin-Dindar,et al.  Systemizing: A Cross-Cultural Constant for Motivation to Learn Science. , 2013 .

[29]  John J. Koran,et al.  The Things of Science: Assessing the Learning Potential of Science Museums. , 1986 .

[30]  Irene-Anna N. Diakidoy,et al.  Facilitating conceptual change in astronomy: a comparison of the effectiveness of two instructional approaches , 2001 .

[31]  Bernie Fanaroff,et al.  Research policy: How to build science capacity , 2012, Nature.

[32]  S. Simon,et al.  Attitudes towards science: A review of the literature and its implications , 2003 .

[33]  Impact of Observatory Visitor Centres on the Public's Understanding of Astronomy , 2000, Publications of the Astronomical Society of Australia.

[34]  R. Crisp,et al.  Superordinate and Subgroup Identification as Predictors of Intergroup Evaluation in Common Ingroup Contexts , 2007 .