Evaluating online learning communities

This article reviews recent evaluation studies of online learning communities to provide a systematic understanding of how such communities are evaluated. Forty-two representative studies were selected and categorized into a newly developed taxonomy of online learning community evaluations. This taxonomy is divided into four components: evaluation purpose, evaluation approach, measures for evaluation, and evaluation techniques. The findings suggest that it is inappropriate to conceptualize evaluation of such communities as a one-size-fits-all, generalizable measure of “good” or “bad.” Instead, we recommend a comprehensive, on-going, diagnostic approach to measuring clusters of indicators, or syndromes, of a particular OLC and examining the causal relation assumed by the evaluators between what is measured and the success of OLC as an imputed outcome.

[1]  Chi-Cheng Chang.  Towards a distributed web-based learning community , 2003 .

[2]  David H. Jonassen,et al.  Handbook of Research for educational Communications and Technology , 1997 .

[3]  Michael Orey,et al.  Learning communities via the Internet á la Epic Learning: you can lead the horses to water, but you cannot get them to drink , 2003 .

[4]  Marlene Scardamalia,et al.  Educational Applications of a Networked Communal Database , 1992, Interact. Learn. Environ..

[5]  M. Scriven.  Prose and Cons about Goal-Free Evaluation , 1991 .

[6]  J. Michael Spector,et al.  Handbook of Research on Educational Communications and Technology, 3rd Edition , 2012 .

[7]  Constance Steinkuehler,et al.  Learning in Massively Multiplayer Online Games , 2004, ICLS.

[8]  Minjuan Wang,et al.  Building a Dynamic Online Learning Community among Adult Learners , 2003 .

[9]  Caroline Haythornthwaite,et al.  Community Development Among Distance Learners: Temporal and Technological Dimensions , 2006, J. Comput. Mediat. Commun..

[10]  V. Lally,et al.  Building a Learning Community On-Line: Towards Socio-Academic Interaction. , 1999 .

[11]  A. P. Rovai.  Building classroom community at a distance: A case study , 2001 .

[12]  Karen Swan,et al.  Building Learning Communities in Online Courses: the importance of interaction , 2002 .

[13]  Christine B. Johnson,et al.  Supporting distributed communities of practice with interactive television , 2000 .

[14]  K. Kuutti.  Activity theory as a potential framework for human-computer interaction research , 1995 .

[15]  Jennifer Preece,et al.  Sociability and usability in online communities: Determining and measuring success , 2001, Behav. Inf. Technol..

[16]  Barry D. Davidson,et al.  Social networks, communication styles, and learning performance in a CSCL community , 2007, Comput. Educ..

[17]  John Cuthell,et al.  MirandaNet: A Learning Community--A Community of Learners. , 2002 .

[18]  Earl Woodruff,et al.  Online community and preservice teachers' conceptions of learning mathematics , 1999, CSCL.

[19]  Amy S. C. Leh.  Action Research on Hybrid Courses and their Online Communities , 2002 .

[20]  D. Maor.  The Teacher's Role in Developing Interaction and Reflection in an Online Learning Community , 2003 .

[21]  Richard A. Schwier,et al.  Considerations in the Construction of Technology-Based Virtual Learning Communities. , 1997 .

[22]  John Pearson.  Electronic networking in initial teacher education: is a virtual faculty of education possible? , 1999, Comput. Educ..

[23]  Alan M. Lesgold,et al.  A Computational Approach to Analyzing Online Knowledge Sharing Interaction , 2003 .

[24]  D. Hindman.  The Virtual Community: Homesteading on the Electronic Frontier , 1996 .

[25]  Leila Alem,et al.  Design and evaluation of an online learning community: a case study at CSIRO , 2005, SIGG.

[26]  Ove Jobring,et al.  The rationale of online learning communities , 2005, Int. J. Web Based Communities.

[27]  B. Nardi.  Context and consciousness: activity theory and human-computer interaction , 1995 .

[28]  Martin Oliver,et al.  An introduction to the Evaluation of Learning Technology , 2000, J. Educ. Technol. Soc..

[29]  Dawn M. Poole.  Student Participation in a Discussion-Oriented Online Course , 2000 .

[30]  Karen Littleton,et al.  The negotiation and co‐construction of meaning and understanding within a postgraduate online learning community , 2005 .

[31]  J. Creswell.  Qualitative inquiry and research design: choosing among five traditions. , 1998 .

[32]  A. P. Rovai.  Sense of community, perceived cognitive learning, and persistence in asynchronous learning networks , 2002, Internet High. Educ..

[33]  Scott D. Johnson,et al.  Team development and group processes of virtual learning teams , 2002, Comput. Educ..

[34]  J. M. Hugo.  Learning community history , 2002 .

[35]  John H. Schuh,et al.  Assessment in Student Affairs: A Guide for Practitioners. The Jossey-Bass Higher and Adult Education Series. , 1996 .

[36]  Paul Mulholland,et al.  Using genre to support active participation in learning communities , 2001 .

[37]  Christopher M. Johnson,et al.  Communities of Practice , 2001 .

[38]  Jennifer Lock,et al.  Laying the Groundwork for the Development of Learning Communities within Online Courses , 2002 .

[39]  Robert M. Fano.  Computer-Mediated Communication , 1985, IEEE Technology and Society Magazine.

[40]  Alexander Ardichvili,et al.  Virtual Knowledge-Sharing Communities of Practice at Caterpillar: Success Factors and Barriers. , 2008 .

[41]  William Spitzer,et al.  LabNet: an international electronic community for professional development , 1995 .

[42]  Patricia Comeaux.  Communication and Collaboration in the Online Classroom: Examples and Applications. , 2002 .

[43]  Sharon J. Derry,et al.  Assessing Knowledge Construction in On-Line Learning Communities. , 2000 .

[44]  D. Hung,et al.  Understanding how thriving internet quasi-communities work: Distinguishing between learning about and learning to be , 2002 .

[45]  David Mioduser,et al.  The Development of Social Climate in Virtual Learning Discussion Groups , 2002 .

[46]  J. Lave,et al.  Understanding Practice: Perspectives on Activity and Context , 1996 .

[47]  D. Chavis,et al.  Sense of community: A definition and theory , 1986 .

[48]  Victoria Hoban,et al.  The Reflective Practitioner , 2013 .

[49]  Maggie McPherson,et al.  The failure of a virtual social space (VSS) designed to create a learning community: lessons learned , 2004, Br. J. Educ. Technol..

[50]  Hunter Gehlbach,et al.  Assessing the Impact of a Large-Scale Online Teacher Professional Development Community , 2000 .

[51]  Jonida Shtylla,et al.  Computer mediated communication , 2011 .

[52]  D. Schoen.  The Reflective Practitioner , 1983 .

[53]  Robert Meyers,et al.  Professional Development: Building Effective Virtual Communities through Cooperative Learning. , 2002 .

[54]  J. Greeno,et al.  Thinking Practices in Mathematics and Science Learning , 1998 .

[55]  Vic Lally,et al.  Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community , 2003 .

[56]  Fengfeng Ke,et al.  Solitary Learner in Online Collaborative Learning: A Disappointing Experience? , 2006 .

[57]  Diana Sell Watkins.  The Development, Pilot Test, Assessment, and Evaluation of a Computerized Online Internet Community System at Programs for Higher Education, Fort Lauderdale, Florida. , 1997 .

[58]  Dianne L. Conrad,et al.  Deep in the Hearts of Learners: Insights into the Nature of Online Community , 2002 .

[59]  A. Bruckman.  Designing for Virtual Communities in the Service of Learning: Co-Evolution of Technological Design and Pedagogy in an Online Learning Community , 2004 .

[60]  Jan Herrington,et al.  An instructional design framework for authentic learning environments , 2000 .

[61]  Kenneth Tobin,et al.  The Practice of constructivism in science education , 1993 .

[62]  K. A. Renninger,et al.  Building Virtual Communities: Learning and Change in Cyberspace , 2002 .

[63]  Barbara Allan,et al.  Virtual Learning Communities , 2004 .

[64]  Etienne Wenger,et al.  Situated Learning: Legitimate Peripheral Participation , 1991 .

[65]  Rena M. Palloff,et al.  Building Learning Communities in Cyberspace: Effective Strategies for the Online Classroom. Jossey-Bass Higher and Adult Education Series. , 1999 .

[66]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1992 .

[67]  Lynda Ginsburg,et al.  Learning Online: Extending the Meaning of Community. A Review of Three Programs from the Southeastern United States. , 1999 .

[68]  Lai Hung Auyeung.  Building a Collaborative Online Learning Community: A Case Study in Hong Kong , 2004 .

[69]  Neil Mercer,et al.  The Guided Construction of Knowledge: Talk Amongst Teachers and Learners , 1995 .

[70]  Randolph A. Miller,et al.  Review: Medical Diagnostic Decision Support Systems - Past, Present, And Future: A Threaded Bibliography and Brief Commentary , 1994, J. Am. Medical Informatics Assoc..

[71]  Michael Quinn Patton,et al.  Utilization-Focused Evaluation , 1979 .

[72]  Christopher M. Hoadley,et al.  Anonymity options and professional participation in an online community of practice , 2005, CSCL.

[73]  Y. Engeström,et al.  Developmental studies of work as a testbench of activity theory: The case of primary care medical practice , 1993 .

[74]  K. Moore,et al.  Research and evaluation to serve the public: A commentary on Catalana, Berglund, Ryan, Lonczak, and Hawkins (2002). , 2002 .

[75]  M. J. W. Thomas,et al.  Learning within incoherent structures: the space of online discussion forums , 2002, J. Comput. Assist. Learn..

[76]  Christopher Plekss Hoadley,et al.  Creating context: design-based research in creating and understanding CSCL , 2002, CSCL.

[77]  Edwin H. Carpenter,et al.  Multimedia: asynchronous distributed education—a review and case study , 2000 .

[78]  Starr Roxanne Hiltz,et al.  What makes learning networks effective? , 2002, CACM.

[79]  Mary Lynn Rice-Lively.  Wired Warp and Woof: An Ethnographic Study of a Networking Class , 1994 .