Why open educational resources repositories fail : the contribution of quality approaches to the success of repositories

Clements, Kati Why Open Educational Resources Repositories Fail – the Contribution of Quality Approaches to the Success of Repositories University of Jyväskylä, 2016, 79p. (Jyväskylä Studies in Computing, ISSN 1456-5390; 233) ISBN 978-951-39-6587-7 (nid.) ISBN 978-951-39-6588-4 (PDF) Over the last two decades, millions of Open Educational Resources (OER) have become freely available for teachers and learners online. These resources are usually accessible through Learning Object Repositories (LORs), which are databases with a user interface. LORs are rarely used to their full potential and often considered as ‘failures’. Previous studies have shown that quality has a critical role in the success of information systems such as LORs. However there are no common guidelines for quality assurance approaches for LORs, nor are there studies into the contribution made by different LOR quality assurance approaches to the success of a repository. Measuring quality and success have proven to be a challenge in the research community due to the subjective and dynamic nature of such concepts. To meet this challenge, this thesis studied the perceptions of LOR users and developers with mutiple methodological perspectives. The contribution of this thesis to academic discourse comes from its deepening of the understanding of LOR users’ and developers’ perceptions on the quality and success of repositories. Frameworks for LOR quality assurance approaches and LOR success metrics were developed in order to identify and design approaches that would augment LOR success, as a theoretical contribution. The findings of these studies show that expert reviews combined with user-generated quality approaches (e.g. recommendation systems, peer reviews, commenting, tagging etc.) contribute towards LOR success. The practical contribution of this thesis is a set of recommendations towards the design of LOR quality approaches. These recommendations have already been used as guidelines for creating quality assurance approaches for four major European LORs.

[1]  Lisa A. Petrides,et al.  Open educational resources: inquiring into author use and reuse , 2008 .

[2]  J. Barton,et al.  Quality assurance for digital learning object repositories: issues for the metadata creation process , 2004 .

[3]  Marshall S. Smith,et al.  The Promise of Open Educational Resources , 2006 .

[4]  Ronald A. Phipps,et al.  Quality on the Line: Benchmarks for Success in Internet-Based Distance Education. , 2000 .

[5]  Jan M. Pawlowski,et al.  WHY OPEN EDUCATIONAL RESOURCES REPOSITORIES FAIL - REVIEW OF QUALITY ASSURANCE APPROACHES , 2014 .

[6]  Eugenijus Kurilovas.  Learning Content Repositories and Learning Management Systems Based on Customization and Metadata , 2009, 2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns.

[7]  Daniel E. Atkins and John Seely Brown and Allen L. Hammond,et al.  A Review of the Open Educational Resources (OER) Movement: Achievements, Challenges, and New Opportunities , 2007 .

[8]  Susanne Dobratz,et al.  The nestor Catalogue of Criteria for Trusted Digital Repository Evaluation and Certification , 2006, J. Digit. Inf..

[9]  Jan M. Pawlowski,et al.  Quality in European e-learning: An introduction , 2006 .

[10]  David A. Wiley,et al.  Online Self-Organizing Social Systems: The Decentralized Future of Online Learning , 2002 .

[11]  Nik Bessis,et al.  User-centred evaluation of an e-learning repository , 2006, NordiCHI '06.

[12]  Hugh C. Davis,et al.  Bootstrapping a Culture of Sharing to Facilitate Open Educational Resources , 2010, IEEE Transactions on Learning Technologies.

[13]  Jan M. Pawlowski,et al.  Quality mark e-learning: developing process- and product-oriented quality for learning, education and training , 2007, Int. J. Learn. Technol..

[14]  Beverley G. Hope,et al.  A Model for the Effective Management of Reusable Learning Objects (RLO's): Lessons from a Case Study , 2005, PACIS.

[15]  Neil Butcher,et al.  A Basic Guide to Open Educational Resources (OER) , 2011 .

[16]  M. Cruickshank,et al.  Total Quality Management in the higher education sector: A literature review from an international and Australian perspective , 2003 .

[17]  Sarah Currier,et al.  Building Quality Assurance into Metadata Creation: An Analysis based on the Learning Objects and e-Prints Communities of Practice , 2003, Dublin Core Conference.

[18]  S. Robinson.  Trust and Breach of the Psychological Contract , 1996 .

[19]  Carole L. Palmer,et al.  Strategies for Institutional Repository Development: A Case Study of Three Evolving Initiatives , 2009, Libr. Trends.

[20]  David A. Wiley,et al.  The Learning Objects Literature , 2007 .

[21]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[22]  Les Gasser,et al.  Metadata Quality For Federated Collections , 2004, ICIQ.

[23]  Jan M. Pawlowski,et al.  Handbook on Quality and Standardisation in E-Learning , 2006 .

[24]  Lawrence D. Fredendall,et al.  Evaluating the Deming Management Model of Total Quality in Services , 2004, Decis. Sci..

[25]  J. M. Pawlowski,et al.  User-oriented quality for OER: understanding teachers' views on re-use, quality, and trust , 2012, J. Comput. Assist. Learn..

[26]  Rory McGreal,et al.  Learning objects: A practical definition , 2004 .

[27]  Rory McGreal,et al.  A Typology of Learning Object Repositories , 2008 .

[28]  Erik Duval,et al.  Metadata interoperability in agricultural learning repositories: an analysis. , 2010 .

[29]  Ling Liu,et al.  PeerTrust: supporting reputation-based trust for peer-to-peer electronic communities , 2004, IEEE Transactions on Knowledge and Data Engineering.

[30]  Paul Stacey.  Open educational resources in a global context , 2007, First Monday.

[31]  Ramzi Farhat,et al.  A framework to design quality-based learning objects , 2005, Fifth IEEE International Conference on Advanced Learning Technologies (ICALT'05).

[32]  Tina Wilson,et al.  New Ways of Mediating Learning: Investigating the implications of adopting open educational resources for tertiary education at an institution in the United Kingdom as compared to one in South Africa , 2008 .

[33]  Elizabeth D. Liddy,et al.  Automatic metadata generation & evaluation , 2002, SIGIR '02.

[34]  Mahmoud M. Yasin,et al.  TQM practices in service organizations: an exploratory study into the implementation, outcome and effectiveness , 2004 .

[35]  Erik Duval,et al.  Relevance Ranking Metrics for Learning Objects , 2007, IEEE Transactions on Learning Technologies.

[36]  D. Sandy Staples,et al.  Toward Contextualized Theories of Trust: The Role of Trust in Global Virtual Teams , 2004, Inf. Syst. Res..

[37]  Deepti Mehrotra,et al.  Quality metrics a quanta for retrieving learning object by clustering techniques , 2012, 2012 Second International Conference on Digital Information and Communication Technology and it's Applications (DICTAP).

[38]  Ying Zhang,et al.  IFLA FRBR as user-centered metadata evaluation framework for the Moving Image Collections , 2004 .

[39]  Montse Guitert,et al.  Learner Generated Content: Quality Criteria in online Collaborative Learning , 2011 .

[40]  Jan M. Pawlowski,et al.  Understanding Social OER Environments—A Quantitative Study on Factors Influencing the Motivation to Share and Collaborate , 2014, IEEE Transactions on Learning Technologies.

[41]  Meg O'Reilly,et al.  Sharing quality resources for teaching and learning: A peer review model for the ALTC Exchange in Australia , 2009 .

[42]  Susan M. Mitchell,et al.  Assessing the Value of Computer Science Course Material Repositories , 2006, 19th Conference on Software Engineering Education and Training Workshops (CSEETW'06).

[43]  Jay F. Nunamaker,et al.  Special Issue: Information Systems Success , 2003, J. Manag. Inf. Syst..

[44]  Rob Koper,et al.  Evidence of Cross-boundary Use and Reuse of Digital Educational Resources , 2009, Int. J. Emerg. Technol. Learn..

[45]  Allison Littlejohn,et al.  Issues in reusing online resources , 2003 .

[46]  Ulf-Daniel Ehlers.  Quality in e-learning: use and dissemination of quality approaches in European e-learning: a study by the European Quality Observatory , 2005 .

[47]  J. Temple.  Conducting Research Literature Reviews: from the Internet to Paper, 2nd edn , 2006 .

[48]  Judith M. Tanur,et al.  Methods for Large-Scale Surveys and Experiments , 1983 .

[49]  Tom Browne,et al.  The challenges of OER to Academic Practice , 2010 .

[50]  Lynne C. Howarth.  Designing a Common Namespace for Searching Metadata-Enabled Knowledge Repositories: An International Perspective , 2003 .

[51]  Eric S. Raymond,et al.  The cathedral and the bazaar - musings on Linux and Open Source by an accidental revolutionary , 2001 .

[52]  Robert W. Hanna,et al.  Gathering faculty teaching evaluations by in‐class and online surveys: their effects on response rates and evaluations , 2004 .

[53]  Ilkka Tuomi.  Open Educational Resources: What they are and why do they matter , 2006 .

[54]  Jay F. Nunamaker,et al.  Systems Development in Information Systems Research , 1990, J. Manag. Inf. Syst..

[55]  Jan M. Pawlowski,et al.  The Quality Adaptation Model: Adaptation and Adoption of the Quality Standard ISO/IEC 19796-1 for Learning, Education, and Training , 2007, J. Educ. Technol. Soc..

[56]  Dan Johnson,et al.  Metadata Clean Sweep: A Digital Library Audit Project , 2012, D Lib Mag..

[57]  M. D. Myers,et al.  Dialectical hermeneutics: a theoretical framework for the implementation of information systems , 1995, Inf. Syst. J..

[58]  Henri Pirkkalainen,et al.  Collaborating on ideas: Tackling Barriers to Open Education , 2014 .

[59]  Ulf-Daniel Ehlers,et al.  Quality Literacy - Competencies for Quality Development in Education and e-Learning , 2007, J. Educ. Technol. Soc..

[60]  Nigel Bevan,et al.  International standards for HCI and usability , 2001, Int. J. Hum. Comput. Stud..

[61]  Juan Manuel Dodero,et al.  Ranking Learning Objects through Integration of Different Quality Indicators , 2010, IEEE Transactions on Learning Technologies.

[62]  Saonee Sarker,et al.  Exploring Agility in Distributed Information Systems Development Teams: An Interpretive Study in an Offshoring Context , 2009, Inf. Syst. Res..

[63]  Brandon Muramatsu,et al.  Oer recommender: linking nsdl pathways and opencourseware repositories , 2008, JCDL '08.

[64]  Stephen Downes.  Learning Objects: Resources For Distance Education Worldwide , 2001 .

[65]  Erik Duval,et al.  Interoperability for Searching Learning Object Repositories: The ProLearn Query Language , 2008, D Lib Mag..

[66]  Christian M. Stracke,et al.  Quality Development and Standards in e-Learning: Benefits and Guidelines for Implementations , 2013 .

[67]  Douglas B. Terry,et al.  Using collaborative filtering to weave an information tapestry , 1992, CACM.

[68]  David E. Millard,et al.  The HumBox: Changing educational practice around a learning resource repository , 2013, Comput. Educ..

[69]  Hans van Buuren,et al.  Measuring perceived sociability of computer-supported collaborative learning environments , 2007, Comput. Educ..

[70]  John C. Nesbit,et al.  A Framework for Evaluating the Quality of Multimedia Learning Resources , 2007, J. Educ. Technol. Soc..

[71]  C. Johnson-George,et al.  Measurement of specific interpersonal trust: Construction and validation of a scale to assess trust in a specific other. , 1982 .

[72]  Robert H. McDonald,et al.  Measuring and comparing participation patterns in digital repositories: repositories by the numbers part 1 , 2007 .

[73]  Wolfram Horstmann.  Building Repository Networks with DRIVER: Joint Explorations with the IR Grid in China , 2009 .

[74]  Erik Duval,et al.  Quantitative Analysis of Learning Object Repositories , 2008, IEEE Transactions on Learning Technologies.

[75]  Dr. Jan Hylén.  Open Educational Resources: Opportunities and Challenges , 2008 .

[76]  S. Downes.  Models for Sustainable Open Educational Resources , 2007 .

[77]  Steve Ryan,et al.  Reuse, repurposing and learning design - Lessons from the DART project , 2008, Comput. Educ..

[78]  Ephraim R. McLean,et al.  Measuring information systems success: models, dimensions, measures, and interrelationships , 2008, Eur. J. Inf. Syst..

[79]  Jane Sinclair,et al.  A Practice-Oriented Review of Learning Objects , 2013, IEEE Transactions on Learning Technologies.

[80]  Symeon Retalis,et al.  QUALITY ASSURANCE PROCEDURES AND e-ODL , 2003 .

[81]  Bailey,et al.  Open Access Bibliography: Liberating Scholarly Literature with E-Prints and Open Access Journals , 2005 .

[82]  Ephraim R. McLean,et al.  Information systems success revisited , 2002, Proceedings of the 35th Annual Hawaii International Conference on System Sciences.

[83]  Gordana Dodig Crnkovic,et al.  Constructive Research and Info-computational Knowledge Generation , 2010 .

[84]  Moez Limayem,et al.  The role of modelling in achieving information systems success: UML to the rescue? , 2009, Inf. Syst. J..

[85]  Heimo H. Adelsberger,et al.  Learning from experience - towards a formal model for contextualization of experiences for quality development , 2004, IEEE International Conference on Advanced Learning Technologies, 2004. Proceedings..

[86]  Mohamed Ally,et al.  A Study of the Design and Evaluation of a Learning Object and Implications for Content Development , 2005 .

[87]  Jarmo Saarti,et al.  Multi-Faceted Measuring of the Quality as a Tool for Quality Improvement in the Kuopio University Library, Finland , 2010 .

[88]  D. Wiley,et al.  A decade of development… , 2009 .

[89]  William E. Moen,et al.  The Role of Content Analysis in Evaluating Metadata for the U.S. Government Information Locator Service (GILS): Results from an Exploratory Study , 1997 .

[90]  Thomas Richter,et al.  Barriers and Motivators for Using Open Educational Resources in Schools , 2010 .

[91]  Andrew Lane,et al.  Production of OER, a Quest for Efficiency , 2010 .

[92]  D. Wiley,et al.  Open Educational Resources: A Review of the Literature , 2014 .

[93]  Julià Minguillón Alfonso,et al.  Extending Learning Objects by Means of Social Networking , 2010 .

[94]  David Massart.  Towards a Pan-European Learning Resource Exchange Infrastructure , 2009, NGITS.

[95]  Nikos Manouselis,et al.  Quality in metadata: a schema for e-commerce , 2006, Online Inf. Rev..

[96]  Nikos Manouselis,et al.  Analysis and Classification of Multi-Criteria Recommender Systems , 2007, World Wide Web.

[97]  Demetrios G. Sampson,et al.  Recommendation of quality approaches for the European Quality Observatory , 2004, IEEE International Conference on Advanced Learning Technologies, 2004. Proceedings..

[98]  Robin D. Burke,et al.  Hybrid Recommender Systems: Survey and Experiments , 2002, User Modeling and User-Adapted Interaction.

[99]  Teemu Leinonen,et al.  Information Architecture and Design Solutions Scaffolding Authoring of Open Educational Resources , 2010, IEEE Transactions on Learning Technologies.

[100]  Elena García Barriocanal,et al.  Social models in open learning object repositories: A simulation approach for sustainable collections , 2011, Simul. Model. Pract. Theory.

[101]  Amanda J. Wilson,et al.  Toward Releasing the Metadata Bottleneck , 2007 .

[102]  Darina Dicheva,et al.  Is it time to change the OER repositories role? , 2012, JCDL '12.

[103]  Mimi Recker,et al.  Using web metrics to analyze digital libraries , 2008, JCDL.

[104]  Xavier Ochoa.  Connexions: a Social and Successful Anomaly among Learning Object Repositories , 2010 .

[105]  Bryan A. Pendleton,et al.  Power of the Few vs. Wisdom of the Crowd: Wikipedia and the Rise of the Bourgeoisie , 2006 .

[106]  John C. Nesbit,et al.  A Convergent Participation Model for Evaluation of Learning Objects , 2002 .

[107]  Ephraim R. McLean,et al.  The DeLone and McLean Model of Information Systems Success: A Ten-Year Update , 2003, J. Manag. Inf. Syst..

[108]  Kati Clements,et al.  Implementing Quality Standards for Knowledge-Intensive Organizations. , 2010 .

[109]  Audun Jøsang,et al.  A survey of trust and reputation systems for online service provision , 2007, Decis. Support Syst..

[110]  Panagiotis Zervas,et al.  A quantitative analysis of learning object repositories as knowledge management systems , 2014 .

[111]  A. Zaheer,et al.  Does Trust Matter? Exploring the Effectsof Interorganizational and Interpersonaltrust on Performance , 1998 .

[112]  J. Stephen Town,et al.  Benchmarking and library quality maturity , 2006 .

[113]  Vive Kumar,et al.  Rating learning object quality with distributed Bayesian belief networks: the why and the how , 2005, Fifth IEEE International Conference on Advanced Learning Technologies (ICALT'05).

[114]  Sofoklis Sotiriou.  COSMOS: An Advanced Scientific Repository for Science Teaching and Learning , 2008, 2008 Eighth IEEE International Conference on Advanced Learning Technologies.

[115]  Marlene Scardamalia,et al.  Computer Support for Knowledge-Building Communities , 1994 .

[116]  Hal Abelson,et al.  The Creation of OpenCourseWare at MIT , 2008 .

[117]  Leo Waaijers,et al.  Quality of Research Data, an Operational Approach , 2011, D Lib Mag..

[118]  Cat S. McDowell.  Evaluating Institutional Repository Deployment in American Academe Since Early 2005: Repositories by the Numbers, Part 2 , 2007, D Lib Mag..

[119]  S. Retalis.  Usable and Interoperable E-Learning Resources Repositories , 2005 .

[120]  Diana Laurillard,et al.  Open teaching: The key to sustainable and effective open education , 2008 .

[121]  Fernando Rosell-Aguilar,et al.  Delivering unprecedented access to learning through podcasting as OER, but who's listening? A profile of the external iTunes U user , 2013, Comput. Educ..

[122]  M. Heidegger,et al.  Introduction to Phenomenological Research , 2005 .

[123]  Nikolaos Palavitsinis,et al.  Metadata quality issues in learning repositories , 2014 .

[124]  Thomas Richter,et al.  Overcoming Cultural Distance in Social OER Environments , 2014, CSEDU.

[125]  John C. Nesbit,et al.  Learning Object Evaluation: Computer-Mediated Collaboration And Inter-Rater Reliability , 2003 .

[126]  Christian M. Stracke,et al.  Quality Development and Quality Standards in e Learning: Adoption, Implementation, and Adaptation , 2007 .

[127]  Peter R. Newsted,et al.  Survey Instruments in Information Systems , 1998, MIS Q..

[128]  Marie A. Bienkowski,et al.  The Learning Registry: Applying Social Metadata for Learning Resource Recommendations , 2014, Recommender Systems for Technology Enhanced Learning.

[129]  Robert H. McDonald,et al.  Measuring and Comparing Participation Patterns In Digital Repositories, Repositories by the Numbers Part I , 2007 .

[130]  Nikos Manouselis,et al.  Exploratory study of multi-criteria recommendation algorithms over technology enhanced learning datasets , 2014 .

[131]  Dragan Bjelica,et al.  Measuring Project Success in Project-Oriented , 2013 .

[132]  Erik Duval,et al.  Metadata for social recommendations: storing, sharing and reusing evaluations of learning resources , 2008 .

[133]  Ulf-Daniel Ehlers,et al.  Web 2.0 – e‐learning 2.0 – quality 2.0? Quality for new learning cultures , 2009 .

[134]  Chris Pegler.  Herzberg, hygiene and the motivation to reuse: Towards a three-factor theory to explain motivation to share and use OER , 2012 .

[135]  Carmel McNaught.  Developing Criteria for Successful Learning Repositories , 2006, WEBIST.

[136]  Erik Duval,et al.  Peer-to-Peer versus Federated Search: Towards more Interoperable Learning Object Repositories , 2005 .

[137]  Denise M. Rousseau,et al.  What's a good reason to change? Motivated reasoning and social accounts in promoting organizational change. , 1999 .

[138]  Heather Wharrad,et al.  Sharing and reuse in OER: experiences gained from open reusable learning objects in health , 2010 .

[139]  Kati Clements,et al.  Trusted educational networks for the internationalization of open educational resources , 2013 .

[140]  L. Lipponen,et al.  Practices and orientations of CSCL , 2004 .

[141]  Javiera Atenas,et al.  Quality assurance in the open: an evaluation of OER repositories , 2013 .

[142]  Erik Duval,et al.  Reusable learning objects: a survey of LOM-based repositories , 2002, MULTIMEDIA '02.

[143]  Brian Kahin,et al.  The Impact of ICT on Tertiary Education: Advances and Promises , 2006 .

[144]  Riina Vuorikari,et al.  An Overview of Learning Object Repositories , 2009, Database Technologies: Concepts, Methodologies, Tools, and Applications.

[145]  Chia-Ping Yu,et al.  Toward an Understanding of MIS Survey Research Methodology: Current Practices, Trends, and Implications for Future Research , 2003 .

[146]  H. Gaunt,et al.  Total Quality Management for Schools , 1994 .

[147]  Yannis Dimitriadis,et al.  New design approaches to repurposing open educational resources for collaborative learning using mediating artefacts , 2009 .

[148]  Panagiotis Zervas,et al.  Learning object repositories as knowledge management systems , 2013 .

[149]  D. Ferrin,et al.  The Role of Trust in Organizational Settings , 2001 .

[150]  D. S. Morris,et al.  Overcoming the barriers to TQM , 1996 .

[151]  Alain Pinsonneault,et al.  Survey Research Methodology in Management Information Systems: An Assessment , 1993, J. Manag. Inf. Syst..

[152]  Philippe Vidal,et al.  Learning Object Virtualization Allowing for Learning Object Assessments and Suggestions for Use , 2008, 2008 Eighth IEEE International Conference on Advanced Learning Technologies.

[153]  T. Richter,et al.  Open educational resources: education for the world? , 2012 .

[154]  Walter Fernandez,et al.  Success in IT projects: A matter of definition? , 2008 .

[155]  Jane Greenberg.  Theoretical Considerations of Lifecycle Modeling: An Analysis of the Dryad Repository Demonstrating Automatic Metadata Propagation, Inheritance, and Value System Adoption , 2009 .

[156]  D. Larcker,et al.  Quality strategy, strategic control systems, and organizational performance , 1997 .

[157]  P. Dillenbourg.  What do you mean by collaborative learning , 1999 .

[158]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[159]  Tung-Ran Park,et al.  Semantic interoperability and metadata quality : An analysis of metadata item records of digital image collections , 2006 .

[160]  Sarah Higgins,et al.  The dcc curation lifecycle model , 2008, JCDL '08.

[161]  Bo-Christer Björk,et al.  Open access to scientific publications - an analysis of the barriers to change , 2003, Inf. Res..

[162]  Natasha Boskic.  Learning objects design: what do educators think about the quality and reusability of learning objects? , 2003, Proceedings 3rd IEEE International Conference on Advanced Technologies.

[163]  Pithamber R. Polsani,et al.  Use and Abuse of Reusable Learning Objects , 2006, J. Digit. Inf..

[164]  Erik Duval,et al.  Recommender Systems for Learning , 2012, Springer Briefs in Electrical and Computer Engineering.

[165]  Rich Anderson.  ISO 9000 Quality Systems Handbook (Book) , 2002 .

[166]  Matthias Jarke,et al.  The future of e-learning: a shift to knowledge networking and social software , 2007, Int. J. Knowl. Learn..

[167]  Alexander Mikroyannidis,et al.  “Colearning” - collaborative networks for creating, sharing and reusing OER through social media , 2012 .

[168]  U. Ehlers.  Quality in e-Learning from a Learner’s Perspective , 2004, Distances et médiations des savoirs.

[169]  Debra Howcroft,et al.  Re-conceptualising failure: social shaping meets IS research , 2002, Eur. J. Inf. Syst..

[170]  Hendrik Drachsler,et al.  Recommender Systems in Technology Enhanced Learning , 2011, Recommender Systems Handbook.