Supporting the discoverability of Open Educational Resources : on the scent of a hidden treasury

Open Educational Resources (OERs), now available in large numbers, have a considerable potential to improve many aspects of society, yet one of the factors limiting this positive impact is the difficulty to discover them. This thesis investigates and proposes strategies to better support educators in discovering OERs. The literature suggests that the effectiveness of existing search systems, including for OER discovery, could be improved by supporting users, such as teachers, in carrying out more exploratory search activities closer to their existing methods of working. Hence, a preliminary taxonomy of OER-related search tasks was produced, based on an analysis of the literature, interpreted through Information Foraging Theory. This taxonomy was empirically evaluated to preliminarily identify a set of search tasks that involve finding other OERs similar to one that has already been identified, a process that is generally referred to as Query By Example (QBE). Following the Design Science Research methodology, three prototypes to support as well as to refine those tasks were iteratively designed, implemented, and evaluated involving an increasing number of educators in usability oriented studies. The resulting high-level and domain-oriented blended search/recommendation strategy transparently replicates Google searches in specialized networks, and identifies similar resources with a QBE strategy. It makes use of a domain-oriented similarity metric based on shared alignments to educational standards, and clusters results in expandable classes of comparable degrees of similarity. The summative evaluation shows that educators do appreciate this strategy because it is exploratory and – balancing similarity and diversity – it supports their high-level tasks, such as lesson planning and personalization of education. Finally, potential barriers and opportunities for the uptake of OER discovery tools were investigated in a structured interview study with experts from the OER field. Identified issues included how to work across multiple OER portals, variability in the use of metadata and how to align with the working practices of teachers. The findings of the thesis can be used to inform the research and development of methods and tools for OER discovery as well as their deployment to serve the needs of educators.

[1]  Mikael Nilsson,et al.  From Interoperability to Harmonization in Metadata Standardization: Designing an Evolvable Framework for Metadata Harmonization , 2010 .

[2]  S. Chatterjee,et al.  Design Science Research in Information Systems , 2010 .

[3]  Ben Shneiderman,et al.  Users can change their web search tactics: Design guidelines for categorized overviews , 2008, Inf. Process. Manag..

[4]  Steven E. Stemler,et al.  An overview of content analysis , 2003 .

[5]  Luanne Freund,et al.  Assigning search tasks designed to elicit exploratory search behaviors , 2012, HCIR '12.

[6]  Daniel R. Rehak,et al.  The Learning Registry: Sharing Federal Learning Resources , 2011 .

[7]  Juhani Iivari,et al.  A Paradigmatic Analysis of Information Systems As a Design Science , 2007, Scand. J. Inf. Syst..

[8]  Marten Schönherr,et al.  Outline of a design science research process , 2009, DESRIST.

[9]  Terry A. Slocum,et al.  Thematic cartography and geovisualization, 3rd Edition , 2008 .

[10]  Stephen L. Pruitt The Next Generation Science Standards: The Features and Challenges , 2014 .

[11]  Phil Barker,et al.  Into the wild – Technology for open educational resources , 2012 .

[12]  Mansaf Alam,et al.  A Review on Clustering of Web Search Result , 2012, ACITY.

[13]  King-Lup Liu,et al.  Building efficient and effective metasearch engines , 2002, CSUR.

[14]  J. Krosnick,et al.  Survey research. , 1999, Annual review of psychology.

[15]  Luanne Freund,et al.  Documenting and studying the use of assigned search tasks: RepAST , 2014, ASIST.

[16]  Sanjaya Mishra,et al.  The impact of OER and MOOCs on ODL: an international perspective , 2015 .

[17]  George W. Furnas,et al.  Model-driven formative evaluation of exploratory search: A study under a sensemaking framework , 2008, Inf. Process. Manag..

[18]  Alan R. Hevner,et al.  POSITIONING AND PRESENTING DESIGN SCIENCE RESEARCH FOR MAXIMUM IMPACT 1 , 2013 .

[19]  J. Cho,et al.  Reducing Confusion about Grounded Theory and Qualitative Content Analysis: Similarities and Differences , 2014 .

[20]  John E. Stith When I was your age , 2000, Nature.

[21]  Milad Shokouhi,et al.  Federated Search , 2011, Found. Trends Inf. Retr..

[22]  Ben Shneiderman,et al.  From Keyword Search to Exploration: Designing Future Search Interfaces for the Web , 2010, Found. Trends Web Sci..

[23]  Jon Mason,et al.  The Dublin Core and Metadata for Educational Resources , 2001, Dublin Core Conference.

[24]  Hengtao Tang,et al.  Understanding K-12 teachers' intention to adopt open educational resources: A mixed methods inquiry , 2020, Br. J. Educ. Technol..

[25]  H. Rangwala,et al.  Learning Analytics in Higher Education , 2017 .

[26]  John Domingue,et al.  A Linked Data-driven & Service-oriented Architecture for Sharing Educational Resources , 2011, Linked Learning@ESWC.

[28]  Neil Butcher,et al.  A Basic Guide to Open Educational Resources (OER) , 2011 .

[29]  Viswanath Venkatesh,et al.  Assessing a Firm's Web Presence: A Heuristic Evaluation Procedure for the Measurement of Usability , 2002, Inf. Syst. Res..

[30]  Luc Moreau,et al.  PROV-Overview. An Overview of the PROV Family of Documents , 2013 .

[31]  Marti A. Hearst Search User Interfaces , 2009 .

[32]  Y. Zaid,et al.  Sustaining Open Educational Resources (OER) initiatives in Nigerian Universities , 2020 .

[33]  Monica M. C. Schraefel,et al.  Building Knowledge: What's beyond Keyword Search? , 2009, Computer.

[34]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[35]  Marcia J. Bates,et al.  Where should the person stop and the information search interface start? , 1990, Inf. Process. Manag..

[36]  Marcia J. Bates,et al.  Information search tactics , 1979, J. Am. Soc. Inf. Sci..

[37]  John W. Creswell,et al.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2010 .

[38]  Søren Lauesen,et al.  Task descriptions versus use cases , 2011, Requirements Engineering.

[39]  Marie A. Bienkowski,et al.  The learning registry: building a foundation for learning resource analytics , 2012, LAK '12.

[40]  Lee Martie,et al.  Understanding the impact of support for iteration on code search , 2017, ESEC/SIGSOFT FSE.

[41]  Terry Ballard,et al.  Google Custom Search , 2012 .

[42]  Alexander Mikroyannidis,et al.  “Colearning” - collaborative networks for creating, sharing and reusing OER through social media , 2012 .

[43]  Katrien Verbert,et al.  Recommender Systems for Technology Enhanced Learning , 2014, Springer New York.

[44]  A. Lane The systemic implications of constructive alignment of higher education level learning outcomes and employer or professional body based competency frameworks , 2017 .

[45]  David R. Karger The Semantic Web and End Users: What's Wrong and How to Fix It , 2014, IEEE Internet Computing.

[46]  Janice Rattray,et al.  Essential elements of questionnaire design and development. , 2007, Journal of clinical nursing.

[47]  Allison Druin,et al.  Technology probes: inspiring design for and with families , 2003, CHI '03.

[48]  Gary Geisler Building Blocks For Rapid Development of Information Seeking Support Systems , 2007 .

[49]  Andrew Lane The potential of MOOCs to widen access to, and success in, higher education study , 2013 .

[50]  G. Norman Likert scales, levels of measurement and the “laws” of statistics , 2010, Advances in health sciences education : theory and practice.

[51]  Pierre Andrews,et al.  A classification of semantic annotation systems , 2012, Semantic Web.

[52]  F. Reichheld The one number you need to grow. , 2003, Harvard business review.

[53]  Huajun Chen,et al.  The Semantic Web , 2011, Lecture Notes in Computer Science.

[54]  Rob Nadolski,et al.  OpenScout: Competence based management education with community-improved open educational resources , 2010 .

[55]  Dean Leffingwell,et al.  Features, Use Cases, Requirements, Oh My! , 2001 .

[56]  Lois A. Ritter,et al.  Designing and Developing the Survey Instrument , 2007 .

[57]  Daniela Giordano,et al.  Interlinking educational resources and the web of data: A survey of challenges and approaches , 2013, Program.

[58]  Jurek Kirakowski,et al.  Functionality, usability, and user experience: three areas of concern , 2006, INTR.

[59]  Gary Marchionini,et al.  Exploratory search and HCI: designing and evaluating interfaces to support exploratory search interaction , 2007, CHI Extended Abstracts.

[60]  Ismar Frango Silveira,et al.  OER and MOOC: The Need for Openness , 2016 .

[61]  Talea Anderson,et al.  Strategies for Supporting OER Adoption through Faculty and Instructor Use of a Federated Search Tool , 2019, Journal of Librarianship and Scholarly Communication.

[62]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[63]  C. W. Cleverdon,et al.  The ASLIB CRANFIELD RESEARCH PROJECT ON The COMPARATIVE EFFICIENCY OF INDEXING SYSTEMS , 1960 .

[64]  Javiera Atenas,et al.  Questions of quality in repositories of open educational resources: a literature review , 2014 .

[65]  Ernesto Priego,et al.  Opening Teaching Landscapes: The Importance of Quality Assurance in the Delivery of Open Educational Resources , 2014, Open Praxis.

[66]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[67]  Kati Clements,et al.  Why open educational resources repositories fail : the contribution of quality approaches to the success of repositories , 2016 .

[68]  Preben Hansen,et al.  Work tasks as units for analysis in information seeking and retrieval studies , 2002 .

[69]  Lorna M. Campbell,et al.  3. Technology Strategies for Open Educational Resource Dissemination , 2016 .

[70]  David Bawden Logic and the organization of information , 2014, J. Documentation.

[71]  Bob J. Wielinga,et al.  The added value of task and ontology-based markup for information retrieval , 2004, J. Assoc. Inf. Sci. Technol..

[72]  Mark M. Hall,et al.  Exploring Large Digital Library Collections Using a Map-Based Visualisation , 2013, TPDL.

[73]  Roel Wieringa,et al.  Design Science Methodology for Information Systems and Software Engineering , 2014, Springer Berlin Heidelberg.

[74]  Phil Barker,et al.  LRMI, Learning Resource Metadata on the Web , 2015, WWW.

[75]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[76]  David Wiley,et al.  Defining OER-Enabled Pedagogy , 2018, The International Review of Research in Open and Distributed Learning.

[77]  H. Bernard,et al.  Techniques to Identify Themes , 2003 .

[78]  Petr Knoth,et al.  Visual Search for Supporting Content Exploration in Large Document Collections , 2012, D Lib Mag..

[79]  K. Squire,et al.  Design-Based Research: Putting a Stake in the Ground , 2004 .

[80]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[81]  Arrow Buttons Frequently asked questions , 2009 .

[82]  Rebecca Ferguson,et al.  Social learning analytics: five approaches , 2012, LAK.

[83]  Martin Šurina,et al.  The National Science Digital Library , 2008 .

[84]  Harold W. Thimbleby,et al.  User interface design , 1990, ACM Press Frontier Series.

[85]  Catherine Murphy Curriculum-Enhanced MARC (CEMARC): A New Cataloging Format for School Libraries. , 1995 .

[86]  Ryen W. White,et al.  Evaluating advanced search interfaces using established information-seeking models , 2009, J. Assoc. Inf. Sci. Technol..

[87]  Javiera Atenas,et al.  Model for democratisation of the contents hosted in MOOCs , 2015, International Journal of Educational Technology in Higher Education.

[88]  Bryan Marshall,et al.  Does Sample Size Matter in Qualitative Research?: A Review of Qualitative Interviews in is Research , 2013, J. Comput. Inf. Syst..

[89]  Jacky Akoka,et al.  Artifact Evaluation in Information Systems Design-Science Research - a Holistic View , 2014, PACIS.

[90]  Alberto Giretti,et al.  Metadata in Architecture Education - First Evaluation Results of the MACE System , 2009, EC-TEL.

[91]  M. Jones The open university. , 1974, Nursing times.

[92]  Marti A. Hearst,et al.  Sewing the Seams of Sensemaking: A Practical Interface for Tagging and Organizing Saved Search Results , 2013, HCIR '13.

[93]  Michael J. Wright,et al.  Linking learning goals and educational resources through interactive concept map visualizations , 2005, International Journal on Digital Libraries.

[94]  Brandon Muramatsu,et al.  Draft Standard for Learning Object Metadata , 2002 .

[95]  Mitesh Patel,et al.  Accessing the deep web , 2007, CACM.

[96]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[97]  Ji-Rong Wen,et al.  WWW 2007 / Track: Search Session: Personalization A Largescale Evaluation and Analysis of Personalized Search Strategies ABSTRACT , 2022 .

[98]  Mathieu d'Aquin,et al.  Analysing and Improving Embedded Markup of Learning Resources on the Web , 2017, WWW.

[99]  Armelle Brun,et al.  A Review on Explanations in Recommender Systems , 2017 .

[100]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[101]  Naihua Duan,et al.  Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research , 2015, Administration and Policy in Mental Health and Mental Health Services Research.

[102]  Shirley Gregor,et al.  Design Theory in Information Systems , 2002, Australas. J. Inf. Syst..

[103]  Anne Diekema,et al.  Map- or list-based recommender agents? Does the map metaphor fulfill its promise? , 2017, Inf. Vis..

[104]  Anil K. Jain Data clustering: 50 years beyond K-means , 2010, Pattern Recognit. Lett..

[105]  Petr Knoth,et al.  Linking textual resources to support information discovery , 2015 .

[106]  Javiera Atenas,et al.  Quality assurance in the open: an evaluation of OER repositories , 2013 .

[107]  Conor Hayes,et al.  Using Linked Data to Build Open, Collaborative Recommender Systems , 2010, AAAI Spring Symposium: Linked Data Meets Artificial Intelligence.

[108]  Daniela Petrelli,et al.  Affective graphs: The visual appeal of Linked Data , 2015, Semantic Web.

[109]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[110]  John Zimmerman,et al.  Research Through Design in HCI , 2014, Ways of Knowing in HCI.

[111]  Ivan Herman,et al.  RDFa Core 1.1 - Third Edition (Japanese translation), W3C Recommendation , 2018 .

[112]  Jon A. Krosnick,et al.  The Future of Survey Research: Challenges and Opportunities , 2015 .

[113]  Oren Etzioni,et al.  Grouper: A Dynamic Clustering Interface to Web Search Results , 1999, Comput. Networks.

[114]  John Zimmerman,et al.  An analysis and critique of Research through Design: towards a formalization of a research approach , 2010, Conference on Designing Interactive Systems.

[115]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[116]  Barry Smyth,et al.  Improving Recommendation Diversity , 2001 .

[117]  Ishan Sudeera Abeywardena,et al.  OERScout technology framework: A novel approach to open educational resources search , 2013 .

[118]  Daniel E. Atkins and John Seely Brown and Allen L. Hammond,et al.  A Review of the Open Educational Resources (OER) Movement: Achievements, Challenges, and New Opportunities , 2007 .

[119]  D. Cobb Placing the Spotlight on Open Educational Resources: Global Phenomenon or Cultural Guise?. , 2018 .

[120]  Pierluigi Ritrovato,et al.  SUPPORTING LEARNING OBJECT REPOSITORY BY AUTOMATIC EXTRACTION OF METADATA , 2015 .

[121]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[122]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[123]  H. Davies Competence-Based Curricula in the Context of Bologna and EU Higher Education Policy , 2017, Pharmacy.

[124]  Allison Woodruff,et al.  Using thumbnails to search the Web , 2001, CHI.

[125]  Elena García Barriocanal,et al.  Evaluating collaborative filtering recommendations inside large learning object repositories , 2013, Inf. Process. Manag..

[126]  Ahmed Tlili,et al.  Current state of open educational resources in the Arab region: an investigation in 22 countries , 2020, Smart Learning Environments.

[127]  Paolo Traverso,et al.  Service-Oriented Computing: State of the Art and Research Challenges , 2007, Computer.

[128]  Jonathan P. Tennant,et al.  A tale of two 'opens': intersections between Free and Open Source Software and Open Scholarship , 2020 .

[129]  Panagiotis D. Bamidis,et al.  Depicting Educational Content Repurposing Context and Inheritance , 2009, IEEE Transactions on Information Technology in Biomedicine.

[130]  Alan R. Hevner,et al.  Design Science in Information Systems Research , 2004, MIS Q..

[131]  Erik Duval,et al.  Dataset-driven research for improving recommender systems for learning , 2011, LAK.

[132]  G. Pyke Optimal Foraging Theory: A Critical Review , 1984 .

[133]  Hugh C. Davis,et al.  The evolution of metadata from standards to semantics in E-learning applications , 2006, HYPERTEXT '06.

[134]  Jill Gerhardt‐Powals,et al.  Cognitive engineering principles for enhancing human-computer performance , 1996, Int. J. Hum. Comput. Interact..

[135]  Daniela Giordano,et al.  A Survey on Linked Data and the Social Web as Facilitators for TEL Recommender Systems , 2014, Recommender Systems for Technology Enhanced Learning.

[136]  Raman Chandrasekar,et al.  Do thumbnail previews help users make better relevance decisions about web search results? , 2002, SIGIR '02.

[137]  Hendrik Drachsler,et al.  D8.1 Review of Social Data Requirements , 2012 .

[138]  Nicholas J. Belkin,et al.  Braque: Design of an Interface to Support User Interaction in Information Retrieval , 1993, Inf. Process. Manag..

[139]  Stuart A. Sutton Metadata Quality, Utility and the Semantic Web: The Case of Learning Resources and Achievement Standards , 2008 .

[140]  Giovanni Maria Sacco,et al.  Dynamic Taxonomies: A Model for Large Information Bases , 2000, IEEE Trans. Knowl. Data Eng..

[141]  N. Mackenzie,et al.  Research dilemmas: Paradigms, methods and methodology , 2006 .

[142]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[143]  Marian Petre,et al.  The unwritten rules of PhD research , 2004 .

[144]  Dorte Haltrup Hansen,et al.  Facilitating Metadata Interoperability in CLARIN-DK , 2016, LREC.

[145]  Andrew Lane,et al.  Open Education and the Sustainable Development Goals: Making Change Happen , 2017, Journal of Learning for Development.

[146]  Jakob Nielsen,et al.  Guerrilla HCI: using discount usability engineering to penetrate the intimidation barrier , 1994 .

[147]  Ann Abraham Information Seeking from Web-Based Resources: Sensemaking Strategies and Implications for Interaction Design , 2013 .

[148]  Bracha Shapira,et al.  Recommender Systems Handbook , 2015, Springer US.

[149]  Björn Niehaves,et al.  Epistemological perspectives on IS research: a framework for analysing and systematizing epistemological assumptions , 2007, Inf. Syst. J..

[150]  Susan T. Dumais,et al.  WaveLens: a new view onto Internet search results , 2004, CHI.

[151]  Toine Bogers Tag-Based Recommendation , 2018, Social Information Access.

[152]  Gordon I. McCalla,et al.  The Pedagogical Value of Papers: a Collaborative-Filtering based Paper Recommender , 2009, J. Digit. Inf..

[153]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[154]  Nathan R. Yergler Search and Discovery: OER's Open Loop , 2010 .

[155]  Mathieu d'Aquin,et al.  Assessing the educational linked data landscape , 2013, WebSci.

[156]  J. M. Pawlowski,et al.  User-oriented quality for OER: understanding teachers' views on re-use, quality, and trust , 2012, J. Comput. Assist. Learn..

[157]  Tim Berners-Lee,et al.  Linked Data - The Story So Far , 2009, Int. J. Semantic Web Inf. Syst..

[158]  Robert B. Miller,et al.  Response time in man-computer conversational transactions , 1899, AFIPS Fall Joint Computing Conference.

[159]  Judith S. Olson,et al.  Ways of Knowing in HCI , 2014, Springer New York.

[160]  Jan Pries-Heje,et al.  A Comprehensive Framework for Evaluation in Design Science Research 1 , 2022 .

[161]  Jakob Nielsen,et al.  Improving a human-computer dialogue , 1990, CACM.

[163]  Nicholas J. Belkin,et al.  Strategies for Evaluation of Interactive Multimedia Information Retrieval Systems , 1995, MIRO.

[164]  Peter Ingwersen,et al.  Information seeking research needs extension toward tasks and technology , 2004, Inf. Res..

[165]  Ulrike Cress,et al.  Learning in the Synergy of Multiple Disciplines, 4th European Conference on Technology Enhanced Learning, EC-TEL 2009, Nice, France, September 29 - October 2, 2009, Proceedings , 2009, EC-TEL.

[166]  Adam Croom Annotation is now a web standard – Hypothesis , 2017 .

[167]  Edmundo Tovar Caro,et al.  An approach for description of Open Educational Resources based on semantic technologies , 2010, IEEE EDUCON 2010 Conference.

[168]  Tian Luo,et al.  The power of open: benefits, barriers, and strategies for integration of open educational resources , 2019, Open Learning: The Journal of Open, Distance and e-Learning.

[169]  Martin Weller,et al.  Big and little OER , 2010 .

[170]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[171]  Peter Pirolli,et al.  Information Foraging , 2009, Encyclopedia of Database Systems.

[172]  Samir Chatterjee,et al.  A Design Science Research Methodology for Information Systems Research , 2008 .