Learnometrics: metrics for learning objects

The field of Technology Enhanced Learning (TEL) in general, has the potential to solve one of the most important challenges of our time: enable everyone to learn anything, anytime, anywhere. However, if we look back at more than 50 years of research in TEL, it is not clear where we are in terms of reaching our goal and whether we are, indeed, moving forward. The pace at which technology and new ideas evolve have created a rapid, even exponential, rate of change. This rapid change, together with the natural difficulty to measure the impact of technology in something as complex as learning, has lead to a field with abundance of new, good ideas and scarcity of evaluation studies. This lack of evaluation has resulted into the duplication of efforts and a sense of no "ground truth" or "basic theory' of TEL. This article is an attempt to stop, look back and measure, if not the impact, at least the status of a small fraction of TEL, Learning Object Technologies, in the real world. The measured apparent inexistence of the reuse paradox, the two phase linear growth of repositories or the ineffective metadata quality assessment of humans are clear reminders that even bright theoretical discussions do not compensate the lack of experimentation and measurement. Both theoretical and empirical studies should go hand in hand in order to advance the status of the field. This article is an invitation to other researchers in the field to apply Informetric techniques to measure, understand and apply in their tools the vast amount of information generated by the usage of Technology Enhanced Learning systems.

[1]  Udo Winand,et al.  The sharable content object reference model (SCORM) - a critical review , 2002, International Conference on Computers in Education, 2002. Proceedings..

[2]  Erik Duval,et al.  Repurposing Learning Object Components , 2005, OTM Workshops.

[3]  Erik Duval,et al.  Quantitative analysis of user-generated content on the Web , 2008 .

[4]  Erik Duval,et al.  The ALOCOM Framework: Towards Scalable Content Reuse , 2008, J. Digit. Inf..

[5]  Geoffrey A. Moore Crossing the chasm : marketing and selling high-tech products to mainstream customers , 1999 .

[6]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[7]  Erik Duval,et al.  Towards Automatic Evaluation of Learning Object Metadata Quality , 2006, ER.

[8]  Susan T. Dumais,et al.  Learning user interaction models for predicting web search result preferences , 2006, SIGIR.

[9]  Miranda Lee Pao An empirical examination of Lotka's law , 1986, J. Am. Soc. Inf. Sci..

[10]  Jennifer Widom,et al.  SimRank: a measure of structural-context similarity , 2002, KDD.

[11]  Marieke Guy,et al.  Improving the quality of metadata in eprint archives , 2004 .

[12]  S. Downes Models for Sustainable Open Educational Resources , 2007 .

[13]  Derek de Solla Price,et al.  A general theory of bibliometric and other cumulative advantage processes , 1976, J. Am. Soc. Inf. Sci..

[14]  Hao-hua Chu,et al.  Search En-gines for the World Wide Web: A Compara-tive Study and Evaluation Methodology , 1996 .

[15]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[16]  Amol Kamat,et al.  A Metadata Search Engine for Digital Language Archives , 2005, D Lib Mag..

[17]  Erik Duval,et al.  THE ACTUAL USE OF METADATA IN ARIADNE: AN EMPIRICAL ANALYSIS , 2003 .

[18]  Concepción S. Wilson,et al.  The Literature of Bibliometrics, Scientometrics, and Informetrics , 2001, Scientometrics.

[19]  Erik Duval,et al.  SAmgI: Automatic Metadata Generation v2.0 , 2007 .

[20]  Rudolf Kammerl,et al.  Context Aware Reuse of Learning Resources , 2004 .

[21]  Jon Kleinberg,et al.  Authoritative sources in a hyperlinked environment , 1999, SODA '98.

[22]  Mary Reddick,et al.  Open courseware and shared knowledge in higher education , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[23]  Dragan Gasevic,et al.  Ontology-Based Learning Content Repurposing: The ALOCoM Framework. , 2006 .

[24]  Daniel Olmedilla,et al.  Realizing interoperability of e-learning repositories , 2007 .

[25]  M. L. Apuzzo Summa cum laude. , 1999, Neurosurgery.

[26]  Charles F. Harrington,et al.  Course Management System Utilization and Implications for Practice: A National Survey of Department Chairpersons. , 2004 .

[27]  Erik Duval,et al.  Making Metadata go away: "Hiding everything but the benefits" , 2004, Dublin Core Conference.

[28]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[29]  R. Rousseau Sitations: an exploratory study , 1997 .

[30]  Erik Duval,et al.  Quantitative Analysis of Learning Object Repositories , 2008, IEEE Transactions on Learning Technologies.

[31]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[32]  E. Montroll,et al.  On 1/f noise and other distributions with long tails. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[33]  Ann Chapman,et al.  A catalogue quality audit tool , 2002 .

[34]  Jane Greenberg,et al.  Author-generated Dublin Core Metadata for Web Resources: A Baseline Study in an Organization , 2001, J. Digit. Inf..

[35]  Erik Duval,et al.  A Context-Aware Service Oriented Framework for Finding, Recommending and Inserting Learning Objects , 2006, EC-TEL.

[36]  David Wiley,et al.  Overcoming the Limitations of Learning Objects , 2004 .

[37]  Serge Mankovskii,et al.  Service Oriented Architecture , 2009, Encyclopedia of Database Systems.

[38]  Besiki Stvilia Measuring Information Quality , 2006 .

[39]  Kristin R. Eschenfelder,et al.  Software as Protest: The Unexpected Resiliency of U.S.-Based DeCSS Posting and Linking , 2004, Inf. Soc..

[40]  Ramani Duraiswami,et al.  A fast algorithm for learning large scale preference relations , 2007, AISTATS.

[41]  Erik Duval,et al.  Finding Appropriate Learning Objects: An Empirical Evaluation , 2005, ECDL.

[42]  Tie-Yan Liu,et al.  Learning to rank for information retrieval (LR4IR 2007) , 2007, SIGF.

[43]  Reihaneh Safavi-Naini,et al.  Digital Rights Management for Content Distribution , 2003, ACSW.

[44]  Mark E. J. Newman,et al.  Structure and Dynamics of Networks , 2009 .

[45]  Erik Duval,et al.  Frameworks for the Automatic Indexation of Learning Management Systems Content into Learning Object Repositories , 2005 .

[46]  MacKenzie Smith,et al.  The DSpace institutional digital repository system: current functionality , 2003, 2003 Joint Conference on Digital Libraries, 2003. Proceedings..

[47]  Erik Duval,et al.  Relevance ranking of learning objects based on usage and contextual information , 2007 .

[48]  Jeffrey Beall,et al.  Metadata and Data Quality Problems in the Digital Library , 2005, J. Digit. Inf..

[49]  Sanjiva Weerawarana,et al.  Unraveling the Web services web: an introduction to SOAP, WSDL, and UDDI , 2002, IEEE Internet Computing.

[50]  Erik Duval,et al.  Reusable learning objects: a survey of LOM-based repositories , 2002, MULTIMEDIA '02.

[51]  Xavier Ochoa Connexions: a Social and Successful Anomaly among Learning Object Repositories , 2010 .

[52]  Leo Egghe,et al.  Systems without low-productive sources , 2006, Inf. Process. Manag..

[53]  Mark E. J. Newman,et al.  Power-Law Distributions in Empirical Data , 2007, SIAM Rev..

[54]  Gustavo Alonso,et al.  Web Services: Concepts, Architectures and Applications , 2009 .

[55]  Jung-ran Park,et al.  An Assessment of Metadata Quality: A Case Study of the National Science Digital Library Metadata Repository , 2013 .

[56]  Leo Egghe,et al.  Generalized Success-Breeds-Success Principle Leading to Time-Dependent Informetric Distributions , 1995, J. Am. Soc. Inf. Sci..

[57]  Diane M. Strong,et al.  Data quality in context , 1997, CACM.

[58]  Lloyd Sokvitne Manager An Evaluation of the Effectiveness of Current Dublin Core Metadata for Retrieval , 2000 .

[59]  Rong Yan,et al.  Efficient Margin-Based Rank Learning Algorithms for Information Retrieval , 2006, CIVR.

[60]  Michel L. Goldstein,et al.  Problems with fitting to the power-law distribution , 2004, cond-mat/0402322.

[61]  Erik Duval,et al.  Relevance Ranking Metrics for Learning Objects , 2007, IEEE Transactions on Learning Technologies.

[62]  Duncan Lindsey,et al.  Production and Citation Measures in the Sociology of Science: The Problem of Multiple Authorship , 1980 .

[63]  Baden Hughes Metadata Quality Evaluation: Experience from the Open Language Archives Community , 2004, ICADL.

[64]  Erik Duval,et al.  Bridging an LCMS and an LMS: a Blackboard Building Block for the ARIADNE Knowledge Pool System , 2003 .

[65]  Kôiti Hasida,et al.  POLYPHONET: An advanced social network extraction system from the Web , 2007, J. Web Semant..

[66]  Robert N. Broadus Toward a definition of “bibliometrics” , 1987, Scientometrics.

[67]  M. Degroot,et al.  Probability and Statistics , 2021, Examining an Operational Approach to Teaching Probability.

[68]  Erik Duval,et al.  Evaluating the ALOCOM Approach for Scalable Content Repurposing , 2007, EC-TEL.

[69]  Hinrich Schütze,et al.  Personalized search , 2002, CACM.

[70]  Ian H. Witten,et al.  Thesaurus based automatic keyphrase indexing , 2006, Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '06).

[71]  Marek Hatala,et al.  The Evolution of Learning Object Repository Technologies: Portals for On-line Objects for Learning , 2002 .

[72]  Karen Sparck Jones A statistical interpretation of term specificity and its application in retrieval , 1972 .

[73]  Erik Duval,et al.  A Simple Query Interface for Interoperable Learning Repositories , 2005 .

[74]  John C. Nesbit,et al.  A Convergent Participation Model for Evaluation of Learning Objects , 2002 .

[75]  Pithamber R. Polsani,et al.  Use and Abuse of Reusable Learning Objects , 2006, J. Digit. Inf..

[76]  Erik Duval,et al.  Quality Metrics for Learning Object Metadata , 2006 .

[77]  Michael Mitzenmacher,et al.  A Brief History of Generative Models for Power Law and Lognormal Distributions , 2004, Internet Math..

[78]  Erik Duval,et al.  Automatic evaluation of metadata quality in digital repositories , 2009, International Journal on Digital Libraries.

[79]  Tim Pohle,et al.  Dynamic Playlist Generation Based on Skipping Behavior , 2005, ISMIR.

[80]  Erik Duval,et al.  Use of contextualized attention metadata for ranking and recommending learning objects , 2006, CAMA '06.

[81]  Alfred J. Lotka,et al.  The frequency distribution of scientific productivity , 1926 .

[82]  James Dalziel,et al.  Implementing learning design : the Learning Activity Management System (LAMS) , 2003 .

[83]  Steve Kirsch Infoseek's experiences searching the internet , 1998, SIGF.

[84]  J. Avery,et al.  The long tail. , 1995, Journal of the Tennessee Medical Association.

[85]  Kris Cardinaels,et al.  A dynamic learning object life cycle and its implications for automatic metadata generation , 2007 .

[86]  Erik Duval,et al.  Open metadata for open educational resources in an open infrastructure , 2007 .

[87]  Jaideep Srivastava,et al.  Automatic personalization based on Web usage mining , 2000, CACM.

[88]  Allison Littlejohn,et al.  Reusing Online Resources: A Sustainable Approach to E-Learning , 2003 .

[89]  Stephen Downes E-learning 2.0 , 2005, ELERN.

[90]  Erik Duval,et al.  Automating metadata generation: the simple indexing interface , 2005, WWW '05.

[91]  Erik Duval,et al.  A LOM Research Agenda , 2003, WWW.

[92]  Toru Iiyoshi,et al.  Opening Up Education: The Collective Advancement of Education through Open Technology, Open Content, and Open Knowledge , 2008 .

[93]  Amy Friedlander,et al.  D-Lib Magazine: Publishing as the Honest Broker , 1998 .

[94]  B. Epstein Some Applications of the Mellin Transform in Statistics , 1948 .

[95]  Erik Duval,et al.  Attention metadata: collection and management , 2006 .

[96]  Amanda J. Wilson,et al.  Toward Releasing the Metadata Bottleneck , 2007 .

[97]  Herbert Van de Sompel,et al.  Resource Harvesting within the OAI-PMH Framework , 2004, D Lib Mag..

[98]  Quentin L. Burrell,et al.  The Gini Index and the Leimkuhler Curve for Bibliometric Processes , 1992, Inf. Process. Manag..

[99]  G. Hanley,et al.  MERLOT: A faculty-focused Web site of educational resources , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[100]  Sarah E. Thomas Quality in Bibliographic Control , 1996, Libr. Trends.

[101]  M. Lynne Markus,et al.  Toward A Theory of Knowledge Reuse : Types of Knowledge Reuse Situations and Factors in Reuse Success , 2022 .

[102]  Jason Cole,et al.  Using Moodle - teaching with the popular open source course management system , 2007 .

[103]  M. Newman Power laws, Pareto distributions and Zipf's law , 2005 .

[104]  Kurt Maly,et al.  Arc - An OAI Service Provider for Digital Library Federation , 2001, D Lib Mag..

[105]  L. Egghe,et al.  The power of power laws and an interpretation of Lotkaian informetric systems as self-similar fractals , 2005, J. Assoc. Inf. Sci. Technol..

[106]  Ed H. Chi,et al.  Using information scent to model user information needs and actions and the Web , 2001, CHI.

[107]  Xavier Ochoa Learning object repositories are useful, but are they usable? , 2005, IADIS AC.

[108]  Katrina Strampel,et al.  Learning Objects in Use: 'Lite' Assessment for Field Studies , 2005 .

[109]  Laurence F Johnson,et al.  Elusive Vision: Challenges Impeding the Learning Object Economy , 2003 .

[110]  P. Whittle,et al.  A Model Explaining the Pareto Distribution of Wealth , 1957 .

[111]  Sarah Currier,et al.  Building Quality Assurance into Metadata Creation: An Analysis based on the Learning Objects and e-Prints Communities of Practice , 2003, Dublin Core Conference.

[112]  Ronald Fagin,et al.  Comparing and aggregating rankings with ties , 2004, PODS '04.

[113]  Ravi Kumar,et al.  Searching with context , 2006, WWW '06.

[114]  Mahesh H. Dodani The Dark Side of Object Learning: Learning Objects , 2002, J. Object Technol..

[115]  Erik Duval,et al.  The Ariadne knowledge pool system , 2001, CACM.

[116]  Otis Gospodnetic,et al.  Lucene in Action (In Action series) , 2004 .

[117]  Erik Duval We're on the road to... , 2004 .

[118]  Douglas C. Engelbart,et al.  Toward augmenting the human intellect and boosting our collective IQ , 1995, CACM.

[119]  M. E. J. Newman,et al.  Power laws, Pareto distributions and Zipf's law , 2005 .

[120]  John C. Huber A new model that generates Lotka's law , 2002, J. Assoc. Inf. Sci. Technol..

[121]  James L. Peterson,et al.  Computer-based readability indexes , 1982, ACM '82.

[122]  Miguel-Angel Sicilia,et al.  On the Concepts of Usability and Reusability of Learning Objects , 2003 .

[123]  Q. Vuong Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[124]  Filip Radlinski,et al.  Search Engines that Learn from Implicit Feedback , 2007, Computer.

[125]  Kevin Sweeney,et al.  Quantifying the reuse of learning objects , 2007 .

[126]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[127]  Erik Duval,et al.  User Behavior in Learning Object Repositories: An Empirical Analysis , 2004 .

[128]  Rory McGreal,et al.  A Typology of Learning Object Repositories , 2008 .

[129]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[130]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[131]  William Shockley,et al.  On the Statistics of Individual Variations of Productivity in Research Laboratories , 1957, Proceedings of the IRE.

[132]  Peter Ingwersen,et al.  Informetric analyses on the world wide web: methodological approaches to 'webometrics' , 1997, J. Documentation.

[133]  Elena García Barriocanal,et al.  Complete metadata records in learning object repositories: some evidence and requirements , 2005, Int. J. Learn. Technol..

[134]  P. Jacsó A Deficiency in the Algorithm for Calculating the Impact Factor of Scholarly Journals: The Journal Impact Factor , 2001, Cortex.

[135]  Oleg Liber,et al.  Learning objects: conditions for viability , 2005, J. Comput. Assist. Learn..

[136]  F. Massey The Kolmogorov-Smirnov Test for Goodness of Fit , 1951 .