Quality Metrics for Learning Object Metadata

The quality of the learning objects metadata records stored in a repository is important its operation and interoperability. While several studies have tried to define and measure quality for metadata, a scalable and effective way to assess this quality is not currently available. This work converts the fuzzy quality definitions found in those studies into implementable measures (metrics). Several of these metrics are proposed. They are based on the same quality parameters used for human review of metadata: completeness, accuracy, provenance, conformance to expectations, logical consistency and coherence, timeliness, and accessibility. The information requirements to calculate the proposed metrics are also detailed. Some of these metrics are implemented and tested over to two collection of Learning Object Metadata, one of mainly human generated metadata, the other generated by automated means. Early results suggest that the metrics are indeed sensible to quality features in the metadata. Finally, this work recommends further work to validate and calibrate the proposed metrics.

[1]  Philip Resnik,et al.  Using Information Content to Evaluate Semantic Similarity in a Taxonomy , 1995, IJCAI.

[2]  Elizabeth U. Mangan.  The Making of a Standard. , 1995 .

[3]  Stephen Evans,et al.  A TQM Approach to the Improvement of Information Quality , 1996, IQ.

[4]  Arie Segev.  On Information Quality and the WWW Impact: A Position Paper , 1996, IQ.

[5]  Rachel Heery.  Review of metadata formats , 1996 .

[6]  Diane M. Strong,et al.  Data quality in context , 1997, CACM.

[7]  William E. Moen,et al.  An Evaluation of the Federal Government's Implementation of the Government Information Locator Service (GILS): Final Report , 1997 .

[8]  William E. Moen,et al.  The Role of Content Analysis in Evaluating Metadata for the U.S. Government Information Locator Service (GILS): Results from an Exploratory Study , 1997 .

[9]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[10]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[11]  Susan Gauch,et al.  Incorporating quality metrics in centralized/distributed information retrieval on the World Wide Web , 2000, SIGIR '00.

[12]  Diane M. Strong,et al.  Information quality benchmarks: product and service performance , 2002, CACM.

[13]  Oleg Kiselyov.  SXML specification , 2002, SIGP.

[14]  Diane Hillmann,et al.  Analyzing Metadata for Effective Use and Re-Use , 2003, Dublin Core Conference.

[15]  Akiko Aizawa,et al.  An information-theoretic perspective of tf-idf measures , 2003, Inf. Process. Manag..

[16]  Erik Duval,et al.  THE ACTUAL USE OF METADATA IN ARIADNE: AN EMPIRICAL ANALYSIS , 2003 .

[17]  Sarah Currier,et al.  Building Quality Assurance into Metadata Creation: An Analysis based on the Learning Objects and e-Prints Communities of Practice , 2003, Dublin Core Conference.

[18]  Erik Duval,et al.  User Behavior in Learning Object Repositories: An Empirical Analysis , 2004 .

[19]  Marieke Guy,et al.  Improving the quality of metadata in eprint archives , 2004 .

[20]  Diane I. Hillmann,et al.  The Continuum of Metadata Quality: Defining, Expressing, Exploiting , 2004 .

[21]  Baden Hughes.  Metadata Quality Evaluation: Experience from the Open Language Archives Community , 2004, ICADL.

[22]  Marcia Lei Zeng,et al.  Metadata Quality Study for the National Science Digital Library (NSDL) Metadata Repository , 2004, ICADL.

[23]  Erik Duval,et al.  Frameworks for the Automatic Indexation of Learning Management Systems Content into Learning Object Repositories , 2005 .

[24]  Erik Duval,et al.  Attention Metadata Management: Tracking the use of Learning Objects through Attention.XML , 2005 .

[25]  Erik Duval,et al.  Automating metadata generation: the simple indexing interface , 2005, WWW '05.

[26]  Stuart Weibel,et al.  Border Crossings: Reflections on a Decade of Metadata Consensus Building , 2005, D Lib Mag..

[27]  Besiki Stvilia,et al.  Is 'Quality' Metadata 'Shareable' Metadata? The Implications of Local Metadata Practice on Federated Collections , 2005 .