User-Aware Music Retrieval

Personalized and user-aware systems for retrieving multimedia items are becoming increasingly important as the amount of available multimedia data has been spiraling. A personalized system is one that incorporates information about the user into its data processing part (e.g., a particular user taste for a movie genre). A context-aware system, in contrast, takes into account dynamic aspects of the user context when processing the data (e.g., location and time where/when a user issues a query). Today's user-adaptive systems often incorporate both aspects. Particularly focusing on the music domain, this article gives an overview of different aspects we deem important to build personalized music retrieval systems. In this vein, we first give an overview of factors that influence the human perception of music. We then propose and discuss various requirements for a personalized, user-aware music retrieval system. Eventually, the state-of-the-art in building such systems is reviewed, taking in particular aspects of "similarity" and "serendipity" into account.

[1]  Matthias Jarke,et al.  Adaptive Multimodal Exploration of Music Collections , 2009, ISMIR.

[2]  Peter Knees,et al.  A music search engine built upon audio-based and web-based similarity measures , 2007, SIGIR.

[3]  Jin Ha Lee,et al.  How Similar Is Too Similar?: Exploring Users' Perceptions of Similarity in Playlist Evaluation , 2011, ISMIR.

[4]  Perfecto Herrera,et al.  Automatic Detection of Emotion in Music: Interaction with Emotionally Sensitive Machines , 2018 .

[5]  Sebastian Streich,et al.  Music complexity: a multi-faceted description of audio content , 2007 .

[6]  Davide Rocchesso,et al.  Sound to Sense - Sense to Sound: A state of the art in Sound and Music Computing , 2008 .

[7]  Peter Knees,et al.  Context-based Music Similarity Estimation , 2009 .

[8]  Andreas Nürnberger,et al.  Weighted Self-Organizing Maps: Incorporating User Feedback , 2003, ICANN.

[9]  Daniel Wolff,et al.  Combining Sources of Description for Approximating Music Similarity Ratings , 2011, Adaptive Multimedia Retrieval.

[10]  Justin Donaldson,et al.  Uncovering Affinity of Artists to Multiple Genres from Social Behaviour Data , 2008, ISMIR.

[11]  Sebastian Stober,et al.  Adaptive Distance Measures for Exploration and Structuring of Music Collections , 2011, Semantic Audio.

[12]  Yuval Shavitt,et al.  Song Clustering Using Peer-to-Peer Co-occurrences , 2009, 2009 11th IEEE International Symposium on Multimedia.

[13]  Pierre-Yves Rolland.  Adaptive User Modeling in a Content-Based Music Retrieval System , 2001 .

[14]  François Pachet,et al.  Improving Timbre Similarity : How high’s the sky ? , 2004 .

[15]  Gert R. G. Lanckriet,et al.  Smarter than Genius? Human Evaluation of Music Recommender Systems , 2009, ISMIR.

[16]  G. H. Wakefield,et al.  To catch a chorus: using chroma-based representations for audio thumbnailing , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[17]  Beth Logan,et al.  Mel Frequency Cepstral Coefficients for Music Modeling , 2000, ISMIR.

[18]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[19]  Andreas Nürnberger,et al.  Towards User-Adaptive Structuring and Organization of Music Collections , 2008, Adaptive Multimedia Retrieval.

[20]  François Pachet,et al.  ON THE USE OF ZERO-CROSSING RATE FOR AN APPLICATION OF CLASSIFICATION OF PERCUSSIVE SOUNDS , 2000 .

[21]  Peter Knees,et al.  Building an Interactive Next-Generation Artist Recommender Based on Automatically Derived High-Level Concepts , 2007, 2007 International Workshop on Content-Based Multimedia Indexing.

[22]  Fabio Vignoli,et al.  A Music Retrieval System Based on User Driven Similarity and Its Evaluation , 2005, ISMIR.

[23]  Yehuda Koren,et al.  Factorization meets the neighborhood: a multifaceted collaborative filtering model , 2008, KDD.

[24]  Andreas Nürnberger,et al.  Supporting Folk-Song Research by Automatic Metric Learning and Ranking , 2009, ISMIR.

[25]  J. Jośe.  A HIERARCHICAL APPROACH TO AUTOMATIC MUSICAL GENRE CLASSIFICATION , 2003 .

[26]  Simon Dixon,et al.  Ethnographic Observations of Musicologists at the British Library: Implications for Music Information Retrieval , 2011, ISMIR.

[27]  Steve Lawrence,et al.  Inferring Descriptions and Similarity for Music from Community Metadata , 2002, ICMC.

[28]  Kilian Q. Weinberger,et al.  ISMIR 2008 – Session 3a – Content-Based Retrieval, Categorization and Similarity 1 LEARNING A METRIC FOR MUSIC SIMILARITY , 2022 .

[29]  Markus Schedl,et al.  Country of origin determination via Web mining techniques , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[30]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[31]  J. Stephen Downie,et al.  "The Pain, the Pain": Modelling Music Information Behavior and the Songs We Hate , 2005, ISMIR.

[32]  George Tzanetakis,et al.  Musical genre classification of audio signals , 2002, IEEE Trans. Speech Audio Process..

[33]  Emilia Gómez Gutiérrez,et al.  Tonal description of music audio signals , 2006 .

[34]  Elias Pampalk,et al.  Computational Models of Music Similarity and their Application in Music Information Retrieval , 2006 .

[35]  François Pachet,et al.  Musical data mining for electronic music distribution , 2001, Proceedings First International Conference on WEB Delivering of Music. WEDELMUSIC 2001.

[36]  Geoffrey E. Hinton,et al.  Neighbourhood Components Analysis , 2004, NIPS.

[37]  Mark B. Sandler,et al.  A Semantic Space for Music Derived from Social Tags , 2007, ISMIR.

[38]  Jens Grivolla,et al.  Multimodal Music Mood Classification Using Audio and Lyrics , 2008, 2008 Seventh International Conference on Machine Learning and Applications.

[39]  Stephan Baumann,et al.  AN ECOLOGICAL APPROACH TO MULTIMODAL SUBJECTIVE MUSIC SIMILARITY PERCEPTION , 2004 .

[40]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[41]  Nina Reeves,et al.  An ethnographic study of music information seeking: implications for the design of a music digital library , 2003, 2003 Joint Conference on Digital Libraries, 2003. Proceedings..

[42]  Òscar Celma,et al.  Foafing the Music: A Music Recommendation System based on RSS Feeds and User Preferences , 2005, ISMIR.

[43]  S. Stober,et al.  A Survey on the Acceptance of Listening Context Logging for MIR Applications , 2009 .

[44]  Emilia Gómez,et al.  A content-based system for music recommendation and visualization of user preferences working on semantic notions , 2011, 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI).

[45]  Emilia Gómez,et al.  The Musical Avatar: a visualization of musical preferences by means of audio content description , 2010, Audio Mostly Conference.

[46]  Sally Jo Cunningham,et al.  A user-centered design of a personal digital library for music exploration , 2010, JCDL '10.

[47]  Thorsten Joachims,et al.  A support vector method for multivariate performance measures , 2005, ICML.

[48]  Andreas Rauber,et al.  Rhyme and Style Features for Musical Genre Classification by Song Lyrics , 2008, ISMIR.

[49]  Marc Leman,et al.  Content-Based Music Information Retrieval: Current Directions and Future Challenges , 2008, Proceedings of the IEEE.

[50]  Roger B. Dannenberg,et al.  TagATune: A Game for Music and Sound Annotation , 2007, ISMIR.

[51]  Peter Knees,et al.  Assigning and Visualizing Music Genres by Web-based Co-Occurrence Analysis , 2006, ISMIR.

[52]  Markus Schedl,et al.  Three web-based heuristics to determine a person's or institution's country of origin , 2010, SIGIR '10.

[53]  Peter Knees,et al.  Searching for Music Using Natural Language Queries and Relevance Feedback , 2007, Adaptive Multimedia Retrieval.

[54]  Barry Vercoe,et al.  Using User Models in Music Information Retrieval Systems , 2000, ISMIR.

[55]  Peter Knees,et al.  Artist Classification with Web-Based Data , 2004, ISMIR.

[56]  Daniel P. W. Ellis,et al.  Signal Processing for Music Analysis , 2011, IEEE Journal of Selected Topics in Signal Processing.

[57]  Enric Guaus i Termens.  Audio content processing for automatic music genre classification: descriptors, databases, and classifiers , 2010 .

[58]  Xavier Serra,et al.  Content Processing of Music Audio Signals , 2008 .

[59]  Tomer Hertz,et al.  Learning a Mahalanobis Metric from Equivalence Constraints , 2005, J. Mach. Learn. Res..

[60]  Gert R. G. Lanckriet,et al.  Learning Similarity from Collaborative Filters , 2010, ISMIR.

[61]  Gert R. G. Lanckriet,et al.  Heterogeneous Embedding for Subjective Artist Similarity , 2009, ISMIR.

[62]  Gert R. G. Lanckriet,et al.  Metric Learning to Rank , 2010, ICML.

[63]  Malcolm Slaney,et al.  Construction and evaluation of a robust multifeature speech/music discriminator , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[64]  Bing Liu,et al.  Web Data Mining: Exploring Hyperlinks, Contents, and Usage Data , 2006, Data-Centric Systems and Applications.

[65]  Bernd Ludwig,et al.  InCarMusic: Context-Aware Music Recommendations in a Car , 2011, EC-Web.

[66]  Qiang Yang,et al.  User language model for collaborative personalized search , 2009, TOIS.

[67]  Peter Knees,et al.  The Quest for Ground Truth in Musical Artist Tagging in the Social Web Era , 2007, ISMIR.

[68]  J. Korst,et al.  Efficient Lyrics Retrieval and Alignment , .

[69]  Andreas Nürnberger,et al.  An Experimental Comparison of Similarity Adaptation Approaches , 2011, Adaptive Multimedia Retrieval.

[70]  Elias Pampalk,et al.  Content-based organization and visualization of music archives , 2002, MULTIMEDIA '02.

[71]  Graham E. Poliner,et al.  Melody Transcription From Music Audio: Approaches and Evaluation , 2007, IEEE Transactions on Audio, Speech, and Language Processing.

[72]  Markus Schedl,et al.  Intelligent structuring and exploration of digital music collections , 2005 .

[73]  Peter Knees,et al.  Multiple Lyrics Alignment: Automatic Retrieval of Song Lyrics , 2005, ISMIR.

[74]  Daniel P. W. Ellis,et al.  Please Scroll down for Article Journal of New Music Research a Web-based Game for Collecting Music Metadata a Web-based Game for Collecting Music Metadata , 2022 .

[75]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[76]  Daniel P. W. Ellis,et al.  Toward Evaluation Techniques for Music Similarity , 2003, SIGIR 2003.

[77]  Andreas Nürnberger,et al.  Everything in its right place ? Learning a user ’ s view of a music collection , 2009 .

[78]  Erik Duval,et al.  A Web-based Approach to Determine the Origin of an Artist , 2009, ISMIR.

[79]  Bingjun Zhang,et al.  CompositeMap: a novel framework for music similarity measure , 2009, SIGIR.

[80]  Daniel P. W. Ellis,et al.  The Quest for Ground Truth in Musical Artist Similarity , 2002, ISMIR.

[81]  J. Stephen Downie,et al.  Everyday Life Music Information-Seeking Behaviour of Young Adults , 2006, ISMIR.

[82]  Alexander Lerch,et al.  A HIERARCHICAL APPROACH TO AUTOMATIC MUSICAL GENRE CLASSIFICATION , 2003 .

[83]  Gert R. G. Lanckriet,et al.  A Game-Based Approach for Collecting Semantic Annotations of Music , 2007, ISMIR.

[84]  Ishwar K. Sethi,et al.  Classification of general audio data for content-based retrieval , 2001, Pattern Recognit. Lett..

[85]  Jin Ha Lee.  Analysis of user needs and information features in natural language queries seeking music information , 2010 .

[86]  Edith Law,et al.  Input-agreement: a new mechanism for collecting data using human computation games , 2009, CHI.

[87]  Peter Knees,et al.  A WEB-BASED APPROACH TO ASSESSING ARTIST SIMILARITY USING CO-OCCURRENCES , 2005 .

[88]  Bingjun Zhang,et al.  CompositeMap: a novel music similarity measure for personalized multimodal music search , 2009, MM '09.

[89]  L. Duchene,et al.  An Optimal Transformation for Discriminant and Principal Component Analysis , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[90]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[91]  Peter Knees,et al.  “Reinventing the Wheel”: A Novel Approach to Music Player Interfaces , 2007, IEEE Transactions on Multimedia.

[92]  Eyke Hüllermeier,et al.  Learning Similarity Functions from Qualitative Feedback , 2008, ECCBR.

[93]  Beth Logan,et al.  Semantic analysis of song lyrics , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[94]  George A. Tsihrintzis,et al.  MUSIPER: a system for modeling music similarity perception based on objective feature subset selection , 2008, User Modeling and User-Adapted Interaction.