Intelligent User Interfaces for Music Discovery: The Past 20 Years and What's to Come

Providing means to assist the user in finding music is one of the original motivations underlying the research field known as Music Information Retrieval (MIR). Therefore, already the first edition of ISMIR in the year 2000 called for papers addressing the topic of “User interfaces for music IR”. Since then, the way humans interact with technology to access and listen to music has substantially changed, not least driven by the advances of MIR and related research fields such as machine learning and recommender systems. In this paper, we reflect on the evolution of MIR-driven user interfaces for music browsing and discovery over the past two decades. We argue that three major developments have transformed and shaped user interfaces during this period, each connected to a phase of new listening practices: first, connected to personal music collections, intelligent audio processing and content description algorithms that facilitate the automatic organization of repositories and finding music according to sound qualities; second, connected to collective web platforms, the exploitation of user-generated metadata pertaining to semantic descriptions; and third, connected to streaming services, the collection of online music interaction traces on a large scale and their exploitation in recommender systems. We review and contextualize work from ISMIR and related venues from all three phases and extrapolate current developments to outline possible scenarios of music recommendation and listening interfaces of the future.

[1]  Youngmoo E. Kim,et al.  MoodSwings: A Collaborative Game for Music Mood Label Collection , 2008, ISMIR.

[2]  Paul Lamere,et al.  Using 3D Visualizations to Explore and Discover Music , 2007, ISMIR.

[3]  Malcolm Slaney,et al.  Web-Scale Multimedia Analysis: Does Content Matter? , 2011, IEEE MultiMedia.

[4]  Markus Schedl,et al.  On the Interrelation Between Listener Characteristics and the Perception of Emotions in Classical Orchestra Music , 2018, IEEE Transactions on Affective Computing.

[5]  Matthias Jarke,et al.  Adaptive Multimodal Exploration of Music Collections , 2009, ISMIR.

[6]  Mohamed Sordo.  Semantic annotation of music collections: A computational approach , 2012 .

[7]  Dietmar Jannach,et al.  Automated Generation of Music Playlists: Survey and Experiments , 2014, ACM Comput. Surv..

[8]  Peter Knees,et al.  Music Recommender Systems , 2015, Recommender Systems Handbook.

[9]  Bruce Ferwerda,et al.  Personality Traits and Music Genres: What Do People Prefer to Listen To? , 2017, UMAP.

[10]  Thierry Bertin-Mahieux,et al.  Automatic Generation of Social Tags for Music Recommendation , 2007, NIPS.

[11]  Carles Fernandes Julià,et al.  SongExplorer: A Tabletop Application for Exploring Large Collections of Songs , 2009, ISMIR.

[12]  Hiromu Yakura,et al.  FocusMusicRecommender: A System for Recommending Music to Listen to While Working , 2018, IUI.

[13]  Fabio Vignoli,et al.  Visual Playlist Generation on the Artist Map , 2005, ISMIR.

[14]  Masataka Goto,et al.  Grand Challenges in Music Information Research , 2012, Multimodal Music Processing.

[15]  Markus Schedl,et al.  Exploring Geospatial Music Listening Patterns in Microblog Data , 2012, Adaptive Multimedia Retrieval.

[16]  Douglas Turnbull,et al.  Using Artist Similarity to Propagate Semantic Information , 2009, ISMIR.

[17]  S. Gosling,et al.  PERSONALITY PROCESSES AND INDIVIDUAL DIFFERENCES The Do Re Mi’s of Everyday Life: The Structure and Personality Correlates of Music Preferences , 2003 .

[18]  Peter Knees,et al.  “Reinventing the Wheel”: A Novel Approach to Music Player Interfaces , 2007, IEEE Transactions on Multimedia.

[19]  Iván Cantador,et al.  Alleviating the new user problem in collaborative filtering by exploiting personality information , 2016, User Modeling and User-Adapted Interaction.

[20]  Masataka Goto,et al.  Songrium: Browsing and Listening Environment for Music Content Creation Community , 2015 .

[21]  Peter Knees,et al.  An innovative three-dimensional user interface for exploring music collections enriched , 2006, MM '06.

[22]  Òscar Celma,et al.  Music Recommendation and Discovery - The Long Tail, Long Fail, and Long Play in the Digital Music Space , 2010 .

[23]  Erik Duval,et al.  A Web-based Approach to Determine the Origin of an Artist , 2009, ISMIR.

[24]  Shankar Vembu,et al.  A Self-Organizing Map Based Knowledge Discovery for Music Recommendation Systems , 2004, CMMR.

[25]  Colin Raffel,et al.  A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music , 2018, ICML.

[26]  Matthew E. P. Davies,et al.  AutoMashUpper: Automatic Creation of Multi-Song Music Mashups , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[27]  Peter Knees,et al.  One-touch access to music on mobile devices , 2007, MUM.

[28]  Masataka Goto,et al.  Instrudive: A Music Visualization System Based on Automatically Recognized Instrumentation , 2018, ISMIR.

[29]  Masataka Goto.  Active Music Listening Interfaces Based on Signal Processing , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[30]  Malcolm Slaney,et al.  Similarity Based on Rating Data , 2007, ISMIR.

[31]  Masahiro Takatsuka,et al.  Spherical self-organizing map using efficient indexed geodesic data structure , 2006, Neural Networks.

[32]  Arthur Flexer,et al.  FM4 SOUNDPARK AUDIO-BASED MUSIC RECOMMENDATION IN EVERYDAY USE , 2009 .

[33]  Daniel P. W. Ellis,et al.  Automatic Record Reviews , 2004, ISMIR.

[34]  J. Stephen Downie,et al.  The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future , 2004, Computer Music Journal.

[35]  Gerhard Widmer,et al.  Playlist Generation using Start and End Songs , 2008, ISMIR.

[36]  Masataka Goto,et al.  MusicSun: A New Approach to Artist Recommendation , 2007, ISMIR.

[37]  Gert R. G. Lanckriet,et al.  The Natural Language of Playlists , 2011, ISMIR.

[38]  Roderick Murray-Smith,et al.  Design and Evaluation of a Probabilistic Music Projection Interface , 2015, ISMIR.

[39]  Douglas Keislar,et al.  Content-Based Classification, Search, and Retrieval of Audio , 1996, IEEE Multim..

[40]  Peter Knees,et al.  Large-scale music exploration in hierarchically organized landscapes using prototypicality information , 2011, ICMR '11.

[41]  Daniel P. W. Ellis,et al.  Please Scroll down for Article Journal of New Music Research a Web-based Game for Collecting Music Metadata a Web-based Game for Collecting Music Metadata , 2022 .

[42]  Peter Knees,et al.  nepDroid: an intelligent mobile music player , 2012, ICMR '12.

[43]  Robin Burke,et al.  Context-aware music recommendation based on latenttopic sequential patterns , 2012, RecSys.

[44]  Stefan Leitich,et al.  Globe of Music - Music Library Visualization Using Geosom , 2007, ISMIR.

[45]  Mario Nöcker,et al.  Databionic Visualization of Music Collections According to Perceptual Distance , 2005, ISMIR.

[46]  Dietmar Jannach,et al.  Effects of recommendations on the playlist creation behavior of users , 2019, User Modeling and User-Adapted Interaction.

[47]  S. Stober,et al.  MusicGalaxy - An Adaptive User-Interface for Exploratory Music Retrieval , 2010 .

[48]  Masataka Goto,et al.  Musicream: Integrated Music-Listening Interface for Active, Flexible, and Unexpected Encounters with Musical Pieces , 2009, J. Inf. Process..

[49]  Peter Knees,et al.  A survey of music similarity and recommendation from music context data , 2013, ACM Trans. Multim. Comput. Commun. Appl..

[50]  Guandong Xu,et al.  Exploring user emotion in microblogs for music recommendation , 2015, Expert Syst. Appl..

[51]  Andrew M. Dai,et al.  Music Transformer: Generating Music with Long-Term Structure , 2018, ICLR.

[52]  Thierry Bertin-Mahieux,et al.  Autotagger: A Model for Predicting Social Tags from Acoustic Features on Large Music Databases , 2008 .

[53]  David S. Rosenblum,et al.  Context-aware mobile music recommendation for daily activities , 2012, ACM Multimedia.

[54]  Douglas Eck,et al.  Learning Latent Representations of Music to Generate Interactive Musical Palettes , 2018, IUI Workshops.

[55]  Masataka Goto,et al.  LyricsRadar: A Lyrics Retrieval System Based on Latent Topics of Lyrics , 2014, ISMIR.

[56]  Hiromasa Fujihara,et al.  A Modeling of Singing Voice Robust to Accompaniment Sounds and Its Application to Singer Identification and Vocal-Timbre-Similarity-Based Music Information Retrieval , 2010, IEEE Transactions on Audio, Speech, and Language Processing.

[57]  Nava Tintarev,et al.  A Diversity Adjusting Strategy with Personality for Music Recommendation , 2018, IntRS@RecSys.

[58]  Andreas Rauber,et al.  Hierarchical Clustering of Document Archives with the Growing Hierarchical Self-Organizing Map , 2001, ICANN.

[59]  Gert R. G. Lanckriet,et al.  User-centered design of a social game to tag music , 2009, HCOMP '09.

[60]  Mark Levy,et al.  3D INTERACTIVE ENVIRONMENT FOR MUSIC COLLECTION NAVIGATION , 2008 .

[61]  Masataka Goto,et al.  MusicRainbow: A New User Interface to Discover Artists Using Audio-based Similarity and Web-based Labeling , 2006, ISMIR.

[62]  George Tzanetakis,et al.  Automatic Musical Genre Classification of Audio Signals , 2001, ISMIR.

[63]  Gerhard Widmer,et al.  Exploring Music Collections by Browsing Different Views , 2004, Computer Music Journal.

[64]  Paul Lamere,et al.  Social Tagging and Music Information Retrieval , 2008 .

[65]  Razvan Pascanu,et al.  Contextual tag inference , 2011, TOMCCAP.

[66]  Masataka Goto,et al.  Music Interfaces Based on Automatic Music Signal Analysis: New Ways to Create and Listen to Music , 2019, IEEE Signal Processing Magazine.

[67]  Gert R. G. Lanckriet,et al.  Towards musical query-by-semantic-description using the CAL500 data set , 2007, SIGIR.

[68]  Masataka Goto,et al.  Lyric Jumper: A Lyrics-Based Music Exploratory Web Service by Modeling Lyrics Generative Process , 2017, ISMIR.

[69]  Katrien Verbert,et al.  Effects of personal characteristics on music recommender systems with different levels of controllability , 2018, RecSys.

[70]  Denis Parra,et al.  Moodplay: Interactive Mood-based Music Discovery and Recommendation , 2016, UMAP.

[71]  Marc Leman,et al.  Content-Based Music Information Retrieval: Current Directions and Future Challenges , 2008, Proceedings of the IEEE.

[72]  Roger B. Dannenberg,et al.  TagATune: A Game for Music and Sound Annotation , 2007, ISMIR.

[73]  Peter Knees,et al.  A music information system automatically generated via Web content mining techniques , 2011, Inf. Process. Manag..

[74]  Yosuke Takashima,et al.  Melody Retrieval with Humming , 1993, ICMC.