Music of the 7Ts: Predicting and Decoding Multivoxel fMRI Responses with Acoustic, Schematic, and Categorical Music Features

Underlying the experience of listening to music are parallel streams of auditory, categorical, and schematic qualia, whose representations and cortical organization remain largely unresolved. We collected high-field (7T) fMRI data in a music listening task, and analyzed the data using multivariate decoding and stimulus-encoding models. Twenty subjects participated in the experiment, which measured BOLD responses evoked by naturalistic listening to twenty-five music clips from five genres. Our first analysis applied machine classification to the multivoxel patterns that were evoked in temporal cortex. Results yielded above-chance levels for both stimulus identification and genre classification–cross-validated by holding out data from multiple of the stimuli during model training and then testing decoding performance on the held-out data. Genre model misclassifications were significantly correlated with those in a corresponding behavioral music categorization task, supporting the hypothesis that geometric properties of multivoxel pattern spaces underlie observed musical behavior. A second analysis employed a spherical searchlight regression analysis which predicted multivoxel pattern responses to music features representing melody and harmony across a large area of cortex. The resulting prediction-accuracy maps yielded significant clusters in the temporal, frontal, parietal, and occipital lobes, as well as in the parahippocampal gyrus and the cerebellum. These maps provide evidence in support of our hypothesis that geometric properties of music cognition are neurally encoded as multivoxel representational spaces. The maps also reveal a cortical topography that differentially encodes categorical and absolute-pitch information in distributed and overlapping networks, with smaller specialized regions that encode tonal music information in relative-pitch representations.

[1]  D. Bendor,et al.  The neuronal representation of pitch in primate auditory cortex , 2005, Nature.

[2]  Marc Leman,et al.  The Cortical Topography of Tonal Structures Underlying Western Music , 2002, Science.

[3]  Marc Leman,et al.  Content-Based Music Information Retrieval: Current Directions and Future Challenges , 2008, Proceedings of the IEEE.

[4]  Stefan Pollmann,et al.  PyMVPA: a Python Toolbox for Multivariate Pattern Analysis of fMRI Data , 2009, Neuroinformatics.

[5]  Jyothi Swaroop Guntupalli Whole brain hyperalignment: Intersubject hyperalignment of local representational spaces , 2013 .

[6]  D. J. Hermes,et al.  Spectro-temporal characterization of auditory neurons: Redundant or necessary? , 1981, Hearing Research.

[7]  R. Zatorre,et al.  A role for the intraparietal sulcus in transforming musical pitch information. , 2010, Cerebral cortex.

[8]  Josh H McDermott,et al.  Music Perception, Pitch, and the Auditory System This Review Comes from a Themed Issue on Sensory Systems Edited Pitch Relations across Time—relative Pitch Relative Pitch—behavioral Evidence Neural Mechanisms of Relative Pitch Representation of Simultaneous Pitches— Chords and Polyphony Summary and , 2022 .

[9]  B. Delgutte,et al.  Neural correlates of the pitch of complex tones. I. Pitch and pitch salience. , 1996, Journal of neurophysiology.

[10]  Mikko Sams,et al.  Large-scale brain networks emerge from dynamic processing of musical timbre, key and rhythm , 2012, NeuroImage.

[11]  Tapani Ristaniemi,et al.  From Vivaldi to Beatles and back: Predicting lateralized brain responses to music , 2013, NeuroImage.

[13]  J. S. Guntupalli,et al.  Decoding neural representational spaces using multivariate pattern analysis. , 2014, Annual review of neuroscience.

[14]  Matthias Abend Cognitive Foundations Of Musical Pitch , 2016 .

[15]  G. H. Wakefield,et al.  To catch a chorus: using chroma-based representations for audio thumbnailing , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[16]  Oliver Speck,et al.  A high-resolution 7-Tesla fMRI dataset from complex natural stimulation with an audio movie , 2014, Scientific Data.

[17]  Yi Chen,et al.  Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): Random permutations and cluster size control , 2011, NeuroImage.

[18]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[19]  Henkjan Honing,et al.  Without it no music: beat induction as a fundamental musical trait , 2012, Annals of the New York Academy of Sciences.

[20]  Xavier Serra,et al.  Essentia: An Audio Analysis Library for Music Information Retrieval , 2013, ISMIR.

[21]  R. Shepard Circularity in Judgments of Relative Pitch , 1964 .

[22]  Dmitri Tymoczko,et al.  The Generalized Tonnetz , 2012 .

[23]  Richard Granger,et al.  Investigation of melodic contour processing in the brain using multivariate pattern-based fMRI , 2011, NeuroImage.

[24]  B. Delgutte,et al.  Neural correlates of the pitch of complex tones. II. Pitch shift, pitch ambiguity, phase invariance, pitch circularity, rate pitch, and the dominance region for pitch. , 1996, Journal of neurophysiology.

[25]  A. Aertsen,et al.  The Spectro-Temporal Receptive Field , 1981, Biological Cybernetics.

[26]  Vinoo Alluri,et al.  Capturing the musical brain with Lasso: Dynamic decoding of musical features from fMRI data , 2014, NeuroImage.

[27]  Michael A. Casey,et al.  High-resolution 7-Tesla fMRI data on the perception of musical , 2016 .

[28]  George Tzanetakis,et al.  Musical genre classification of audio signals , 2002, IEEE Trans. Speech Audio Process..

[29]  Nikolaus Kriegeskorte,et al.  Pattern-information analysis: From stimulus decoding to computational-model testing , 2011, NeuroImage.

[30]  Michael A. Casey,et al.  Population Codes Representing Musical Timbre for High-Level fMRI Categorization of Music Genres , 2011, MLINI.

[31]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[32]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.