Auditory but Not Audiovisual Cues Lead to Higher Neural Sensitivity to the Statistical Regularities of an Unfamiliar Musical Style

It is still a matter of debate whether visual aids improve learning of music. In a multisession study, we investigated the neural signatures of novel music sequence learning with or without aids (auditory-only: AO, audiovisual: AV). During three training sessions on three separate days, participants (nonmusicians) reproduced (note by note on a keyboard) melodic sequences generated by an artificial musical grammar. The AV group (n = 20) had each note color-coded on screen, whereas the AO group (n = 20) had no color indication. We evaluated learning of the statistical regularities of the novel music grammar before and after training by presenting melodies ending on correct or incorrect notes and by asking participants to judge the correctness and surprisal of the final note, while EEG was recorded. We found that participants successfully learned the new grammar. Although the AV group, as compared to the AO group, reproduced longer sequences during training, there was no significant difference in learning between groups. At the neural level, after training, the AO group showed a larger N100 response to low-probability compared with high-probability notes, suggesting an increased neural sensitivity to statistical properties of the grammar; this effect was not observed in the AV group. Our findings indicate that visual aids might improve sequence reproduction while not necessarily promoting better learning, indicating a potential dissociation between sequence reproduction and learning. We suggest that the difficulty induced by auditory-only input during music training might enhance cognitive engagement, thereby improving neural sensitivity to the underlying statistical properties of the learned material.

[1]  T. Stanford,et al.  Superadditivity in multisensory integration: putting the computation in context. , 2007, Neuroreport.

[2]  M. Pearce Statistical learning and probabilistic prediction in music cognition: mechanisms of stylistic enculturation , 2018, Annals of the New York Academy of Sciences.

[3]  Stefan Koelsch,et al.  Processing Expectancy Violations during Music Performance and Perception: An ERP Study , 2010, Journal of Cognitive Neuroscience.

[4]  M. Tervaniemi,et al.  Representation of harmony rules in the human brain: Further evidence from event-related potentials , 2007, Brain Research.

[5]  Christo Pantev,et al.  Musical expertise is related to neuroplastic changes of multisensory nature within the auditory cortex , 2015, The European journal of neuroscience.

[6]  Marcus T. Pearce,et al.  Melodic pitch expectation interacts with neural responses to syntactic but not semantic violations , 2013, Cortex.

[7]  K. Squires,et al.  Variations in early and late event-related components of the auditory evoked potential with task difficulty. , 1983, Electroencephalography and clinical neurophysiology.

[8]  C. Bowles Music Activity Preferences of Elementary Students , 1998 .

[9]  William I. Abler Just Say a System for Remembering Finger Placement in Various Keys on the Violin , 2002 .

[10]  Claude Alain,et al.  Perceptual learning modulates sensory evoked response during vowel segregation. , 2003, Brain research. Cognitive brain research.

[11]  C. J. McGrath,et al.  Effect of exchange rate return on volatility spill-over across trading regions , 2012 .

[12]  E. Pothos Theories of artificial grammar learning. , 2007, Psychological bulletin.

[13]  R. Näätänen,et al.  Early selective-attention effect on evoked potential reinterpreted. , 1978, Acta psychologica.

[14]  D. Pisoni,et al.  ON SPOKEN LANGUAGE PROCESSING Progress Report No . 25 ( 2001-2002 ) Indiana University Some New Findings on Learning , Memory and Cognitive Processes in Deaf Children Following Cochlear Implantation , 2002 .

[15]  Carol A. Seger,et al.  Implicit learning. , 1994, Psychological bulletin.

[16]  Stefan Koelsch,et al.  Differences in Electric Brain Responses to Melodies and Chords , 2010, Journal of Cognitive Neuroscience.

[17]  Axel Mecklinger,et al.  Error and Deviance Processing in Implicit and Explicit Sequence Learning , 2008, Journal of Cognitive Neuroscience.

[18]  Stefan Koelsch,et al.  Under the hood of statistical learning: A statistical MMN reflects the magnitude of transitional probabilities in auditory sequences , 2016, Scientific Reports.

[19]  Yang Zhang,et al.  Learning Temporal Statistics for Sensory Predictions in Aging , 2016, Journal of Cognitive Neuroscience.

[20]  Hans-Jochen Heinze,et al.  Tactile stimulation and hemispheric asymmetries modulate auditory perception and neural responses in primary auditory cortex , 2013, NeuroImage.

[21]  D. Wessel,et al.  Learning and Liking an Artificial Musical System: Effects of Set Size and Repeated Exposure , 2008, Musicae scientiae : the journal of the European Society for the Cognitive Sciences of Music.

[22]  E. Newport,et al.  WORD SEGMENTATION : THE ROLE OF DISTRIBUTIONAL CUES , 1996 .

[23]  David B. Pisoni,et al.  Some Measures of Verbal and Spatial Working Memory in Eight- and Nine-Year-Old Hearing-Impaired Children with Cochlear Implants , 2001, Ear and hearing.

[24]  Job P. Lindsen,et al.  That note sounds wrong! Age-related effects in processing of musical expectation , 2017, Brain and Cognition.

[25]  Stefan Koelsch,et al.  Shared neural resources between music and language indicate semantic processing of musical tension-resolution patterns. , 2008, Cerebral cortex.

[26]  Christopher W. Robinson,et al.  When audition dominates vision: evidence from cross-modal statistical learning. , 2013, Experimental psychology.

[27]  Brenda Rapp,et al.  Principles of cross-modal competition: Evidence from deficits of attention , 2003, Psychonomic bulletin & review.

[28]  Morten H. Christiansen,et al.  Domain generality versus modality specificity: the paradox of statistical learning , 2015, Trends in Cognitive Sciences.

[29]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[30]  C. Ponton,et al.  Central Auditory Plasticity: Changes in the N1-P2 Complex after Speech-Sound Training , 2001, Ear and hearing.

[31]  Clay B. Holroyd,et al.  Frontal midline theta and N200 amplitude reflect complementary information about expectancy and outcome evaluation. , 2013, Psychophysiology.

[32]  Joan Y. Chiao,et al.  An Event-related fMRI Study of Artificial Grammar Learning in a Balanced Chunk Strength Design , 2004, Journal of Cognitive Neuroscience.

[33]  Ruth Stevens,et al.  Improving children's working memory and classroom performance , 2010 .

[34]  N. Kraus,et al.  Auditory training induces asymmetrical changes in cortical neural activity. , 2002, Journal of speech, language, and hearing research : JSLHR.

[35]  David Goodman,et al.  Performance Monitoring in the Anterior Cingulate is Not All Error Related: Expectancy Deviation and the Representation of Action-Outcome Associations , 2007, Journal of Cognitive Neuroscience.

[36]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[37]  Christopher W. Robinson,et al.  Existence of competing modality dominances , 2016, Attention, Perception, & Psychophysics.

[38]  I. Cross,et al.  Modelling unsupervised online-learning of artificial grammars: Linking implicit and statistical learning , 2014, Consciousness and Cognition.

[39]  Elizabeth K. Johnson,et al.  Statistical learning of tone sequences by human infants and adults , 1999, Cognition.

[40]  Kazuo Okanoya,et al.  On-line Assessment of Statistical Learning by Event-related Potentials , 2008, Journal of Cognitive Neuroscience.

[41]  Jessica A. Grahn,et al.  Neural correlates of audiovisual integration in music reading , 2016, Neuropsychologia.

[42]  Ying Liu,et al.  Visual Analysis and Lexical Access of Chinese Characters by Chinese as Second Language Readers , 2006 .

[43]  J. Cohen,et al.  P300, stimulus intensity, modality, and probability. , 1996, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[44]  Martin Rohrmeier,et al.  Implicit Learning and Acquisition of Music , 2012, Top. Cogn. Sci..

[45]  Annette Hohenberger,et al.  N1-P2: Neural markers of temporal expectation and response discrimination in interval timing. , 2019, Acta neurobiologiae experimentalis.

[46]  Scott P. Johnson,et al.  Visual statistical learning in infancy: evidence for a domain general learning mechanism , 2002, Cognition.

[47]  R. Aslin,et al.  Statistical learning of higher-order temporal structure from visual shape sequences. , 2002, Journal of experimental psychology. Learning, memory, and cognition.

[48]  Justyna Humięcka-Jakubowska,et al.  Sweet Anticipation : Music and , 2006 .

[49]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[50]  Marcus T. Pearce,et al.  Musical Syntax II: Empirical Perspectives , 2018 .

[51]  J. Saffran,et al.  Changing the tune: the structure of the input affects infants' use of absolute and relative pitch. , 2005, Developmental science.

[52]  Aaron R. Seitz,et al.  Sound Facilitates Visual Learning , 2006, Current Biology.

[53]  Marcia C. Linn,et al.  Linking Cognitive Science to Education: Generation and Interleaving Effects , 2005 .

[54]  Stefan Koelsch,et al.  The Role of Harmonic Expectancy Violations in Musical Emotions: Evidence from Subjective, Physiological, and Neural Responses , 2006, Journal of Cognitive Neuroscience.

[55]  Psyche Loui,et al.  Learning and Liking of Melody and Harmony: Further Studies in Artificial Grammar Learning , 2012, Top. Cogn. Sci..

[56]  Aaron R. Seitz,et al.  Benefits of multisensory learning , 2008, Trends in Cognitive Sciences.

[57]  Geraint A. Wiggins,et al.  The Role of Expectation and Probabilistic Learning in Auditory Boundary Perception: A Model Comparison , 2010, Perception.

[58]  M. Woldorff,et al.  Effects of attention on the neural processing of harmonic syntax in Western music. , 2005, Brain research. Cognitive brain research.

[59]  Peter M C Harrison,et al.  From learning to creativity: Identifying the behavioural and neural correlates of learning to predict human judgements of musical creativity , 2020, NeuroImage.

[60]  A. Engel,et al.  What is novel in the novelty oddball paradigm? Functional significance of the novelty P3 event-related potential as revealed by independent component analysis. , 2005, Brain research. Cognitive brain research.

[61]  Panagiotis D Bamidis,et al.  Musical expertise is related to altered functional connectivity during audiovisual integration , 2015, Proceedings of the National Academy of Sciences.

[62]  Sibylle C. Herholz,et al.  Audio-Tactile Integration and the Influence of Musical Training , 2014, PloS one.

[63]  C. Spence,et al.  The co-occurrence of multisensory competition and facilitation. , 2008, Acta psychologica.

[64]  Sibylle C. Herholz,et al.  Tones and numbers: A combined EEG–MEG study on the effects of musical expertise in magnitude comparisons of audiovisual stimuli , 2014, Human brain mapping.

[65]  Christo Pantev,et al.  Multisensory Integration during Short-term Music Reading Training Enhances Both Uni- and Multisensory Cortical Processing , 2014, Journal of Cognitive Neuroscience.

[66]  Sibylle C. Herholz,et al.  Statistical learning effects in musicians and non-musicians: An MEG study , 2012, Neuropsychologia.

[67]  M. Pearce,et al.  Electrophysiological correlates of melodic processing in congenital amusia , 2013, Neuropsychologia.

[68]  D. Wechsler Manual for the Wechsler Adult Intelligence Scale. , 1955 .

[69]  E. Brattico,et al.  Distinct neural responses to chord violations: A multiple source analysis study , 2011, Brain Research.

[70]  A. Friederici,et al.  Brain Indices of Music Processing: Nonmusicians are Musical , 2000, Journal of Cognitive Neuroscience.

[71]  John J. Foxe,et al.  Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. , 2004, Cerebral cortex.

[72]  A. Michalski,et al.  Expectation of an important event affects responses to irrelevant stimuli of different modalities. , 2000, Acta neurobiologiae experimentalis.

[73]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[74]  S. McAdams,et al.  Implicit learning of musical timbre sequences: statistical regularities confronted with acoustical (dis)similarities. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[75]  R. Bjork,et al.  The Generation Effect: Support for a Two-Factor Theory , 1988 .

[76]  P. Juslin Emotional Responses to Music , 2008 .

[77]  D Elliott,et al.  Examining the Specificity of Practice Hypothesis: Is Learning Modality Specific? , 2001, Research quarterly for exercise and sport.

[78]  Morten H. Christiansen,et al.  Sequential Expectations: The Role of Prediction-Based Learning in Language , 2010, Top. Cogn. Sci..

[79]  D. Bosnyak,et al.  Distributed auditory cortical representations are modified when non-musicians are trained at pitch discrimination with 40 Hz amplitude modulated tones. , 2004, Cerebral cortex.

[80]  Panagiotis D. Bamidis,et al.  Statistical learning of multisensory regularities is enhanced in musicians: An MEG study , 2018, NeuroImage.

[81]  Alice Mado Proverbio,et al.  Language switching mechanisms in simultaneous interpreters: an ERP study , 2004, Neuropsychologia.

[82]  J. L. Cantero,et al.  The time course of neural changes underlying auditory perceptual learning. , 2002, Learning & memory.

[83]  Morten H. Christiansen,et al.  Timing is everything: Changes in presentation rate have opposite effects on auditory and visual implicit statistical learning , 2011, Quarterly journal of experimental psychology.

[84]  Christo Pantev,et al.  Musical Expertise Induces Audiovisual Integration of Abstract Congruency Rules , 2012, The Journal of Neuroscience.

[85]  S. Koelsch,et al.  When the statistical MMN meets the physical MMN , 2019, Scientific Reports.

[86]  W. Dowling Emotion and Meaning in Music , 2008 .

[87]  T W Picton,et al.  N2 and automatic versus controlled processes. , 1986, Electroencephalography and clinical neurophysiology. Supplement.

[88]  Zoe Kourtzi,et al.  Decoding the future from past experience: learning shapes predictions in early visual cortex , 2015, Journal of neurophysiology.

[89]  Markus J. Hofmann,et al.  No one way ticket from orthography to semantics in recognition memory: N400 and P200 effects of associations , 2016, Brain Research.

[90]  Daniel Müllensiefen,et al.  The Musicality of Non-Musicians: An Index for Assessing Musical Sophistication in the General Population , 2014, PloS one.

[91]  R. Schmidt,et al.  New Conceptualizations of Practice: Common Principles in Three Paradigms Suggest New Concepts for Training , 1992 .

[92]  D. Västfjäll,et al.  Emotional responses to music: the need to consider underlying mechanisms. , 2008, The Behavioral and brain sciences.

[93]  Carla L. Hudson Kam,et al.  Humans Rapidly Learn Grammatical Structure in a New Musical Scale. , 2010, Music perception.

[94]  David B Pisoni,et al.  Effects of Early Musical Experience on Auditory Sequence Memory. , 2008, Empirical musicology review : EMR.

[95]  Timothy D. Lee,et al.  Distribution of Practice in Motor Skill Acquisition: Learning and Performance Effects Reconsidered , 1988 .

[96]  E. Bjork,et al.  Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. , 2011 .

[97]  Odelia Schwartz,et al.  Statistical Learning of Melodic Patterns Influences the Brain's Response to Wrong Notes , 2017, Journal of Cognitive Neuroscience.

[98]  S. Gathercole,et al.  Working Memory and Learning: A Practical Guide for Teachers , 2008 .

[99]  C. Spence Explaining the Colavita visual dominance effect. , 2009, Progress in brain research.

[100]  D. Leutner,et al.  Assessment of Cognitive Load in Multimedia Learning with Dual-Task Methodology: Auditory Load and Modality Effects , 2004 .

[101]  R N Aslin,et al.  Statistical Learning by 8-Month-Old Infants , 1996, Science.

[102]  C. Joyce,et al.  The face-sensitive N170 and VPP components manifest the same brain processes: The effect of reference electrode site , 2005, Clinical Neurophysiology.

[103]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[104]  M. Pearce,et al.  Predictive uncertainty in auditory sequence processing , 2014, Front. Psychol..

[105]  M. Rugg,et al.  Electrophysiology of Mind: Event-Related Brain Potentials and Cognition , 1995 .

[106]  Masato Yumoto,et al.  Statistical learning of music- and language-like sequences and tolerance for spectral shifts , 2015, Neurobiology of Learning and Memory.

[107]  Arthur S. Reber,et al.  Cognition Unawares. (Book Reviews: Implicit Learning and Tacit Knowledge. An Essay on the Cognitive Unconscious.) , 1996 .

[108]  Seung-Goo Kim,et al.  The Effect of Conditional Probability of Chord Progression on Brain Response: An MEG Study , 2011, PloS one.

[109]  Morten H. Christiansen,et al.  PSYCHOLOGICAL SCIENCE Research Article Statistical Learning Within and Between Modalities Pitting Abstract Against Stimulus-Specific Representations , 2022 .

[110]  R. Bjork Memory and metamemory considerations in the training of human beings. , 1994 .

[111]  F. Craik,et al.  Depth of processing and the retention of words , 1975 .

[112]  B. Kopp,et al.  Brain mechanisms of selective learning: event-related potentials provide evidence for error-driven learning in humans , 2000, Biological Psychology.

[113]  I. Cross,et al.  Incidental and online learning of melodic structure , 2011, Consciousness and Cognition.

[114]  Daniel M. Oppenheimer,et al.  Fortune favors the ( ): Effects of disfluency on educational outcomes , 2011, Cognition.

[115]  Marcus T. Pearce,et al.  The construction and evaluation of statistical models of melodic structure in music perception and composition , 2005 .

[116]  Jenny R. Saffran,et al.  Learning Harmony: The Role of Serial Statistics , 2009, Cogn. Sci..

[117]  C. Stevens,et al.  Sweet Anticipation: Music and the Psychology of Expectation, by David Huron . Cambridge, Massachusetts: MIT Press, 2006 , 2007 .

[118]  N. Chater,et al.  Transfer in artificial grammar learning : A reevaluation , 1996 .

[119]  The transfer effect in artificial grammar learning: Reappraising the evidence on the transfer of sequential dependencies. , 1999 .

[120]  Morten H. Christiansen,et al.  Modality-constrained statistical learning of tactile, visual, and auditory sequences. , 2005, Journal of experimental psychology. Learning, memory, and cognition.

[121]  S. Koelsch,et al.  Effects of Unexpected Chords and of Performer's Expression on Brain Responses and Electrodermal Activity , 2008, PloS one.

[122]  Ran El-Yaniv,et al.  On Prediction Using Variable Order Markov Models , 2004, J. Artif. Intell. Res..

[123]  María Herrojo Ruiz,et al.  Unsupervised statistical learning underpins computational, behavioural, and neural manifestations of musical expectation , 2010, NeuroImage.

[124]  C. Winstein,et al.  Learning–performance distinction and memory processes for motor skills: A focused review and perspective , 2012, Behavioural Brain Research.

[125]  Daniel M. Oppenheimer,et al.  Fortune favors the bold (and the Italicized): effects of disfluency on educational outcomes. , 2011, Cognition.

[126]  Ruth H. Maki,et al.  Increased processing enhances calibration of comprehension , 1990 .

[127]  C. Spence,et al.  Visual dominance and attention: The Colavita effect revisited , 2007, Perception & psychophysics.

[128]  A. Reber Implicit learning and tacit knowledge , 1993 .

[129]  Geraint A. Wiggins,et al.  Probabilistic models of expectation violation predict psychophysiological emotional responses to live concert music , 2013, Cognitive, Affective, & Behavioral Neuroscience.

[130]  A. Diederich,et al.  The race model inequality: interpreting a geometric measure of the amount of violation. , 2006, Psychological review.