Hyperintentionality in schizophrenia: An fMRI study

The Sparsity of Simple Recurrent Networks in Musical Structure Learning Kat R. Agres (kra9@cornell.edu) Department of Psychology, Cornell University, 211 Uris Hall Ithaca, NY 14853 USA Jordan E. DeLong (jed245@cornell.edu) Department of Psychology, Cornell University, 211 Uris Hall Ithaca, NY 14853 USA Michael Spivey (spivey@ucmerced.edu) School of Social Sciences, Humanities, and Arts, UC Merced, P.O. Box 2039 Merced, CA 95344 USA Abstract Evidence suggests that sparse coding allows for a more efficient and effective way to distill structural information about the environment. Our simple recurrent network has demonstrated the same to be true of learning musical structure. Two experiments are presented that examine the learning trajectory of a simple recurrent network exposed to musical input. Both experiments compare the network’s internal representations to behavioral data: Listeners rate the network’s own novel musical output from different points along the learning trajectory. The first study focused on learning the tonal relationships inherent in five simple melodies. The developmental trajectory of the network was studied by examining sparseness of the hidden layer activations and the sophistication of the network’s compositions. The second study used more complex musical input and focused on both tonal and rhythmic relationships in music. We found that increasing sparseness of the hidden layer activations strongly correlated with the increasing sophistication of the network’s output. Interestingly, sparseness was not programmed into the network; this property simply arose from learning the musical input. We argue that sparseness underlies the network’s success: It is the mechanism through which musical characteristics are learned and distilled, and facilitates the network’s ability to produce more complex and stylistic novel compositions over time. Keywords: Musical structure; Simple Recurrent Network; Sparsity. Introduction Work in the field of neural network modeling has been useful in creating simulations of functional machinations of human cognition and behavior. While many different architectures and learning algorithms exist, this paper will primarily focus on Elman’s Simple Recurrent Network (SRN) (1990), which was originally developed to process and predict the appearance of sequentially ordered stimuli. This feature makes the SRN a prime candidate for processing the structure of music. Modeling aspects of musical composition has shown that networks can be trained to ‘compose’ music after learning from many examples. One such network is Mozer’s CONCERT, which is a modified Elman network that is trained on input stimuli and attempts to extract two key features: which notes in the scale are musically appropriate, and which of those selected notes is the best stylistically. While ratings of this network were better than compositions chosen from a transition table, they still were compositions only their mother could love (Mozer, 1994). Other approaches have included aspects such as evolutionary algorithms (Todd, 1999) as well as utilizing self-organizing networks instead of relying on learning rules (Page, 1993). While most studies have concentrated on the success of these networks’ compositions, the studies in this paper will concentrate on the internal state of the network as it learns. Additionally, subjects’ ratings of the network’s compositions over time will be examined, as well as other network statistics, such as sparse coding. Sparse coding is a strategy in which a population of neurons completely encode a stimulus using a low number of active units. Taken to an extreme, this strategy is similar to the concept of a ‘Grandmother Cell’ that responds robustly to only one stimulus, and thus has a very low average firing rate. This is directly in contrast to a fully distributed system where every neuron takes part in encoding every stimulus and fires an average of half of the time. Sparse coding allows for the possibility that as a distributed system learns the structure of the world, it begins encoding in a more sparse and efficient manner. The benefits of sparse coding have been reviewed in depth (Field, 1994; Olshausen and Field, 2004), however this paper will concentrate on two of them. The first reason is that encoding stimuli using fewer neurons allows for a complete representation without the biological demands of having every neuron firing (Levy, 1996). The second reason, which is highlighted in these studies, is that a sparse code develops in order to efficiently mirror the structure of the world. By examining the neural network architecture over the learning trajectory, we can investigate how network sparsity changes with experience. Given the conventions of Western tonality in music (e.g. common chord progressions), as outlined by music theory, the progression of tones in music