Decoding mental activity from neuroimaging data — the science behind mind-reading

At the interface of neuroscience and computer science, a new method of analysis has evolved. The idea of reading out mental activity from neuronal measurements has led to increasingly impressive feats of mind-reading. What sounds like science fiction is well-positioned to become a major tool in future brain research. 1. Understanding the brain The mechanisms by which the brain makes sense of the world rely on computational abilities far beyond any human piece of engineering. It has therefore been a fundamental goal in neuroscience to understand just how our central nervous system analyses sensory inputs, forms an internal cognitive state, and produces behavioural outputs. In recent decades, an overwhelming amount of evidence has accumulated that supports the early hypothesis that populations of nerve cells, or neurons, provide the basic functional processing unit of the brain. Neurons drive one another’s activity in highly interconnected groups that assemble and disperse on a millisecond scale, and the dynamics of these ensembles of cells are believed to give rise to the cognitive abilities of the brain. With more and more experimental and theoretical results coming in, many scientists believe that we may eventually solve the neural puzzle—and achieve a detailed understanding of the brain. But what exactly do we mean by an ‘understanding of the brain?’ Can we be said to ‘understand’ the brain once we have come up with a wiring diagram of its 100 billion neurons? Do we ‘understand’ it once we have written down the differential equations governing the dynamics of its 100 trillion synapses? Such approaches must inevitably fail. A more fruitful way of thinking about the question is: can we demonstrate how the inner workings of the brain relate to cognitive abilities? In other words: can we establish a mapping between structure and function, between brain and mind? The idea of reading out, or decoding, mental activity from neuronal measurements has been driving the formation of increasingly multidisciplinary research groups [8]. However, as neuroscientists and computer scientists team up, two fundamental challenges have become apparent. First, there are currently no methods available to record the activity of a larger number of individual nerve cells in an awake human being, let alone to obtain high-resolution whole-brain footage of neural activity. Second, the brain displays immense natural variability in structure, connectivity, and dynamics. Your brain is very different from your friends, and it is very different from itself as it was just a few minutes ago. Yet, more recently, increasingly marked advances in decoding have been achieved. “We show that [our] models make it possible to identify, from a large set of completely novel natural images, which specific image was seen by an observer,” Kendrick Kay and colleagues lately reported in the journal Nature [9]. Their ability to tell, by scanning someones brain, which picture they were looking at, is the result of a study carried out at the University of California, Berkeley. It is about decoding information from the visual system—the part of the brain that processes what we are currently looking at. And being able to tell which image was seen out of a fixed set of images is not the end of the story: “Our results suggest that it may soon be possible to reconstruct a picture of a persons visual experience from measurements of brain activity alone.” [9] The idea of engineering a general brain-reading device has long been stimulating researchers’ imaginations. Psychologists claim it could be used to investigate perception and consciousness [6, 7, 15]. Neurologists say it could be used to construct brain-computer interfaces for paralyzed patients [18]. Lawyers wonder whether it could be used for lie detection [1]. As basic research increasingly elucidates the neural mechanisms underlying cognition, we may begin to use this knowledge in reverse: to decipher a cognitive process from its neural correlates. Decoding relies on two techniques. First, neuroimaging has made it possible to obtain correlates of the summed activity of populations of neurons across the whole human brain [16]. Second, the theory of machine learning has given rise to powerful algorithms that are able to recognize patterns in measured brain activity, and associate them with mental states [13]. The combination of these two fields comes with many challenges, and results require extremely careful interpretation. But it opens up a treasury of exciting applications, and never before have we been so close to their realization. 2. What is the brain thinking about? Measuring neural activity using fMRI High-level phenomena such as memory or consciousness are difficult to localize: they emerge from the distributed activity of many parts of the brain. By contrast, more basic functional building blocks have been pinpointed to particular cortical areas. Sensory inputs, for example, are known to arrive in dedicated hierarchical structures of the brain including the visual cortex (seeing), the auditory cortex (hearing), and the somatosensory cortex (touching). Similarly, behavioural outputs are passed on to the spinal cord by an area referred to as the motor cortex. In between are association areas that effectively allow any sensory input to trigger any motor output. One technology in particular has fuelled these insights: functional magnetic resonance imaging (fMRI) makes it possible to record neural activity from the brain of a participant who is happily performing some kind of cognitive task. Neural activity is expressed in terms of increased signalling between nerve cells, which, in turn, leads to an increased demand in oxygen. As a result, the level of blood oxygenation rises. The precise details of the underlying cascade of biochemical events are not fully understood, but the effect is of immense use: an MRI scanner is able to pick up subtle changes in blood oxygenation as direct correlates of neural activation [16]. How can we employ this technology to infer something interesting about the brain? In a typical fMRI experiment, a participant lies in a large magnetic coil and is asked to watch a screen, listen to a sound, press some buttons, navigate in a 3D maze, or perform any other kind of task. In the same way as a digital camera divides up an image into a grid of small pixels, the MRI scanner divides up the brain into voxels, small cubes with a volume of, e.g., 3× 3× 3 mm. A complete recording then contains a time series of neural activity from each voxel throughout the duration of the experiment. In a cognitive neuroscience setting, for example, participants might be asked to play a gambling game in which they have to place a bet on either of two cards, and, by trial and error, adopt a successful strategy to maximize their winnings. Given their recorded neural activity, we might now begin by looking for those regions that display systematic differences in activity between periods when participants are at rest and periods when they are making a decision. Technically, we predict what the signal in a voxel should look like if the nerve cells in that voxel were concerned with decision making: low activity during rest, and high activity just before a decision. As a result of our analysis we may find various areas in the brain whose activity appears to follow our prediction: low blood oxygenation during rest, and high oxygenation just before a decision. We might then conclude that these regions are involved in the mental process of making a decision. There are many caveats associated with this kind of analysis and the interpretation of its results. Nevertheless, careful experimental design and the use of converging evidence have established fMRI as the method of choice for human brain research. 3. Decoding mental activity When trying to decode mental activity from neural recordings, the conventional analysis described above is modified in two ways. First, rather than predicting the time course of neural activity from a design matrix, we aim to predict parts of the design matrix from the time course of neural activity [8]. Second, rather than considering all voxels independently, we aim to understand how patterns of voxel activities jointly encode information. The first modification is important when the aim is prediction per se, that is, in applications such as lie detection. The second modification is key when it comes to inference on structurefunction mappings, that is, in basic research. In either case, we take a snapshot of the activity measured simultaneously at many locations in the brain, and map it onto a particular mental state. These states are often defined in terms of discrete classes, which, in the example above, could be labelled ‘rest’ and ‘decide.’ In this way, decoding can be viewed as classification, a key problem studied by a branch of computer science known as machine learning [13]. How does it work? The learning methodology. It takes no more than a few years until children can easily recognize digits and letters, or detect a single female face in a series of male ones. For computers, however, tasks of pattern recognition are among the most difficult ones. It is unknown how to teach a machine to flawlessly recognize faces or separate personal emails from unwanted spam because no mathematical model of the problem is available, or its implementation is compu-