Concerning the Neuronal Code

The central problem with understanding brain and mind is the neural code issue: understanding the matter of our brain as basis for the phenomena of our mind. The richness with which our mind represents our environment, the parsimony of genetic data, the tremendous efficiency with which the brain learns from scant sensory input and the creativity with which our mind constructs mental worlds all speak in favor of mind as an emergent phenomenon. This raises the further issue of how the neural code supports these processes of organization. The central point of this communication is that the neural code has the form of structured net fragments that are formed by network self-organization, activate and de-activate on the functional time scale, and spontaneously combine to form larger nets with the same basic structure. The Mind-Body Problem While I am writing this, I am sitting on a sun-bathed terrace surrounded by a concrete piece of reality. Of course this reality, to the extent that it is accessible to me, is a construct of my brain: whatever the brain doesn’t represent cannot touch me. In the normal course of affairs this figment of our brain is what we take as the reality per se. Only occasionally are we aware that the world out there and its reflection in us are fundamentally different. Apart from basic matters of principle, we are on the one hand always restricted to a small part of the world, being limited to a moment Journal of Cognitive Science 19-4:511-550, 2018 ©2018 Institute for Cognitive Science, Seoul National University 512 C. von der Malsburg and a place, limited moreover by our attention, which reflects only part of what is perceptible. On the other hand our mind richly complements what is given by our senses with valuations and imaginations: the situation means something to us in the emotional sense and in the sense of opportunities to act, and our imagination can liberate us from place and time, transporting us mentally into real though distant, or possible or fanciful situations. On closer inspection it isn’t even possible to make out a clear line between immediate reality and imagination. We receive through our senses only scant and incomplete signals, shadows on the wall of Plato’s cave. To construct from these a reality is possible only with the help of constitutional assumptions (Kant’s a priori), with the help of masses of memory traces accumulated over the years (as emphasized by the empiricists), and with the help of extensive mental construction processes. These construction processes are usually subconscious. One should, however, think of the analogous and richly documented thought processes of mathematicians and scientists when constructing their mental edifices (like algebra or geometry, or the construction of geologic history out of myriads of single observations) in order to appreciate the great importance of mental processes for the fabrication of our inner reality. For essential thinkers of the 17th century the nature of the inner and of the (imagined) outer reality — Descartes’ res cogitans and res extensa — was so different that Leibniz couldn’t help attributing the collaboration between his mind formulating a letter and his hand writing that letter to divine intervention (“prestabilized harmony”). Spinoza, in contrast, saw brain and mind merely as different perspectives on the same thing. Today, the dualism of Descartes and Leibniz is seen as left behind and the accepted view is essentially that of Spinoza, although the two perspectives — that of scalpel and electrode on the one hand and introspection, psychophysics and psychology on the other — are still so different in the mind of present-day 513 Concerning the Neuronal Code scientists that they still are dualists for all intents and purposes. Isn’t now, after the preparatory work of the thinkers of the 17th, 18th and early 19th century and after the theoretical and experimental achievements of the late 19th, 20th and the incipient 21st century, isn’t now the time to solve the mind-body problem, to describe the common ground behind the two perspectives, so that their interrelation becomes clear? What shall we wait for? That the solution is going to be forced on us by the simultaneous recording of the activity of all neurons in the waking brain and the complete reconstruction of its synaptic connections? The wiring diagram of the hermaphrodite form of the nematode Caenorhabditis elegans has been known for many years, down to the naming of all of the 302 neurons and to the more than 8000 synapses (White et al., 1986) but the behavior of the worm is still not understood on this basis. As with perception, experimental facts about brain and mind, the analog to the sensory signal, are but shadows on the cave wall. In addition to experimental data it needs effective a priori assumptions and active mental constructions in order to decipher the process behind those two perspectives, brain and mind. Only this triad — data, assumptions and mental construction — can be successful. Essential breakthroughs of science, as Maxwell’s theory of electromagnetism or Boltzmann’s statistic-mechanical explanation of the thermal phenomena, emphasize especially the significance of the third component, mental construction. The Neural Code Issue I think this introduction makes it plain on what question we have to focus: how does the matter of my brain generate the phenomena in my mind? How can a finite material basis generate perceptions, the representation of reality and a universe of imaginations, how does it engender consciousness and the quality of feelings? This problem of the neural code has four 514 C. von der Malsburg inseparably intertwined aspects: 1. What is the nature of the state of the brain or mind at any given moment? 2. What is the nature of memory, whose structural fragments are so essential for the construction of the state? 3. What is the mechanism through which experience and thought form memory content? Answering those three questions must, of course, be guided through observations of material (physiological or anatomical) and mental (introspective, psychological and psychophysical) kind. Prominent is the striking contrast between the seamless unity we perceive in our inner being, in our consciousness, on the one hand and the articulation of our nervous system into a tremendous number of building elements on the other. The mind acts like a force that generates unity in this sack full of fleas. The nontrivial nature of this achievement is made clear by neurological malfunctions, which show how much our mind depends on the physical components of the brain. Malfunctions caused by local lesions have been, by the way, very 1 When trying to emulate the function of the brain in the computer, more prosaic versions of these questions offer themselves: What is the data format of brain state, what the data format of memory, and, what are the algorithms or what is the form of the dynamical processes by which those data objects are generated? I am mentioning the computer here because I do believe that the questions I am discussing are clearly drifting towards a crisis, towards the realization of artificial brains, designated by some as the Singularity. The signs of this crisis are, first, an excitement and expectation gripping all of society under the names of digital revolution or artificial intelligence, second, the availability of the necessary computing power (if still at much too high cost, which is to be reduced drastically by a further technical revolution), third (driven by said attitude of expectation) the emergence of broad application fields for artificial brains, and finally, closely related to that, the availability of gigantic investment funds. In my view the present situation may be likened to a huge body of water held back by a dam. All that is needed to break this dam is giving correct answers to the neural code questions. 515 Concerning the Neuronal Code helpful to appreciate mind as thematically articulated into modalities and sub-modalities and to identify these articulations with regions of the brain. This localization of contents and themes has eventually been refined down to the level of individual neurons, each of which, it seems, is connected to an elementary theme or feature, a stimulus to which it responds by firing or a motor pattern that it triggers. This observation gives part of an answer to the first of my questions, the nature of the neural code: the mind can be decomposed into atoms, into elementary symbols, and these correspond to neurons. It is very important, however, to realize that as with all reduction of complex phenomena to simple building blocks (such as Life to molecules), this decomposition accomplishes only part of the task of deciphering the neural code, the much more complex part having to deal with the assembly of those elements into mental phenomena. This is only possible in the context of answering the two other questions, especially those for the mechanisms generating state and memory. Important constraints come from observing the temporal behavior of the brain. Preparation of spontaneous actions starts a little more than a second before execution (Kornhuber and Deecke, 1965; Libet et al., 1983). The reaction of the brain to new stimuli takes a large part of a second. The transmission of a nervous pulse from one neuron to the next takes already some milliseconds. Well-prepared and standardized processing steps take so little time (Potter and Levy, 1969; Thorpe et al., 1996) that they seem to be realized through pure feed-forward waves of neural activation. The process in the brain is usually interpreted as a sequence of “psychological moments” (Block, 2014), each of which lasts one or two tenths of a second and can, when concentrated on, be reflected as conscious state. According to these temporal relations the bracket between elementary processing steps and whole-system reactions is very t