The Aesthetiscope: Visualizing Aesthetic Readings of Text in Color Space

The traditional story understanding dogma in AI holds that there exists a singular, objective meaning implied by text, which can be uncovered by applying just the right logical inferences. But according to research into the cognition of reading, text can also be read, not objectively, but aesthetically. An aesthetic reading of text engages not only the agency of thought, but also intuition, sensation, and sentiment, and cultural interpretation. We have developed and implemented a basic computational model of aesthetic reading, which employs Carl Jung’s Modes of Interpretation idea to delegate the task of aesthetic interpretation out to several different textual analysis engines, each computing a different mode of interpreting text, such as generating sentimental evocations of a text (Feeling modality), or remembering visual imagery evoked by the text (Sensation modality). The outputs of the various interpretive modalities are merged and then their unification is achieved by mapping them into color space using theories of color psychology. We built the Aesthetiscope, an artwork whose grid of colors are dynamically generated from aesthetic readings of an inputted poem or song, to illustrate the power and potential of going beyond literal understandings of text. On Aesthetic Reading Much of the AI narrative understanding literature subscribes to the dogma that there exists a single rational method of interpreting text, and that resultant interpretations and inferences can always be reconciled into a single consistent world model. One branch of research notably departing from this dogma is concerned with creative reading (Moorman & Ram, 1994). According to the cognitively motivated theory of creative reading, textual understanding involves imagination, the suspension of disbelief, and the projection of inexact memories onto read situations; in contrast, the dogma says that textual understanding should be algorithmized simply as the rote invocation of inference rules. Moorman & Ram’s revolt against the grain of the classical AI narrative understanding literature emboldens us in our task of aesthetic reading, which is the topic of this paper. Copyright © 2005, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. Aesthetic reading is not reading purely for information. It is an emotionalized and personal reading, whereby the text’s primary purpose is to evoke aesthetic rumblings within the reader. Reading theorist Louise Rosenblatt states, “In aesthetic reading, the reader’s attention is centered directly on what he is living through during his relationship with that particular text” (Rosenblatt, 1978, p. 25); but this notion of “living through” can be quite a complex amalgamate of perceptions and sensations. In this work, we sought to develop a computational model of aesthetic reading that could be sophisticated enough to account for the diversity of the ingredients constituting the complex reader-to-text aesthetic relationship, while maintaining computational simplicity. Thus we developed our model around the inspiration of Carl Jung’s Modes of Interpretation (1921), a psychological theory that he put forth to account for the different possible ways that people interpret the world. According to the theory, there are four fundamental modes of interpreting reality: Thinking, Feeling, Sensation, and Intuition. To these four modes, we added a not-sofundamental fifth, Culturalizing, which incorporates Roland Barthes’ thesis (1964) that people also interpret the world through the optics of our culture’s values system. Also, for practical considerations, our work means the Sensation mode to refer solely to the remembrance of visual images. Whereas objective reading relies primarily on the Thinking mode; aesthetic reading invites a reader to employ many, or all, of the Modes of Interpretation to engage with the text, each mode producing some evocations; and we can think of the sum of all produced evocations as the aesthetic interpretation of the text. Based on this simple premise, we developed the following computational model of aesthetic reading: The aesthetic interpretation of a text can be computed by the application of all five Modes of Interpretation to a text. Each mode has its own logic for operation, but given some input text, each is able to output a set of aesthetic evocations, which in our work, is either a set of keywords, or a palette of colors. Ultimately though, even these keywords are translated into color space using some psycho-physiological heuristics which we have developed (more on this later). By combining the weighted outputs of the five Modes of Interpretation into a final palette, an abstract color mosaic is generated, and we say that this is a visualization of the aesthetic reading of the input text; we call this AI-based artwork the Aesthetiscope, as it is able to visualize aesthetic readings of various text like song lyrics and poems within the psychologically and emotionally evocative universe of colors. The rest of this paper is structured as follows. First, we present an overview of the Aesthetiscope as an art installation and explain its design rationale. Second, we discuss the technical mechanisms for computing aesthetic readings of text. Third, we reveal the mechanism for mapping the aesthetic evocations outputted by each Mode of Interpretation into color space. We conclude by discussing some redesigns we have made to the Aesthetiscope and reflecting upon the collected experiences of those who have interacted with the installation.color mosaic is generated, and we say that this is a visualization of the aesthetic reading of the input text; we call this AI-based artwork the Aesthetiscope, as it is able to visualize aesthetic readings of various text like song lyrics and poems within the psychologically and emotionally evocative universe of colors. The rest of this paper is structured as follows. First, we present an overview of the Aesthetiscope as an art installation and explain its design rationale. Second, we discuss the technical mechanisms for computing aesthetic readings of text. Third, we reveal the mechanism for mapping the aesthetic evocations outputted by each Mode of Interpretation into color space. We conclude by discussing some redesigns we have made to the Aesthetiscope and reflecting upon the collected experiences of those who have interacted with the installation. Aesthetiscope as Art Installation The Aesthetiscope is an interactive art installation whose grid of colors visualizes the aesthetic character of some text (a word, a poem, a song); moreover, by adjusting the contribution of each of the five Modes of Interpretation to the gestalt, the visualization can better suit the individualized taste of the perceiver. Figure 1 shows the Aesthetiscope’s visualization of the aesthetic of six different texts, from the point-of-view of a perceiver who has biased the aesthetic interpretation toward the Intuition and Feeling modes. In the remainder of this section, some explanation is offered for the major design decisions of the artwork. Reifying the Aesthetic in Color Space The colors that constitute each color grid represent the combined (and weighted) output of the five Modes of Interpretation applied to the text. Colors are a superb medium of portraiture for the aesthetic character of a text, since color space is a complete micro-consciousness of pathos, just like taste and smell. Mapping the outputs of each Mode of Interpretation into color space is also a most practical way of unifying the outputs of various interpretations into a single undeconstructed gestalt. For example, consider the problem of unifying the visual and affective perceptions of the word “sunset.” In color space, this unification is practical: remembered visual swatches of past seen sunsets can be epitomized as a color palette, and sentimental entailments, such as “warmth, fuzzy, beautiful, serenity and relaxation,” can also be mapped into a color palette, by applying psycho-physiological mapping heuristics learned from the color surveys of Berlin & Kay (1969) and Goethe (1840). When the color palettes resulting from the various Modes of Interpretation are merged to produce the color grid, our goal of conveying the text’s singular, complex aesthetic character to the perceiver is facilitated by the eventuality that the human eye will blend these colors together, and attend to their undeconstructed gestalt rather than to each square individually; hence, the aesthetic character is not a simple sum of individual color squares, but rather, it becomes that Spirit which lives in-between the color squares. As we have learned from our own prior work, the ambivalent and perhaps mystifying nature of colors makes them powerful vehicles for conveying the aesthetic precisely because the aesthetic thrives in spaces of connotation, and is inhibited in spaces of denotation (Liu, 2004). Presentation Particulars The Aesthetiscope is currently installed in a “living room of the future” at the MIT Media Laboratory, and is projected onto one of the room’s walls. The grid of color squares is 16 wide by 9 tall, flanked by black striping on top and bottom. There is a “glimmer” effect added to the colors in the grid, as their Values (i.e. Value, as in the Munsellian Hue-Value-Chroma system for colors) wax and wane according to various periodicities. Finally, the glimmering of the color grid refreshes at 24 frames per second, to complete the cinematic quality of the piece. We intend for the Aesthetiscope not simply to stand alone as a showpiece but also to play a supporting role for other activities in the room. By visualizing the aesthetic character of a poem being read (this activity can be detected by our context-aware room), or of the lyrics to a song being played over the room’s stereo system, we can imagine how the pairing of the Aesthetiscope’s color grid with the poem or song might enhance th