Automating the conversion of natural language fiction to multi-modal 3D animated virtual environments

Popular ction books describe rich visual environments that contain characters, objects, and behaviour. This research develops automated processes for converting text sourced from ction books into animated virtual environments and multi-modal lms. This involves the analysis of unrestricted natural language ction to identify appropriate visual descriptions, and the interpretation of the identi ed descriptions for constructing animated 3D virtual environments. The goal of the text analysis stage is the creation of annotated ction text, which identi es visual descriptions in a structured manner. A hierarchical rule-based learning system is created that induces patterns from example annotations provided by a human, and uses these for the creation of additional annotations. Patterns are expressed as tree structures that abstract the input text on di erent levels according to structural (token, sentence) and syntactic (parts-of-speech, syntactic function) categories. Patterns are generalized using pair-wise merging, where dissimilar sub-trees are replaced with wild-cards. The result is a small set of generalized patterns that are able to create correct annotations. A set of generalized patterns represents a model of an annotator's mental process regarding a particular annotation category. Annotated text is interpreted automatically for constructing detailed scene descriptions. This includes identifying which scenes to visualize, and identifying the contents and behaviour in each scene. Entity behaviour in a 3D virtual environment is formulated using time-based constraints that are automatically derived from annotations. Constraints are expressed as non-linear symbolic functions that restrict the trajectories of a pair of entities over a continuous interval of time. Solutions to these constraints specify precise behaviour. We create an innovative quanti ed constraint optimizer for locating sound solutions, which uses interval arithmetic for treating time and space as contiguous quantities. This optimization method uses a technique of constraint relaxation and tightening that allows solution approximations to be located where constraint systems are inconsistent (an ability not previously explored in interval-based quanti ed constraint solving). 3D virtual environments are populated by automatically selecting geometric models or procedural geometry-creation methods from a library. 3D models are animated according to trajectories derived from constraint solutions. The nal animated lm is sequenced using a range of modalities including animated 3D graphics, textual subtitles, audio narrations, and foleys. Hierarchical rule-based learning is evaluated over a range of annotation categories. Models are induced for di erent categories of annotation without modifying the core learning algorithms, and these models are shown to be applicable to di erent types of books. Models are induced automatically with accuracies ranging between 51.4% and 90.4%, depending on the category. We show that models are re ned if further examples are provided, and this supports a boot-strapping process for training the learning mechanism. The task of interpreting annotated ction text and populating 3D virtual environments is successfully automated using our described techniques. Detailed scene descriptions are created accurately, where between 83% and 96% of the automatically generated descriptions require no manual modi cation (depending on the type of description). The interval-based quanti ed constraint optimizer fully automates the behaviour speci cation process. Sample animated multi-modal 3D lms are created using extracts from ction books that are unrestricted in terms of complexity or subject matter (unlike existing text-to-graphics systems). These examples demonstrate that: behaviour