Realistic avatar eye and head animation using a neurobiological model of visual attention

We describe a neurobiological model of visual attention and eye/head movements in primates, and its application to the automatic animation of a realistic virtual human head watching an unconstrained variety of visual inputs. The bottom-up (image-based) attention model is based on the known neurophysiology of visual processing along the occipito-parietal pathway of the primate brain, while the eye/head movement model is derived from recordings in freely behaving Rhesus monkeys. The system is successful at autonomously saccading towards and tracking salient targets in a variety of video clips, including synthetic stimuli, real outdoors scenes and gaming console outputs. The resulting virtual human eye/head animation yields realistic rendering of the simulation results, both suggesting applicability of this approach to avatar animation and reinforcing the plausibility of the neural model.

[1]  A. Leventhal.  The neural basis of visual function , 1991 .

[2]  J. Haxby,et al.  Human neural systems for face recognition and social communication , 2002, Biological Psychiatry.

[3]  Matthew Brand,et al.  Voice puppetry , 1999, SIGGRAPH.

[4]  Werner Reichardt,et al.  Evaluation of optical motion information by movement detectors , 1987, Journal of Comparative Physiology A.

[5]  J. B. Levitt,et al.  Contrast dependence of contextual effects in primate visual cortex , 1997, nature.

[6]  E Bizzi,et al.  The coordination of eye-head movements. , 1974, Scientific American.

[7]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[8]  Demetri Terzopoulos,et al.  Animat vision: Active vision in artificial animals , 1995, Proceedings of IEEE International Conference on Computer Vision.

[9]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[10]  Laurent Itti,et al.  A Goal Oriented Attention Guidance Model , 2002, Biologically Motivated Computer Vision.

[11]  R. Weller.  Two cortical visual systems in Old World and New World primates. , 1988, Progress in brain research.

[12]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[13]  R. Walker,et al.  A model of saccade generation based on parallel processing and competitive inhibition , 1999, Behavioral and Brain Sciences.

[14]  B. C. Motter,et al.  The guidance of eye movements during active visual search , 1998, Vision Research.

[15]  H. Jones,et al.  Visual cortical mechanisms detecting focal orientation discontinuities , 1995, Nature.

[16]  Norman I. Badler,et al.  Eyes alive , 2002, ACM Trans. Graph..

[17]  A F Monk,et al.  Where am I looking? The accuracy of video-mediated gaze awareness , 2000, Perception & psychophysics.

[18]  H. Nothdurft,et al.  Pop-out of orientation but no pop-out of motion at isoluminance , 1993, Vision Research.

[19]  G. Barnes.  Vestibulo‐ocular function during co‐ordinated head and eye movements to acquire visual targets. , 1979, The Journal of physiology.

[20]  R. Desimone,et al.  Selective attention gates visual processing in the extrastriate cortex. , 1985, Science.

[21]  M. Posner,et al.  Neural systems control of spatial orienting. , 1982, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[22]  Emilio Bizzi,et al.  The coordination of eye and head movement during smooth pursuit , 1978, Brain Research.

[23]  M. Corbetta,et al.  Control of goal-directed and stimulus-driven attention in the brain , 2002, Nature Reviews Neuroscience.

[24]  Carol O'Sullivan,et al.  Synthetic Vision and Memory for Autonomous Virtual Humans , 2002, Comput. Graph. Forum.

[25]  Peter Ford Dominey,et al.  A cortico-subcortical model for generation of spatially accurate sequential saccades. , 1992, Cerebral cortex.

[26]  Tomaso Poggio,et al.  Trainable Videorealistic Speech Animation , 2004, FGR.

[27]  C. Koch,et al.  Constraints on cortical and thalamic projections: the no-strong-loops hypothesis , 1998, Nature.

[28]  J. Wolfe.  Visual search in continuous, naturalistic stimuli , 1994, Vision Research.

[29]  U. Polat,et al.  The architecture of perceptual spatial interactions , 1994, Vision Research.

[30]  G R Barnes,et al.  Visual-vestibular interaction during head-free pursuit of pseudorandom target motion in man. , 1992, Journal of vestibular research : equilibrium & orientation.

[31]  D. G. Albrecht,et al.  Spatial frequency selectivity of cells in macaque visual cortex , 1982, Vision Research.

[32]  R. Hill.  Perceptual Attention in Virtual Humans: Towards Realistic and Believable Gaze Behaviors , 2002 .

[33]  Mark W. Cannon,et al.  Spatial interactions in apparent contrast: Inhibitory effects among grating patterns of different spatial frequencies, spatial positions and orientations , 1991, Vision Research.

[34]  Antonio Torralba,et al.  Contextual Modulation of Target Saliency , 2001, NIPS.

[35]  D. Fitzpatrick,et al.  Patterns of excitation and inhibition evoked by horizontal connections in visual cortex share a common relationship to orientation columns , 1995, Neuron.

[36]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[37]  Michael J. Jones,et al.  Gaze estimation using morphable models , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[38]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[39]  Daniel Thalmann,et al.  A vision-based approach to behavioural animation , 1990, Comput. Animat. Virtual Worlds.

[40]  Alexander Zelinsky,et al.  3-D facial pose and gaze point estimation using a robust real-time tracking paradigm , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[41]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[42]  D L Sparks,et al.  Activity of cells in the deeper layers of the superior colliculus of the rhesus monkey: evidence for a gaze displacement command. , 1997, Journal of neurophysiology.

[43]  Alexander H. Waibel,et al.  Estimating focus of attention based on gaze and sound , 2001, PUI '01.

[44]  Edward G. Freedman,et al.  Interactions between eye and head control signals can account for movement kinematics , 2001, Biological Cybernetics.

[45]  Thierry Pun,et al.  Attentive mechanisms for dynamic and static scene analysis , 1995 .

[46]  Gunilla Borgefors,et al.  Distance transformations in digital images , 1986, Comput. Vis. Graph. Image Process..

[47]  B. Wandell.  Foundations of vision , 1995 .

[48]  A. Opstal,et al.  Human eye-head coordination in two dimensions under different sensorimotor conditions , 1997, Experimental Brain Research.

[49]  J. Lund,et al.  Intrinsic laminar lattice connections in primate visual cortex , 1983, The Journal of comparative neurology.

[50]  K. Hoffmann,et al.  Neural Mechanisms of Saccadic Suppression , 2002, Science.

[51]  John K. Tsotsos,et al.  Modeling Visual Attention via Selective Tuning , 1995, Artif. Intell..

[52]  E. Chen,et al.  Blink rate, neurocognitive impairments, and symptoms in schizophrenia , 1996, Biological Psychiatry.

[53]  Pietro Perona,et al.  Overcomplete steerable pyramid filters and rotation invariance , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[54]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[55]  Laurent Itti,et al.  Real-time high-performance attention focusing in outdoors color video streams , 2002, IS&T/SPIE Electronic Imaging.

[56]  D. Sparks.  The brainstem control of saccadic eye movements , 2002, Nature Reviews Neuroscience.

[57]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[58]  RussLL L. Ds Vnlos,et al.  SPATIAL FREQUENCY SELECTIVITY OF CELLS IN MACAQUE VISUAL CORTEX , 2022 .

[59]  E Bizzi,et al.  Two modes of active eye-head coordination in monkeys. , 1972, Brain research.

[60]  G. Sperling,et al.  Dynamics of automatic and controlled visual attention. , 1987, Science.

[61]  K. Fukuda,et al.  Eye blinks: new indices for the detection of deception. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[62]  T. Wiesel,et al.  Columnar specificity of intrinsic horizontal and corticocortical connections in cat visual cortex , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[63]  Stefan Treue,et al.  Feature-based attention influences motion processing gain in macaque visual cortex , 1999, Nature.

[64]  L. Spillmann,et al.  Visual Perception: The Neurophysiological Foundations , 1989 .