A system for three-dimensional visualization of human jaw motion in speech.

With the development of precise three-dimensional motion measurement systems and powerful computers for three-dimensional graphical visualization, it is possible to record and fully reconstruct human jaw motion. In this paper, we describe a visualization system for displaying three-dimensional jaw movements in speech. The system is designed to take as input jaw motion data obtained from one or multi-dimensional recording systems. In the present application, kinematic records of jaw motion were recorded using an optoelectronic measurement system (Optotrak). The corresponding speech signal was recorded using an analog input channel. The three orientation angles and three positions that describe the motion of the jaw as a rigid skeletal structure were derived from the empirical measurements. These six kinematic variables, which in mechanical terms account fully for jaw motion kinematics, act as inputs that drive a real-time three-dimensional animation of a skeletal jaw and upper skull. The visualization software enables the user to view jaw motion from any orientation and to change the viewpoint during the course of an utterance. Selected portions of an utterance may be replayed and the speed of the visual display may be varied. The user may also display, along with the audio track, individual kinematic degrees of freedom or several degrees of freedom in combination. The system is presently being used as an educational tool and for research into audio-visual speech recognition. Interested researchers may obtain the software and source code free of charge from the authors.