Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies

We present a real-time stereoscopic video-see-through augmented reality (AR) system applied to the medical procedure known as ultrasound-guided needle biopsy of the breast. The AR system was used by a physician during procedures on breast models and during non-invasive examinations of human subjects. The system merges rendered live ultrasound data and geometric elements with stereo images of the patient acquired through head-mounted video cameras and presents these merged images to the physician in a head-mounted display. The physician sees a volume visualization of the ultrasound data directly under the ultrasound probe, properly registered within the patient and with the biopsy needle. Using this system, a physician successfully guided a needle into an artificial tumor within a training phantom of a human breast. We discuss the construction of the AR system and the issues and decisions which led to the system architecture and the design of the video see-through head-mounted display. We designed methods to properly resolve occlusion of the real and synthetic image elements. We developed techniques for realtime volume visualization of timeand position-varying ultrasound data. We devised a hybrid tracking system which achieves improved registration of synthetic and real imagery and we improved on previous techniques for calibration of a magnetic tracker. CR

[1]  Thomas A. Furness.  The Super Cockpit and its Human Factors Challenges , 1986 .

[2]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[3]  David E. Breen,et al.  Annotating Real-World Objects Using Augmented Reality , 1995, Computer Graphics.

[4]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[5]  Dave Sims,et al.  New realities in aircraft design and manufacture , 1994, IEEE Computer Graphics and Applications.

[6]  J. P. Mellor,et al.  Enhanced Reality Visualization in a Surgical Environment , 1995 .

[7]  Ulrich Neumann,et al.  Dynamic registration correction in video-based augmented reality systems , 1995, IEEE Computer Graphics and Applications.

[8]  Mark A. Livingston,et al.  Superior augmented reality registration by integrating landmark tracking and magnetic tracking , 1996, SIGGRAPH.

[9]  David P. Dobkin,et al.  The quickhull algorithm for convex hulls , 1996, TOMS.

[10]  Frank Biocca,et al.  Quantification of adaptation to virtual-eye location in see-thru head-mounted displays , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[11]  Warren Robinett,et al.  A Computational Model for the Stereoscopic Optics of a Head-Mounted Display , 1991, Presence: Teleoperators & Virtual Environments.

[12]  William E. Lorensen,et al.  Enhancing reality in the operating room , 1993, Proceedings Visualization '93.

[13]  T. Todd Elvins,et al.  Visualization of 3D ultrasound data , 1993, IEEE Computer Graphics and Applications.

[14]  Mary C. Whitton,et al.  Real-time incremental visualization of dynamic ultrasound volumes using parallel BSP trees , 1996, Proceedings of Seventh Annual IEEE Visualization '96.

[15]  Hooshang Kangarloo,et al.  Three-Dimensional Reconstruction Of Ultrasound Images , 1989, Medical Imaging.

[16]  Michael Bajura,et al.  Merging Virtual Objects with the Real World , 1992 .

[17]  Ryutarou Ohbuchi,et al.  Incremental volume reconstruction and rendering for 3-D ultrasound imaging , 1992, Other Conferences.

[18]  Hong Chen,et al.  Observing a volume rendered fetus within a pregnant patient , 1994, Proceedings Visualization '94.

[19]  Hong Chen,et al.  Interactive volume visualization on a heterogeneous message-passing multicomputer , 1995, I3D '95.

[20]  Ulrich Neumann,et al.  Accelerating Volume Reconstruction With 3D Texture Hardware , 1994 .

[21]  John Amanatides,et al.  Merging BSP trees yields polyhedral set operations , 1990, SIGGRAPH.

[22]  Ronald Azuma,et al.  Improving static and dynamic registration in an optical see-through HMD , 1994, SIGGRAPH.

[23]  Steve T. Bryson.  Measurement and calibration of static distortion of position data from 3D trackers , 1992, Electronic Imaging.

[24]  Matthias M. Wloka,et al.  Resolving occlusion in augmented reality , 1995, I3D '95.

[25]  Henry Fuchs,et al.  Near real-time shaded display of rigid objects , 1983, SIGGRAPH.

[26]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[27]  Brian Cabral,et al.  Accelerated volume rendering and tomographic reconstruction using texture mapping hardware , 1994, VVS '94.

[28]  FuchsHenry,et al.  Merging virtual objects with the real world , 1992 .

[29]  Thomas A. DeFanti,et al.  Ultrasonic calibration of a magnetic tracker in a virtual reality space , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[30]  Vernon L. Chi,et al.  Computer glasses: a compact, lightweight, and cost-effective display for monocular and tiled wide field-of-view systems , 1995, Optics & Photonics.

[31]  Henry Fuchs,et al.  On visible surface generation by a priori tree structures , 1980, SIGGRAPH '80.