Enhanced Reality Visualization in a Surgical Environment

Enhanced reality visualization is the process of enhancing an image by adding to it information which is not present in the original image. A wide variety of information can be added to an image ranging from hidden lines or surfaces to textual or iconic data about a particular part of the image. Enhanced reality visualization is particularly well suited to neurosurgery. By rendering brain structures which are not visible, at the correct location in an image of a patient''s head, the surgeon is essentially provided with X-ray vision. He can visualize the spatial relationship between brain structures before he performs a craniotomy and during the surgery he can see what''s under the next layer before he cuts through. Given a video image of the patient and a three dimensional model of the patient''s brain the problem enhanced reality visualization faces is to render the model from the correct viewpoint and overlay it on the original image. The relationship between the coordinate frames of the patient, the patient''s internal anatomy scans and the image plane of the camera observing the patient must be established. This problem is closely related to the camera calibration problem. This report presents a new approach to finding this relationship and develops a system for performing enhanced reality visualization in a surgical environment. Immediately prior to surgery a few circular fiducials are placed near the surgical site. An initial registration of video and internal data is performed using a laser scanner. Following this, our method is fully automatic, runs in nearly real-time, is accurate to within a pixel, allows both patient and camera motion, automatically corrects for changes to the internal camera parameters (focal length, focus, aperture, etc.) and requires only a single image.

[1]  Bernard Gibaud,et al.  An Anatomic-Based 3D Registration System of Multimodality and Atlas Data in Neurosurgery , 1991, IPMI.

[2]  Soren W. Henriksen,et al.  Manual of photogrammetry , 1980 .

[3]  Steven D. Pieper,et al.  Interactive graphics for plastic surgery: a task-level analysis and implementation , 1992, I3D '92.

[4]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[5]  Werner Krybus,et al.  CAS — a Navigation Support for Surgery , 1990 .

[6]  Duane C. Brown,et al.  Close-Range Camera Calibration , 1971 .

[7]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[8]  Bidyut Baran Chaudhuri,et al.  Elliptic fit of objects in two and three dimensions by moment of inertia optimization , 1991, Pattern Recognit. Lett..

[9]  Glenn Healey,et al.  Radiometric CCD camera calibration and noise estimation , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Israel Amir,et al.  Design of Fiducials for Accurate Registration Using Machine Vision , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  David S. Watkins,et al.  Fundamentals of matrix computations , 1991 .

[12]  Lydia E. Kavraki,et al.  Treatment planning for a radiosurgical system with general kinematics , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[13]  Henry Fuchs,et al.  A real-time optical 3D tracker for head-mounted display systems , 1990, I3D '90.

[14]  Dean Brown,et al.  Decentering distortion of lenses , 1966 .

[15]  Berthold K. P. Horn.  Robot vision , 1986, MIT electrical engineering and computer science series.

[16]  Behrooz Kamgar-Parsi,et al.  Evaluation of quantization error in computer vision , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Yiu-Tong Chan,et al.  A simple approach for the estimation of circular arc center and its radius , 1989, Comput. Vis. Graph. Image Process..

[18]  Guy Marchal,et al.  Computer assisted stereotactic neurosurgery , 1993, Image Vis. Comput..

[19]  R. Kikinis,et al.  Video registration using fiducials for surgical enhanced reality , 1993, Proceedings of the 15th Annual International Conference of the IEEE Engineering in Medicine and Biology Societ.

[20]  Louis A. Tamburino,et al.  A Unified Approach to the Linear Camera Calibration Problem , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  B. Webber,et al.  Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care , 1994 .

[22]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[23]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[24]  Steven A. Shafer,et al.  What is the center of the image? , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[25]  G. Chiorboli,et al.  Comments on "Design of Fiducials for Accurate Registration Using Machine Vision" , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Beno Benhabib,et al.  Application of moment and Fourier descriptors to the accurate estimation of elliptical-shape parameters , 1992, Pattern Recognit. Lett..

[27]  D. Huttenlocher.  Three-Dimensional Recognition of Solid Objects from a Two- Dimensional Image , 1988 .

[28]  L. Light.  Engineering in medicine and biology , 1970 .

[29]  David N. Levin,et al.  Interactive 3-D Patient-Image Registration , 1991, IPMI.

[30]  Behrooz Kamgar-Parsi,et al.  Evaluation of Quantization Error in Computer Vision , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Its'hak Dinstein,et al.  Imaging system response linearization and shading correction , 1984, ICRA.

[32]  P. Cinquin,et al.  Computer assisted medical interventions , 1995 .

[33]  Ronald Azuma,et al.  Improving static and dynamic registration in an optical see-through HMD , 1994, SIGGRAPH.

[34]  A. Ardeshir Goshtasby,et al.  Correction of image deformation from lens distortion using bezier patches , 1989, Comput. Vis. Graph. Image Process..

[35]  Michael A. Penna.  Camera Calibration: A Quick and Easy Way to Determine the Scale Factor , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Mansur R. Kabuka,et al.  Real-time system for accurate three-dimensional position determination and verification , 1990, IEEE Trans. Robotics Autom..

[37]  Roger Y. Tsai,et al.  Techniques for Calibration of the Scale Factor and Image Center for High Accuracy 3-D Machine Vision Metrology , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  J. Davenport.  Editor , 1960 .

[39]  Sundaram Ganapathy,et al.  Decomposition of transformation matrices for robot vision , 1984, Pattern Recognition Letters.

[40]  Gideon P. Stein.  Internal Camera Calibration using Rotation and Geometric Shapes , 1993 .

[41]  W. Eric L. Grimson,et al.  An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Irwin Sobel,et al.  On Calibrating Computer Controlled Cameras for Perceiving 3-D Scenes , 1973, IJCAI.

[43]  Ronald Azuma,et al.  A demonstrated optical tracker with scalable work area for head-mounted display systems , 1992, I3D '92.

[44]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[45]  Alon Efrat,et al.  Subpixel image registration using circular fiducials , 1994, Int. J. Comput. Geom. Appl..

[46]  Verzekeren Naar Sparen,et al.  Cambridge , 1969, Humphrey Burton: In My Own Time.

[47]  W. Faig.  CALIBRATION OF CLOSE-RANGE PHOTOGRAMMETRIC SYSTEMS: MATHEMATICAL FORMULATION , 1975 .

[48]  John F. Hughes,et al.  Autocalibration for virtual environments tracking hardware , 1993, SIGGRAPH.

[49]  U. M. Landau,et al.  Estimation of a circular arc center and its radius , 1987, Comput. Vis. Graph. Image Process..