Registration and fusion quantification of augmented reality based nasal endoscopic surgery

HighlightsThe impacts of registration and fusion display errors on the navigation accuracy are quantitively studied on a custom ARNES.Accuracy level of a calibrated endoscope is redefined by using a calibration tool with improved structural reliability.A point cloud‐based hybrid tracking method is proposed to rapidly shorten the duration of intraoperative recalibration.The dynamic endoscopic vision expansion, hierarchical rendering, is validated thoroughly in model studies and clinical trials. ABSTRACT This paper quantifies the registration and fusion display errors of augmented reality‐based nasal endoscopic surgery (ARNES). We comparatively investigated the spatial calibration process for front‐end endoscopy and redefined the accuracy level of a calibrated endoscope by using a calibration tool with improved structural reliability. We also studied how registration accuracy was combined with the number and distribution of the deployed fiducial points (FPs) for positioning and the measured registration time. A physically integrated ARNES prototype was customarily configured for performance evaluation in skull base tumor resection surgery with an innovative approach of dynamic endoscopic vision expansion. As advised by surgical experts in otolaryngology, we proposed a hierarchical rendering scheme to properly adapt the fused images with the required visual sensation. By constraining the rendered sight in a known depth and radius, the visual focus of the surgeon can be induced only on the anticipated critical anatomies and vessel structures to avoid misguidance. Furthermore, error analysis was conducted to examine the feasibility of hybrid optical tracking based on point cloud, which was proposed in our previous work as an in‐surgery registration solution. Measured results indicated that the error of target registration for ARNES can be reduced to 0.77 ± 0.07 mm. For initial registration, our results suggest that a trade‐off for a new minimal time of registration can be reached when the distribution of five FPs is considered. For in‐surgery registration, our findings reveal that the intrinsic registration error is a major cause of performance loss. Rigid model and cadaver experiments confirmed that the scenic integration and display fluency of ARNES are smooth, as demonstrated by three clinical trials that surpassed practicality. Graphical abstract Figure. No caption available.