Artificial intelligence assisted display in thoracic surgery: development and possibilities

In this golden age of rapid development of artificial intelligence (AI), researchers and surgeons realized that AI could contribute to healthcare in all aspects, especially in surgery. The popularity of low-dose computed tomography (LDCT) and the improvement of the video-assisted thoracoscopic surgery (VATS) not only bring opportunities for thoracic surgery but also bring challenges on the way forward. Preoperatively localizing lung nodules precisely, intraoperatively identifying anatomical structures accurately, and avoiding complications requires a visual display of individuals’ specific anatomy for surgical simulation and assistance. With the advance of AI-assisted display technologies, including 3D reconstruction/3D printing, virtual reality (VR), augmented reality (AR), and mixed reality (MR), computer tomography (CT) imaging in thoracic surgery has been fully utilized for transforming 2D images to 3D model, which facilitates surgical teaching, planning, and simulation. AI-assisted display based on surgical videos is a new surgical application, which is still in its infancy. Notably, it has potential applications in thoracic surgery education, surgical quality evaluation, intraoperative assistance, and postoperative analysis. In this review, we illustrated the current AI-assisted display applications based on CT in thoracic surgery; focused on the emerging AI applications in thoracic surgery based on surgical videos by reviewing its relevant researches in other surgical fields and anticipate its potential development in thoracic surgery.

[1]  C. Pugh.  Response to the Comment on "Situating Artificial Intelligence in Surgery: A Focus on Disease Severity". , 2021, Annals of Surgery.

[2]  Yuchen Guo,et al.  Comment on "Situating Artificial Intelligence in Surgery A Focus on Disease Severity". , 2021, Annals of Surgery.

[3]  Thomas M. Ward,et al.  Surgical data science and artificial intelligence for surgical education , 2021, Journal of surgical oncology.

[4]  Yudong Zhang,et al.  Subdivision and presentation of the pulmonary vasculature of the right upper lobe for anatomical segmentectomy with three-dimensional computed tomography reconstruction. , 2021, Asian Journal of Surgery.

[5]  J. Elefteriades,et al.  Machine learning: principles and applications for thoracic surgery. , 2021, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[6]  H. Yoon,et al.  Ergonomic effects of medical augmented reality glasses in video-assisted surgery , 2021, Surgical Endoscopy.

[7]  S. Moccia,et al.  A Machine Learning Approach for Postoperative Outcome Prediction: Surgical Data Science Application in a Thoracic Surgery Setting , 2021, World Journal of Surgery.

[8]  T. Clapp,et al.  Developing a virtual reality simulation system for preoperative planning of thoracoscopic thoracic surgery , 2021, Journal of thoracic disease.

[9]  E. Klang,et al.  Deep learning visual analysis in laparoscopic surgery: a systematic review and diagnostic test accuracy meta-analysis , 2021, Surgical Endoscopy.

[10]  B. Zheng,et al.  Augmented Reality and Three-Dimensional Printing Technologies for Guiding Complex Thoracoscopic Surgery. , 2020, The Annals of thoracic surgery.

[11]  Yi-long Wu,et al.  A three-dimensional printing navigational template combined with mixed reality technique for localizing pulmonary nodules. , 2020, Interactive cardiovascular and thoracic surgery.

[12]  Klaus H. Maier-Hein,et al.  Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge , 2020, Medical Image Anal..

[13]  Miao Zhang,et al.  Analysis of the variation pattern in left upper division veins and establishment of simplified vein models for anatomical segmentectomy , 2020, Annals of Translational Medicine.

[14]  M. Ferguson,et al.  Consensus for Thoracoscopic Left Upper Lobectomy-Essential Components and Targets for Simulation. , 2020, The Annals of thoracic surgery.

[15]  Qiang Li,et al.  Comparison of performances of conventional and deep learning-based methods in segmentation of lung vessels and registration of chest radiographs , 2020, Radiological Physics and Technology.

[16]  Lei Gao,et al.  Total superior vena cava reconstruction guided by preoperative three-dimensional (3D)-computed tomography bronchography and angiography , 2020, Translational cancer research.

[17]  J. Assouad,et al.  Artificial intelligence in thoracic surgery: past, present, perspective and limits , 2020, European Respiratory Review.

[18]  Yao Guo,et al.  Application of artificial intelligence in surgery , 2020, Frontiers of Medicine.

[19]  Yi-long Wu,et al.  Three‐dimensional printed navigational template for localizing small pulmonary nodules: A case‐controlled study , 2020, Thoracic cancer.

[20]  B. Qiu,et al.  Three-dimensional reconstruction/personalized three-dimensional printed model for thoracoscopic anatomical partial-lobectomy in stage I lung cancer: a retrospective study , 2020, Translational lung cancer research.

[21]  K. Mori,et al.  Automated Laparoscopic Colorectal Surgery Workflow Recognition using Artificial Intelligence: Experimental Research. , 2020, International journal of surgery.

[22]  Y. Iwashita,et al.  Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomy , 2020, Surgical Endoscopy.

[23]  A. Watanabe,et al.  Pulmonary vessels and bronchial anatomy of the left lower lobe , 2020, Surgery Today.

[24]  Stephanie L. Perkins,et al.  A Patient-Specific Mixed-Reality Visualization Tool for Thoracic Surgical Planning. , 2020, The Annals of thoracic surgery.

[25]  Nicolas Martin,et al.  Assisted phase and step annotation for surgical videos , 2020, International Journal of Computer Assisted Radiology and Surgery.

[26]  A. Bartoli,et al.  SurgAI: deep learning for computerized laparoscopic image understanding in gynaecology , 2020, Surgical Endoscopy.

[27]  Clyde Matava,et al.  A Convolutional Neural Network for Real Time Classification, Identification, and Labelling of Vocal Cord and Tracheal Using Laryngoscopy and Bronchoscopy Video , 2020, Journal of Medical Systems.

[28]  W. Jiao,et al.  Three-dimensional printing in the preoperative planning of thoracoscopic pulmonary segmentectomy. , 2019, Translational lung cancer research.

[29]  Naotake Kamiura,et al.  Real-Time Extraction of Important Surgical Phases in Cataract Surgery Videos , 2019, Scientific Reports.

[30]  Didier Mutter,et al.  Formalizing video documentation of the Critical View of Safety in laparoscopic cholecystectomy: a step towards artificial intelligence assistance to improve surgical safety , 2019, Surgical Endoscopy.

[31]  G. Rosman,et al.  Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy. , 2019, Annals of surgery.

[32]  Jiapeng Li,et al.  Comparing the diagnostic value of 18F-FDG-PET/CT versus CT for differentiating benign and malignant solitary pulmonary nodules: a meta-analysis. , 2019, Journal of thoracic disease.

[33]  Gregory D. Hager,et al.  Assessment of Automated Identification of Phases in Videos of Cataract Surgery Using Machine Learning and Deep Learning Techniques , 2019, JAMA network open.

[34]  Ping Yang,et al.  Accuracy of a 3-Dimensionally Printed Navigational Template for Localizing Small Pulmonary Nodules: A Noninferiority Randomized Clinical Trial , 2019, JAMA surgery.

[35]  Stefanie Speidel,et al.  Video-based surgical skill assessment using 3D convolutional neural networks , 2019, International Journal of Computer Assisted Radiology and Surgery.

[36]  Martin Wagner,et al.  Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data , 2018, International Journal of Computer Assisted Radiology and Surgery.

[37]  L. Konge,et al.  Evaluating competency in video-assisted thoracoscopic surgery (VATS) lobectomy performance using a novel assessment tool and virtual reality simulation , 2018, Surgical Endoscopy.

[38]  A. Jemal,et al.  Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries , 2018, CA: a cancer journal for clinicians.

[39]  G. Rosman,et al.  Artificial Intelligence in Surgery: Promises and Perils , 2018, Annals of surgery.

[40]  Jun Li,et al.  Development and clinical applications of glasses-free three-dimensional (3D) display technology for thoracoscopic surgery. , 2018, Annals of translational medicine.

[41]  Michael D Klein,et al.  Automated robot‐assisted surgical skill evaluation: Predictive analytics approach , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[42]  H. Kuwano,et al.  VATS segmentectomy: past, present, and future , 2018, General Thoracic and Cardiovascular Surgery.

[43]  I. Sauer,et al.  Mixed Reality in Visceral Surgery: Development of a Suitable Workflow and Evaluation of Intraoperative Use-cases , 2017, Annals of surgery.

[44]  Chengming Ding,et al.  [Combining 3D-CTBA and 3D-VATS Single-Operation-Hole to 
Anatomical Segmentectomy in the Treatment of Non-small Cell Lung Cancer]. , 2017, Zhongguo fei ai za zhi = Chinese journal of lung cancer.

[45]  Guy Rosman,et al.  Surgical Video in the Age of Big Data. , 2017, Annals of surgery.

[46]  Lars Konge,et al.  Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy , 2017, Surgical Endoscopy.

[47]  C. Bennett,et al.  Identification of Essential Components of Thoracoscopic Lobectomy and Targets for Simulation. , 2017, The Annals of thoracic surgery.

[48]  H. Kuwano,et al.  Analysis of variation in bronchovascular pattern of the right middle and lower lobes of the lung using three-dimensional CT angiography and bronchography , 2017, General Thoracic and Cardiovascular Surgery.

[49]  Ketan Patel,et al.  Augmented and virtual reality in surgery-the digital surgical environment: applications, limitations and legal pitfalls. , 2016, Annals of translational medicine.

[50]  W. Liang,et al.  Choice of Surgical Procedure for Patients With Non-Small-Cell Lung Cancer ≤ 1 cm or > 1 to 2 cm Among Lobectomy, Segmentectomy, and Wedge Resection: A Population-Based Study. , 2016, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[51]  Andru Putra Twinanda,et al.  EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos , 2016, IEEE Transactions on Medical Imaging.

[52]  D. Gossot,et al.  Major intraoperative complications during video-assisted thoracoscopic anatomical lung resections: an intention-to-treat analysis. , 2015, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[53]  G. Jiang,et al.  Intraoperative bleeding control by uniportal video-assisted thoracoscopic surgery†. , 2015, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[54]  I. Jung,et al.  Analysis of Unexpected Conversion to Thoracotomy During Thoracoscopic Lobectomy in Lung Cancer. , 2015, The Annals of thoracic surgery.

[55]  Ciprian Ionita,et al.  Three-dimensional printing to facilitate anatomic study, device development, simulation, and planning in thoracic surgery. , 2015, The Journal of thoracic and cardiovascular surgery.

[56]  I. Takeyoshi,et al.  An analysis of variations in the bronchovascular pattern of the right upper lobe using three-dimensional CT angiography and bronchography , 2015, General Thoracic and Cardiovascular Surgery.

[57]  T. Ohira,et al.  High-quality 3-dimensional image simulation for pulmonary lobectomy and segmentectomy: results of preoperative assessment of pulmonary vessels and short-term surgical outcomes in consecutive patients undergoing video-assisted thoracic surgery†. , 2014, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[58]  S. Iwano,et al.  Planning of segmentectomy using three-dimensional computed tomography angiography with a virtual safety margin: technique and initial experience. , 2013, Lung cancer.

[59]  Emiliano Schena,et al.  Percutaneous lung biopsy: comparison between an augmented reality CT navigation system and standard CT-guided technique , 2013, International Journal of Computer Assisted Radiology and Surgery.

[60]  M. de Lena,et al.  Percutaneous Computed Tomography-Guided Lung Biopsies: Preliminary Results using an Augmented Reality Navigation System , 2012, Tumori.

[61]  T. Morikawa,et al.  Thoracoscopic lobectomy for treating cancer in a patient with an unusual vein anomaly. , 2011, Annals of thoracic and cardiovascular surgery : official journal of the Association of Thoracic and Cardiovascular Surgeons of Asia.

[62]  C. Gatsonis,et al.  Reduced Lung-Cancer Mortality with Low-Dose Computed Tomographic Screening , 2012 .

[63]  Myrna C B Godoy,et al.  Subsolid pulmonary nodules and the spectrum of peripheral adenocarcinomas of the lung: recommended interim guidelines for assessment and management. , 2009, Radiology.

[64]  M. Yamashita,et al.  Evaluation of video-assisted thoracoscopic surgery lobectomy requiring emergency conversion to thoracotomy. , 2009, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[65]  Toru Nakamura,et al.  The common trunk of the left pulmonary vein injured incidentally during lung cancer surgery. , 2009, The Annals of thoracic surgery.

[66]  K. Fukuhara,et al.  Preoperative assessment of the pulmonary artery by three-dimensional computed tomography before video-assisted thoracic surgery lobectomy. , 2008, European Journal of Cardio-Thoracic Surgery.

[67]  J. Loscertales,et al.  Video-assisted thoracic surgery (VATS) lobectomy: 13 years’ experience , 2008, Surgical Endoscopy.

[68]  Chee Kai Chua,et al.  Indirect fabrication of collagen scaffold based on inkjet printing technique , 2006 .

[69]  P. Kvale,et al.  Update in screening of lung cancer , 2005, Respirology.

[70]  Alessandro Marro,et al.  Three-Dimensional Printing and Medical Imaging: A Review of the Methods and Applications. , 2016, Current problems in diagnostic radiology.

[71]  S. Broderick,et al.  Intraoperative conversion from video-assisted thoracoscopic surgery lobectomy to open thoracotomy: a study of causes and implications. , 2015, The Journal of thoracic and cardiovascular surgery.