How much deep learning is enough for automatic identification to be reliable?

OBJECTIVES To determine the optimal quantity of learning data needed to develop artificial intelligence (AI) that can automatically identify cephalometric landmarks. MATERIALS AND METHODS A total of 2400 cephalograms were collected, and 80 landmarks were manually identified by a human examiner. Of these, 2200 images were chosen as the learning data to train AI. The remaining 200 images were used as the test data. A total of 24 combinations of the quantity of learning data (50, 100, 200, 400, 800, 1600, and 2000) were selected by the random sampling method without replacement, and the number of detecting targets per image (19, 40, and 80) were used in the AI training procedures. The training procedures were repeated four times. A total of 96 different AIs were produced. The accuracy of each AI was evaluated in terms of radial error. RESULTS The accuracy of AI increased linearly with the increasing number of learning data sets on a logarithmic scale. It decreased with increasing numbers of detection targets. To estimate the optimal quantity of learning data, a prediction model was built. At least 2300 sets of learning data appeared to be necessary to develop AI as accurate as human examiners. CONCLUSIONS A considerably large quantity of learning data was necessary to develop accurate AI. The present study might provide a basis to determine how much learning data would be necessary in developing AI.

[1]  Raphaël Marée,et al.  Automatic Cephalometric X-Ray Landmark Detection Challenge 2014: A tree-based algorithm , 2014 .

[2]  Ho-Jin Lee,et al.  Predicting soft tissue changes after orthognathic surgery: The sparse partial least squares method. , 2019, The Angle orthodontist.

[3]  Ji-Hoon Park,et al.  Automated identification of cephalometric landmarks: Part 1-Comparisons between the latest deep-learning methods YOLOV3 and SSD. , 2019, The Angle orthodontist.

[4]  Arvid Lundervold,et al.  An overview of deep learning in medical imaging focusing on MRI , 2018, Zeitschrift fur medizinische Physik.

[5]  Shin-Jae Lee,et al.  Evaluation of an automated superimposition method for computer-aided cephalometrics. , 2020, The Angle orthodontist.

[6]  Richard E Donatelli,et al.  A more accurate soft-tissue prediction model for Class III 2-jaw surgeries. , 2014, American journal of orthodontics and dentofacial orthopedics : official publication of the American Association of Orthodontists, its constituent societies, and the American Board of Orthodontics.

[7]  Amandeep Kaur,et al.  Automatic cephalometric landmark detection using Zernike moments and template matching , 2015, Signal Image Video Process..

[8]  Huiqi Li,et al.  Automatic Analysis of Lateral Cephalograms Based on Multiresolution Decision Tree Regression Voting , 2018, Journal of healthcare engineering.

[9]  Richard E Donatelli,et al.  A sparse principal component analysis of Class III malocclusions. , 2019, The Angle orthodontist.

[10]  Chengwen Chu,et al.  Evaluation and Comparison of Anatomical Landmark Detection Methods for Cephalometric X-Ray Images: A Grand Challenge , 2015, IEEE Transactions on Medical Imaging.

[11]  Ho-Jin Lee,et al.  Testing a better method of predicting postsurgery soft tissue response in Class II patients: A prospective study and validity assessment. , 2015, The Angle orthodontist.

[12]  Richard E Donatelli,et al.  A more accurate method of predicting soft tissue changes after mandibular setback surgery. , 2012, Journal of oral and maxillofacial surgery : official journal of the American Association of Oral and Maxillofacial Surgeons.

[13]  Chonho Lee,et al.  Deep Learning based Cephalometric Landmark Identification using Landmark-dependent Multi-scale Patches , 2019, ArXiv.

[14]  Heung-Il Suk,et al.  Deep Learning in Medical Image Analysis. , 2017, Annual review of biomedical engineering.

[15]  Hye-Won Hwang,et al.  Automated identification of cephalometric landmarks: Part 2-Might it be better than human? , 2019, The Angle orthodontist.

[16]  Shin-Jae Lee,et al.  Time series analysis of patients seeking orthodontic treatment at Seoul National University Dental Hospital over the past decade , 2017, Korean journal of orthodontics.

[17]  Richard E Donatelli,et al.  A better statistical method of predicting postsurgery soft tissue response in Class II patients. , 2014, The Angle orthodontist.

[18]  M Oshagh,et al.  Accuracy of computerized automatic identification of cephalometric landmarks by a designed software. , 2013, Dento maxillo facial radiology.

[19]  Bulat Ibragimov.  Automatic Cephalometric X-Ray Landmark Detection by Applying Game Theory and Random Forests , 2014 .

[20]  Timothy F. Cootes,et al.  A benchmark for comparison of dental radiography analysis algorithms , 2016, Medical Image Anal..

[21]  Predrag Vucinić,et al.  Automatic landmarking of cephalograms using active appearance models. , 2010, European journal of orthodontics.

[22]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[23]  Shin-Jae Lee,et al.  Modern trends in Class III orthognathic treatment: A time series analysis. , 2016, The Angle orthodontist.

[24]  Bulat Ibragimov,et al.  Fully automated quantitative cephalometry using convolutional neural networks , 2017, Journal of medical imaging.

[25]  Ching-Wei Wang,et al.  Fully Automatic System for Accurate Localisation and Analysis of Cephalometric Landmarks in Lateral Cephalograms , 2016, Scientific Reports.