ProX: A Reversed Once-for-All Network Training Paradigm for Efficient Edge Models Training in Medical Imaging

The usage of edge models in medical field has a huge impact on promoting the accessibility of real-time medical services in the under-developed regions. However, the handling of latency-accuracy trade-off to produce such an edge model is very challenging. Although the recent Once-For-All (OFA) network is able to directly produce a set of sub-network designs with Progressive Shrinking (PS) algorithm, it still suffers from training resource and time inefficiency downfall. In this paper, we propose a new OFA training algorithm, namely the Progressive Expansion (ProX). Empirically, we showed that the proposed paradigm can reduce training time up to 68%; while still able to produce sub-networks that have either similar or better accuracy compared to those trained with OFA-PS in ROCT (classification), BRATS and Hippocampus (3D-segmentation) public medical datasets.

[1]  Joon Huang Chuah,et al.  End-to-End Supermask Pruning: Learning to Prune Image Captioning Models , 2021, Pattern Recognit..

[2]  Alexey Tumanov,et al.  CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment , 2021, ArXiv.

[3]  Song Han,et al.  Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution , 2020, ECCV.

[4]  Jose Javier Gonzalez Ortiz,et al.  What is the State of Neural Network Pruning? , 2020, MLSys.

[5]  Chuang Gan,et al.  Once for All: Train One Network and Specialize it for Efficient Deployment , 2019, ICLR.

[6]  Yi Pan,et al.  Fast Deep Learning Training through Intelligently Freezing Layers , 2019, 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData).

[7]  Houqiang Li,et al.  Quantization Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Paul J. Kennedy,et al.  Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges , 2019, Journal of Digital Imaging.

[9]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[10]  Chee Seng Chan,et al.  COMIC: Toward A Compact Image Captioning Model With Attention , 2019, IEEE Transactions on Multimedia.

[11]  Ronald M. Summers,et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms , 2019, ArXiv.

[12]  Ning Xu,et al.  Slimmable Neural Networks , 2018, ICLR.

[13]  Michael Carbin,et al.  The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks , 2018, ICLR.

[14]  Daniel S. Kermany,et al.  Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning , 2018, Cell.

[15]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[17]  Kiyoshi Tanaka,et al.  Fuzzy qualitative deep compression network , 2017, Neurocomputing.

[18]  Sung Ju Hwang,et al.  Lifelong Learning with Dynamically Expandable Networks , 2017, ICLR.

[19]  Xiaofeng Wang,et al.  Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Jeffrey Humpherys,et al.  Forward Thinking: Building and Training Neural Networks One Layer at a Time , 2017, ArXiv.

[21]  Song Han,et al.  Trained Ternary Quantization , 2016, ICLR.

[22]  Lovedeep Gondara,et al.  Medical Image Denoising Using Convolutional Denoising Autoencoders , 2016, 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW).

[23]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[24]  Razvan Pascanu,et al.  Progressive Neural Networks , 2016, ArXiv.

[25]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.