AirNet: Neural Network Transmission over the Air

State-of-the-art performance for many emerging edge applications is achieved by deep neural networks (DNNs). Often, the employed DNNs are location- and time-dependent, and the parameters of a specific DNN must be delivered from an edge server to the edge device rapidly and efficiently to carry out time-sensitive inference tasks. This can be considered as a joint source-channel coding (JSCC) problem, in which the goal is not to recover the DNN coefficients with the minimal distortion, but in a manner that provides the highest accuracy in the downstream task. For this purpose we introduce AirNet, a novel training and analog transmission method to deliver DNNs over the air. We first train the DNN with noise injection to counter the wireless channel noise. We also employ pruning to identify the most significant DNN parameters that can be delivered within the available channel bandwidth, knowledge distillation, and nonlinear bandwidth expansion to provide better error protection for the most important network parameters. We show that AirNet achieves significantly higher test accuracy compared to the separation-based alternative, and exhibits graceful degradation with channel quality.

[1]  K. Mikolajczyk,et al.  Channel-Adaptive Wireless Image Transmission With OFDM , 2022, IEEE Wireless Communications Letters.

[2]  Mahdi Boloursaz Mashhadi,et al.  LIDAR and Position-Aided mmWave Beam Selection With Non-Local CNNs and Curriculum Training , 2021, IEEE Transactions on Vehicular Technology.

[3]  Zhongwei Si,et al.  Nonlinear Transform Source-Channel Coding for Semantic Communications , 2021, IEEE Journal on Selected Areas in Communications.

[4]  Deniz Gündüz,et al.  DeepWiVe: Deep-Learning-Aided Wireless Video Transmission , 2021, IEEE Journal on Selected Areas in Communications.

[5]  Federico Chiariotti,et al.  Remote Anomaly Detection in Industry 4.0 Using Resource-Constrained Devices , 2021, 2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[6]  Deniz Gündüz,et al.  Denoising Noisy Neural Networks: A Bayesian Approach With Compensation , 2021, IEEE Transactions on Signal Processing.

[7]  Walid Saad,et al.  Distributed Learning in Wireless Networks: Recent Progress and Future Challenges , 2021, IEEE Journal on Selected Areas in Communications.

[8]  Zhijin Qin,et al.  Semantic Communication Systems for Speech Transmission , 2021, IEEE Journal on Selected Areas in Communications.

[9]  T. Weissman,et al.  An Information-Theoretic Justification for Model Pruning , 2021, AISTATS.

[10]  Deniz Gündüz,et al.  Federated mmWave Beam Selection Utilizing LIDAR Data , 2021, IEEE Wireless Communications Letters.

[11]  Hun-Seok Kim,et al.  Deep Joint Source Channel Coding for Wireless Image Transmission with OFDM , 2021, ICC 2021 - IEEE International Conference on Communications.

[12]  Deniz Gündüz,et al.  Communicate to Learn at the Edge , 2020, IEEE Communications Magazine.

[13]  Deniz Gündüz,et al.  Bandwidth-Agile Image Transmission With Deep Joint Source-Channel Coding , 2020, IEEE Transactions on Wireless Communications.

[14]  Deniz Gündüz,et al.  Wireless Image Retrieval at the Edge , 2020, IEEE Journal on Selected Areas in Communications.

[15]  K. Mikolajczyk,et al.  Joint Device-Edge Inference over Wireless Links with Pruning , 2020, 2020 IEEE 21st International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[16]  Rongrong Ji,et al.  HRank: Filter Pruning Using High-Rank Feature Map , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Michael W. Mahoney,et al.  ZeroQ: A Novel Zero Shot Quantization Framework , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Xianglong Liu,et al.  Towards Unified INT8 Training for Convolutional Neural Network , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Zhenyu A. Liao,et al.  AdaBits: Neural Network Quantization With Adaptive Bit-Widths , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  David Burth Kurka,et al.  DeepJSCC-f: Deep Joint Source-Channel Coding of Images With Feedback , 2019, IEEE Journal on Selected Areas in Information Theory.

[21]  Jun Zhang,et al.  BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems , 2019, 2020 IEEE International Conference on Communications Workshops (ICC Workshops).

[22]  Heiko Schwarz,et al.  DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks , 2019, IEEE Journal of Selected Topics in Signal Processing.

[23]  Bingbing Ni,et al.  Variational Convolutional Neural Network Pruning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Pavlo Molchanov,et al.  Importance Estimation for Neural Network Pruning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Houqiang Li,et al.  Quantization Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Kurt Keutzer,et al.  HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Thomas S. Huang,et al.  Universally Slimmable Networks and Improved Training Techniques , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[28]  Massoud Pedram,et al.  BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services , 2019, 2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED).

[29]  Philip H. S. Torr,et al.  SNIP: Single-shot Network Pruning based on Connection Sensitivity , 2018, ICLR.

[30]  Elad Hoffer,et al.  ACIQ: Analytical Clipping for Integer Quantization of neural networks , 2018, ArXiv.

[31]  David Burth Kurka,et al.  Deep Joint Source-Channel Coding for Wireless Image Transmission , 2018, IEEE Transactions on Cognitive Communications and Networking.

[32]  G. Hua,et al.  LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks , 2018, ECCV.

[33]  Andrew Gordon Wilson,et al.  Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs , 2018, NeurIPS.

[34]  Andrea J. Goldsmith,et al.  Deep Learning for Joint Source-Channel Coding of Text , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[35]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  Tao Zhang,et al.  A Survey of Model Compression and Acceleration for Deep Neural Networks , 2017, ArXiv.

[37]  Jianxin Wu,et al.  ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[38]  Xiangyu Zhang,et al.  Channel Pruning for Accelerating Very Deep Neural Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Xiangyu Zhang,et al.  ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[41]  Timo Aila,et al.  Pruning Convolutional Neural Networks for Resource Efficient Inference , 2016, ICLR.

[42]  Hanan Samet,et al.  Pruning Filters for Efficient ConvNets , 2016, ICLR.

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[45]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[46]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[47]  Lorenzo L. Pesce,et al.  Noise injection for training artificial neural networks: a comparison with weight decay and early stopping. , 2009, Medical physics.

[48]  Tsachy Weissman,et al.  Successive Pruning for Model Compression via Rate Distortion Theory , 2021, ArXiv.

[49]  Berivan Isik Noisy Neural Network Compression for Analog Storage Devices , 2020 .

[50]  Ya Le,et al.  Tiny ImageNet Visual Recognition Challenge , 2015 .

[51]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[52]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[53]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[54]  George L. Turin,et al.  The theory of optimum noise immunity , 1959 .

[55]  R. Mises,et al.  Praktische Verfahren der Gleichungsauflösung . , 1929 .