Overfitting and Underfitting Analysis for Deep Learning Based End-to-end Communication Systems

In this paper, we study the deep learning (DL) based end- to-end transmission systems, then we present the analysis for the underfitting and overfitting phenomena which happen during the training of the neural networks (NNs). Different from the DL based image processing, the transmission systems require the bit error rate to be as low as 10–4 to achieve high reliability, thus the training errors induced by the underfitting and overfitting may greatly degrade the transmission reliability performances of DL-based communications. In considered DL-based communication systems, we propose to use the average, variance and minimum of transmitted signals' minimum Euclidean distance to estimate the effects of underfitting and overfitting on the error rate performances in terms of the energy per bit to noise power spectral ratio Eb/ No of signals. Furthermore, we propose to apply the regularization scheme to alleviate the overfitting issue. Simulations are performed to demonstrate the underfitting and overfitting analysis for transmission systems over the additive white Gaussian noise (AWGN) channel, and validate the improved performances by applying the regularization.

[1]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[2]  Geoffrey Ye Li,et al.  Deep Learning-Based Channel Estimation for Beamspace mmWave Massive MIMO Systems , 2018, IEEE Wireless Communications Letters.

[3]  Jakob Hoydis,et al.  An Introduction to Deep Learning for the Physical Layer , 2017, IEEE Transactions on Cognitive Communications and Networking.

[4]  V. Zinoviev,et al.  Codes on euclidean spheres , 2001 .

[5]  T. Charles Clancy,et al.  Convolutional Radio Modulation Recognition Networks , 2016, EANN.

[6]  Norman C. Beaulieu,et al.  Euclidean and Space–Time Block Codes: Relationship, Optimality, Performance Analysis Revisited , 2015, IEEE Transactions on Communications.

[7]  Yair Be'ery,et al.  Learning to decode linear codes using deep learning , 2016, 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[8]  Thamer M. Jamel,et al.  CHANNEL CODING , 2018, Digital Communication For Practicing Engineers.

[9]  Aaron C. Courville,et al.  Improved Conditional VRNNs for Video Prediction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[11]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[12]  Gang Chen,et al.  Adaptive Lightweight Regularization Tool for Complex Analytics , 2018, 2018 IEEE 34th International Conference on Data Engineering (ICDE).

[13]  Luis Perez,et al.  The Effectiveness of Data Augmentation in Image Classification using Deep Learning , 2017, ArXiv.

[14]  Leslie N. Smith,et al.  A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay , 2018, ArXiv.