Machine Learning for Intelligent Optical Networks: A Comprehensive Survey

Abstract With the rapid development of Internet and communication systems, both in the aspect of services and technologies, communication networks have been suffering increasing complexity. It is imperative to improve intelligence in communication networks, and several aspects have been incorporating with Artificial Intelligence (AI) and Machine Learning (ML). The optical network, which plays an important role both in core and access network in communication networks, also faces great challenges of system complexity and the requirement of manual operations. To overcome the current limitations and address the issues of future optical networks, it is essential to deploy more intelligence capability to enable autonomous and flexible network operations. ML techniques are proved to have superiority on solving complex problems, and thus recently, ML techniques have been used for many optical network applications. In this paper, a detailed survey of existing applications of ML for intelligent optical networks is presented. The applications of ML are classified in terms of their use cases, which are categorised into optical network control and resource management, and optical network monitoring and survivability. These applications are analyzed and compared according to the used ML techniques. Besides, a tutorial for ML applications is provided from the aspects of the introduction of common ML algorithms, paradigms of ML, and motivations of applying ML. Lastly, challenges and possible solutions of ML application in intelligent optical networks are also discussed, which intends to inspire future innovations in leveraging ML to build intelligent optical networks.

[1]  Pablo Pavón-Mariño,et al.  Net2plan: an open source network planning tool for bridging the gap between academia and industry , 2015, IEEE Network.

[2]  Min Zhang,et al.  Failure prediction using machine learning and time series in optical network. , 2017, Optics express.

[3]  A.E. Willner,et al.  Optical performance monitoring , 2004, Journal of Lightwave Technology.

[4]  Zuqing Zhu,et al.  When Deep Learning Meets Inter-Datacenter Optical Network Management: Advantages and Vulnerabilities , 2018, Journal of Lightwave Technology.

[5]  Cedric Baudoin,et al.  Towards the Deployment of Machine Learning Solutions in Network Traffic Classification: A Systematic Survey , 2019, IEEE Communications Surveys & Tutorials.

[6]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[7]  J. Kahn,et al.  Compensation of Dispersion and Nonlinear Impairments Using Digital Backpropagation , 2008, Journal of Lightwave Technology.

[8]  C. Siva Ram Murthy,et al.  A Reinforcement Learning Framework for Path Selection and Wavelength Selection in Optical Burst Switched Networks , 2007, IEEE Journal on Selected Areas in Communications.

[9]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[10]  Francesco Musumeci,et al.  Machine-Learning-Assisted Routing in SDN-Based Optical Networks , 2018, 2018 European Conference on Optical Communication (ECOC).

[11]  Xin Wang,et al.  Machine Learning for Networking: Workflow, Advances and Opportunities , 2017, IEEE Network.

[12]  F. Richard Yu,et al.  A Survey of Machine Learning Techniques Applied to Software Defined Networking (SDN): Research Issues and Challenges , 2019, IEEE Communications Surveys & Tutorials.

[13]  Rui Xu,et al.  Survey of clustering algorithms , 2005, IEEE Transactions on Neural Networks.

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Yongli Zhao,et al.  Multi-link faults localization and restoration based on fuzzy fault set for dynamic optical networks. , 2013, Optics express.

[16]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[17]  Hwee Pink Tan,et al.  Machine Learning in Wireless Sensor Networks: Algorithms, Strategies, and Applications , 2014, IEEE Communications Surveys & Tutorials.

[18]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[19]  Mohsen Guizani,et al.  Deep Learning for IoT Big Data and Streaming Analytics: A Survey , 2017, IEEE Communications Surveys & Tutorials.

[20]  Gang Wang,et al.  Global Context-Aware Attention LSTM Networks for 3D Action Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[22]  Zhao Yang Dong,et al.  Optical Performance Monitoring Using Artificial Neural Network Trained With Asynchronous Amplitude Histograms , 2010, IEEE Photonics Technology Letters.

[23]  A.E. Willner,et al.  Optical Performance Monitoring Using Artificial Neural Networks Trained With Eye-Diagram Parameters , 2009, IEEE Photonics Technology Letters.

[24]  Ehsan Mesbahi,et al.  Artificial neural networks: fundamentals , 2003 .

[25]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Frank R. Kschischang,et al.  Optical Nonlinear Phase Noise Compensation for $9\times 32$ -Gbaud PolDM-16 QAM Transmission Using a Code-Aided Expectation-Maximization Algorithm , 2015, Journal of Lightwave Technology.

[27]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[28]  Zheng Yan,et al.  A survey on network data collection , 2018, J. Netw. Comput. Appl..

[29]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[30]  Ljiljana Trajkovic,et al.  Deflection routing in complex networks , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[31]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[32]  Faisal Nadeem Khan,et al.  Simultaneous optical performance monitoring and modulation format/bit-rate identification using principal component analysis , 2014, IEEE/OSA Journal of Optical Communications and Networking.

[33]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[34]  Yuefeng Ji,et al.  All-optical signal processing technologies in flexible optical networks , 2019, Photonic Network Communications.

[35]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[36]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[37]  Namkyoo Park,et al.  Optimization of SNR improvement in the noncoherent OTDR based on simplex codes , 2006, Journal of Lightwave Technology.

[38]  Keren Bergman,et al.  Dynamic power pre-adjustments with machine learning that mitigate EDFA excursions during defragmentation , 2017, 2017 Optical Fiber Communications Conference and Exhibition (OFC).

[39]  Amina Adadi,et al.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) , 2018, IEEE Access.

[40]  Kin K. Leung,et al.  Adaptive Federated Learning in Resource Constrained Edge Computing Systems , 2018, IEEE Journal on Selected Areas in Communications.

[41]  Roberto Proietti,et al.  Self-Taught Anomaly Detection With Hybrid Unsupervised/Supervised Machine Learning in Optical Networks , 2019, Journal of Lightwave Technology.

[42]  Chao Lu,et al.  An Optical Communication's Perspective on Machine Learning and Its Applications , 2019, Journal of Lightwave Technology.

[43]  E. Riccardi,et al.  Fully Disaggregated ROADM White Box with NETCONF/YANG Control, Telemetry, and Machine Learning-based Monitoring , 2018, 2018 Optical Fiber Communications Conference and Exposition (OFC).

[44]  Piero Castoldi,et al.  Marginless Operation of Optical Networks , 2019, Journal of Lightwave Technology.

[45]  Patricia Layec,et al.  Seamless Optical Path Restoration with Just-in-Time Resource Allocation Leveraging Machine Learning , 2018, 2018 European Conference on Optical Communication (ECOC).

[46]  Huaxi Gu,et al.  Improving Cloud-Based IoT Services Through Virtual Network Embedding in Elastic Optical Inter-DC Networks , 2019, IEEE Internet of Things Journal.

[47]  Francesco Musumeci,et al.  Machine Learning-Based Routing and Wavelength Assignment in Software-Defined Optical Networks , 2019, IEEE Transactions on Network and Service Management.

[48]  Christoph Molnar,et al.  Interpretable Machine Learning , 2020 .

[49]  Hai Le Vu,et al.  OBS contention resolution performance , 2007, Perform. Evaluation.

[50]  D. Ishii,et al.  A self-learning route selection scheme using multi-path searching packets in an OBS network , 2005, HPSR. 2005 Workshop on High Performance Switching and Routing, 2005..

[51]  Barry Smyth,et al.  Retrieval, reuse, revision and retention in case-based reasoning , 2005, The Knowledge Engineering Review.

[52]  C. Siva Ram Murthy,et al.  A Multi-Agent Reinforcement Learning Approach to Path Selection in Optical Burst Switching Networks , 2009, 2009 IEEE International Conference on Communications.

[53]  Qi Hao,et al.  Deep Learning for Intelligent Wireless Networks: A Comprehensive Survey , 2018, IEEE Communications Surveys & Tutorials.

[54]  Mounia Lourdiane,et al.  Quality of Transmission Prediction with Machine Learning for Dynamic Operation of Optical WDM Networks , 2017, 2017 European Conference on Optical Communication (ECOC).

[55]  Konstantinos Christodoulopoulos,et al.  Elastic Bandwidth Allocation in Flexible OFDM-Based Optical Networks , 2010, Journal of Lightwave Technology.

[56]  Gil Zussman,et al.  A machine learning approach for dynamic optical channel add/drop strategies that minimize EDFA power excursions , 2016 .

[57]  Xiaobo Wu,et al.  A Method of Emotional Analysis of Movie Based on Convolution Neural Network and Bi-directional LSTM RNN , 2017, 2017 IEEE Second International Conference on Data Science in Cyberspace (DSC).

[58]  Yi Pan,et al.  A study of the routing and spectrum allocation in spectrum-sliced Elastic Optical Path networks , 2011, 2011 Proceedings IEEE INFOCOM.

[59]  Subhadeep Sahoo,et al.  Adaptive path splitting based survivable virtual network embedding in elastic optical networks , 2020 .

[60]  J. Bobadilla,et al.  Recommender systems survey , 2013, Knowl. Based Syst..

[61]  Sudharman K. Jayaweera,et al.  A Survey on Machine-Learning Techniques in Cognitive Radios , 2013, IEEE Communications Surveys & Tutorials.

[62]  Abdelhakim Hafid,et al.  A Reinforcement Learning-Based Deflection Routing Scheme for Buffer-Less OBS Networks , 2008, IEEE GLOBECOM 2008 - 2008 IEEE Global Telecommunications Conference.

[63]  Marc Ruiz,et al.  Applying Data Visualization for Failure Localization , 2018, 2018 Optical Fiber Communications Conference and Exposition (OFC).

[64]  GhaffariAli,et al.  Software defined networks , 2016 .

[65]  Zuqing Zhu,et al.  Dynamic RMSA in elastic optical networks with an adaptive genetic algorithm , 2012, 2012 IEEE Global Communications Conference (GLOBECOM).

[66]  Luca Barletta,et al.  Machine-learning method for quality of transmission prediction of unestablished lightpaths , 2018, IEEE/OSA Journal of Optical Communications and Networking.

[67]  Stephan Pachnicke,et al.  Machine Learning-Based Traffic Prediction for Optical Switching Resource Allocation in Hybrid Intra-Data Center Networks , 2019, 2019 Optical Fiber Communications Conference and Exhibition (OFC).

[68]  Blaise Agüera y Arcas,et al.  Federated Learning of Deep Networks using Model Averaging , 2016, ArXiv.

[69]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[70]  Hui Yang,et al.  Efficient Hybrid Multi-Faults Location Based on Hopfield Neural Network in 5G Coexisting Radio and Optical Wireless Networks , 2019, IEEE Transactions on Cognitive Communications and Networking.

[71]  Xiaoxia Wu,et al.  Applications of Artificial Neural Networks in Optical Performance Monitoring , 2009, Journal of Lightwave Technology.

[72]  Yang Li,et al.  Leveraging double-agent-based deep reinforcement learning to global optimization of elastic optical networks with enhanced survivability. , 2019, Optics express.

[73]  Takeshi Hoshida,et al.  Convolutional neural network-based optical performance monitoring for optical transport networks , 2018, IEEE/OSA Journal of Optical Communications and Networking.

[74]  Masaki Kobayashi,et al.  Hyperbolic Hopfield Neural Networks , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[75]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[76]  Kumar N. Sivarajan,et al.  Routing and wavelength assignment in all-optical networks , 1995, TNET.

[77]  Ali Ghaffari,et al.  Software defined networks: A survey , 2016, J. Netw. Comput. Appl..

[78]  Danish Rafique,et al.  Fault Management Based on Machine Learning , 2019, 2019 Optical Fiber Communications Conference and Exhibition (OFC).

[79]  Darko Zibar,et al.  Machine Learning Techniques in Optical Communication , 2015, Journal of Lightwave Technology.

[80]  Chao Lu,et al.  Optical Performance Monitoring: A Review of Current and Future Technologies , 2016, Journal of Lightwave Technology.

[81]  Víctor López,et al.  Elastic Spectrum Allocation for Time-Varying Traffic in FlexGrid Optical Networks , 2013, IEEE Journal on Selected Areas in Communications.

[82]  Vinod Vokkarane,et al.  Burst segmentation: an approach for reducing packet loss in optical burst switched networks , 2002, 2002 IEEE International Conference on Communications. Conference Proceedings. ICC 2002 (Cat. No.02CH37333).

[83]  Elio Salvadori,et al.  Cognitive, Heterogeneous and Reconfigurable Optical Networks: The CHRON Project , 2014, Journal of Lightwave Technology.

[84]  Eiji Oki,et al.  Routing and Spectrum Allocation in Elastic Optical Networks: A Tutorial , 2015, IEEE Communications Surveys & Tutorials.

[85]  Xiaoping Zheng,et al.  Routing without Routing Algorithms: An AI-Based Routing Paradigm for Multi-Domain Optical Networks , 2019, 2019 Optical Fiber Communications Conference and Exhibition (OFC).

[86]  Massimo Tornatore,et al.  Routing and Spectrum Assignment Integrating Machine-Learning-Based QoT Estimation in Elastic Optical Networks , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[87]  Admela Jukan,et al.  Machine-learning-based prediction for resource (Re)allocation in optical data center networks , 2018, IEEE/OSA Journal of Optical Communications and Networking.

[88]  Abdelhakim Hafid,et al.  Novel reinforcement learning-based approaches to reduce loss probability in buffer-less OBS networks , 2009, Comput. Networks.

[89]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[90]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[91]  Michiaki Hayashi Machine Learning-assisted Management of a Virtualized Network , 2018, 2018 Optical Fiber Communications Conference and Exposition (OFC).

[92]  Li Yang,et al.  Adaptive path selection in OBS networks , 2006, Journal of Lightwave Technology.

[93]  Yuefeng Ji,et al.  Towards converged, collaborative and co-automatic (3C) optical networks , 2018, Science China Information Sciences.

[94]  Chi Sing Leung,et al.  Extreme learning machine for estimating blocking probability of bufferless OBS/OPS networks , 2017, IEEE/OSA Journal of Optical Communications and Networking.

[95]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[96]  Danish Rafique,et al.  Machine learning for network automation: overview, architecture, and applications [Invited Tutorial] , 2018, IEEE/OSA Journal of Optical Communications and Networking.

[97]  Sergey Levine,et al.  End-to-End Training of Deep Visuomotor Policies , 2015, J. Mach. Learn. Res..

[98]  Hui Yang,et al.  Accurate Fault Location based on Deep Neural Evolution Network in Optical Networks for 5G and Beyond , 2019, 2019 Optical Fiber Communications Conference and Exhibition (OFC).

[99]  Piero Castoldi,et al.  Network Telemetry Streaming Services in SDN-Based Disaggregated Optical Networks , 2018, Journal of Lightwave Technology.

[100]  S. Bigo,et al.  Proactive Fiber Damage Detection in Real-time Coherent Receiver , 2017, 2017 European Conference on Optical Communication (ECOC).

[101]  Andreas Maletti,et al.  Survey: Finite-state technology in natural language processing , 2017, Theor. Comput. Sci..

[102]  Xiaoxia Wu,et al.  ANN-Based Optical Performance Monitoring of QPSK Signals Using Parameters Derived From Balanced-Detected Asynchronous Diagrams , 2011, IEEE Photonics Technology Letters.

[103]  Takeshi Hoshida,et al.  OSNR monitoring by deep neural networks trained with asynchronously sampled data , 2016, 2016 21st OptoElectronics and Communications Conference (OECC) held jointly with 2016 International Conference on Photonics in Switching (PS).

[104]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[105]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[106]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[107]  Jie Zhang,et al.  Leveraging Deep Learning to Achieve Efficient Resource Allocation with Traffic Evaluation in Datacenter Optical Networks , 2018, 2018 Optical Fiber Communications Conference and Exposition (OFC).

[108]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[109]  Marc Ruiz,et al.  Learning From the Optical Spectrum: Failure Detection and Identification , 2019, Journal of Lightwave Technology.

[110]  Ioannis Tomkos,et al.  A Cognitive Quality of Transmission Estimator for Core Optical Networks , 2012, Journal of Lightwave Technology.

[111]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[112]  Halima Elbiaze,et al.  Graphical Probabilistic Routing Model for OBS Networks with Realistic Traffic Scenario , 2009, GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference.

[113]  Nen-Fu Huang,et al.  Performance analysis of deflection routing in optical burst-switched networks , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[114]  Roberto Proietti,et al.  Experimental demonstration of machine-learning-aided QoT estimation in multi-domain elastic optical networks with alien wavelengths , 2018, IEEE/OSA Journal of Optical Communications and Networking.

[115]  Mohit Chamania,et al.  Artificial Intelligence (AI) Methods in Optical Networks: A Comprehensive Survey , 2018, Opt. Switch. Netw..

[116]  I A Basheer,et al.  Artificial neural networks: fundamentals, computing, design, and application. , 2000, Journal of microbiological methods.

[117]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[118]  Huaxi Gu,et al.  Flexible and Scalable Optical Interconnects for Data Centers: Trends and Challenges , 2019, IEEE Communications Magazine.

[119]  Miroslaw Klinkowski,et al.  A performance survey on deflection routing techniques for OBS networks , 2009, 2009 11th International Conference on Transparent Optical Networks.

[120]  Achim Autenrieth,et al.  Cognitive Assurance Architecture for Optical Network Fault Management , 2018, Journal of Lightwave Technology.

[121]  Yvan Pointurier,et al.  Design of low-margin optical networks , 2017, IEEE/OSA Journal of Optical Communications and Networking.

[122]  Min Zhang,et al.  Convolutional Neural Network-Based Deep Learning for Intelligent OSNR Estimation on Eye Diagrams , 2017, 2017 European Conference on Optical Communication (ECOC).

[123]  Ljiljana Trajkovic,et al.  A Predictive Q-Learning Algorithm for Deflection Routing in Buffer-less Networks , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[124]  Roberto Proietti,et al.  Deep-RMSA: A Deep-Reinforcement-Learning Routing, Modulation and Spectrum Assignment Agent for Elastic Optical Networks , 2018, 2018 Optical Fiber Communications Conference and Exposition (OFC).

[125]  Xiaoxia Wu,et al.  Optical performance monitoring of PSK data channels using artificial neural networks trained with parameters derived from delay-tap asynchronous diagrams via balanced detection , 2009, 2009 35th European Conference on Optical Communication.