Network monitoring: Present and future

Network monitoring guides network operators in understanding the current behavior of a network. Therefore, accurate and efficient monitoring is vital to ensure that the network operates according to the intended behavior and then to troubleshoot any deviations. However, the current practice of network-monitoring largely depends on manual operations, and thus enterprises spend a significant portion of their budgets on the workforce that monitor their networks. We analyze present network-monitoring technologies, identify open problems, and suggest future directions. In particular, our findings are based on two different analyses. The first analysis assesses how well present technologies integrate with the entire cycle of network-management operations: design, deployment, and monitoring. Network operators first design network configurations, given a set of requirements, then they deploy the new design, and finally they verify it by continuously monitoring the network’s behavior. One of our observations is that the efficiency of this cycle can be greatly improved by automated deployment of pre-designed configurations, in response to changes in monitored network behavior. Our second analysis focuses on network-monitoring technologies and group issues in these technologies into five categories. Such grouping leads to the identification of major problem groups in network monitoring, e.g., efficient management of increasing amounts of measurements for storage, analysis, and presentation. We argue that continuous effort is needed in improving network-monitoring since the presented problems will become even more serious in the future, as networks grow in size and carry more data.

[1]  Andrew W. Moore,et al.  Internet traffic classification using bayesian analysis techniques , 2005, SIGMETRICS '05.

[2]  Paramvir Bahl,et al.  Towards highly reliable enterprise network services via inference of multi-level dependencies , 2007, SIGCOMM '07.

[3]  Renata Teixeira,et al.  Early Recognition of Encrypted Applications , 2007, PAM.

[4]  Yin Zhang,et al.  Finding critical traffic matrices , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[5]  HariGovind V. Ramasamy,et al.  Automated Incident Management for a Platform-as-a-Service Cloud , 2011, Hot-ICE.

[6]  Pradipta De,et al.  Tracking configuration changes proactively in large IT environments , 2012, 2012 IEEE Network Operations and Management Symposium.

[7]  Dennis Gamayunov,et al.  Visualization of complex attacks and state of attacked network , 2009, 2009 6th International Workshop on Visualization for Cyber Security.

[8]  Anja Feldmann,et al.  Deriving traffic demands for operational IP networks: methodology and experience , 2000, SIGCOMM.

[9]  Danny Dolev,et al.  Facilitating Efficient and Reliable Monitoring through HAMSA , 2003, Integrated Network Management.

[10]  Hyong S. Kim,et al.  Automatic detection of firewall misconfigurations using firewall and network routing policies , 2009 .

[11]  Jun Zhang,et al.  Network Traffic Classification Using Correlation Information , 2013, IEEE Transactions on Parallel and Distributed Systems.

[12]  Steve Romig,et al.  The OSU Flow-tools Package and CISCO NetFlow Logs , 2000, LISA.

[13]  WillingerWalter,et al.  Spatio-temporal compressive sensing and internet traffic matrices , 2009 .

[14]  Emilio Leonardi,et al.  Estimating Dynamic Traffic Matrices by Using Viable Routing Changes , 2007, IEEE/ACM Transactions on Networking.

[15]  Fulvio Risso,et al.  Lightweight, Payload-Based Traffic Classification: An Experimental Evaluation , 2008, 2008 IEEE International Conference on Communications.

[16]  Stefano Giordano,et al.  On Multi-gigabit Packet Capturing with Multi-core Commodity Hardware , 2012, PAM.

[17]  Benoit Claise,et al.  Advanced network monitoring brings life to the awareness plane , 2008, IEEE Communications Magazine.

[18]  Hyong S. Kim,et al.  Improving manageability through reorganization of routing-policy configurations , 2012, Comput. Networks.

[19]  Rajeev Rastogi,et al.  Robust Monitoring of Link Delays and Faults in IP Networks , 2003, IEEE/ACM Transactions on Networking.

[20]  George Varghese,et al.  Every microsecond counts: tracking fine-grain latencies with a lossy difference aggregator , 2009, SIGCOMM '09.

[21]  Srikanth Kandula,et al.  Shrink: a tool for failure diagnosis in IP networks , 2005, MineNet '05.

[22]  PadhyeJitendra,et al.  Detailed diagnosis in enterprise networks , 2009 .

[23]  Xiaoyun Zhu,et al.  DAPA: Diagnosing Application Performance Anomalies for Virtualized Infrastructures , 2012, Hot-ICE.

[24]  Michele Wright Using policies for effective network management , 1999, Int. J. Netw. Manag..

[25]  Malgorzata Steinder,et al.  A survey of fault localization techniques in computer networks , 2004, Sci. Comput. Program..

[26]  Kostas G. Anagnostakis,et al.  cing: measuring network-internal delays using only existing infrastructure , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[27]  Grenville J. Armitage,et al.  A survey of techniques for internet traffic classification using machine learning , 2008, IEEE Communications Surveys & Tutorials.

[28]  George Varghese,et al.  Automatically inferring patterns of resource consumption in network traffic , 2003, SIGCOMM '03.

[29]  kc claffy,et al.  Bandwidth estimation: metrics, measurement techniques, and tools , 2003, IEEE Netw..

[30]  William Yurcik,et al.  Visualizing NetFlows for Security at Line Speed: The SIFT Tool Suite , 2005, LISA.

[31]  Alan D. George,et al.  Adaptive Sampling for Network Management , 2001, Journal of Network and Systems Management.

[32]  Konstantina Papagiannaki,et al.  Toward the Accurate Identification of Network Applications , 2005, PAM.

[33]  Yin Zhang,et al.  Troubleshooting chronic conditions in large IP networks , 2008, CoNEXT '08.

[34]  Walter Willinger,et al.  cSamp: A System for Network-Wide Flow Monitoring , 2008, NSDI.

[35]  Vasilis Friderikos,et al.  Cross-Layer Optimization to Maximize Fairness Among TCP Flows of Different TCP Flavors , 2008, IEEE GLOBECOM 2008 - 2008 IEEE Global Telecommunications Conference.

[36]  Aiko Pras,et al.  On the future of Internet management technologies , 2003, IEEE Commun. Mag..

[37]  Chase Cotton,et al.  Packet-level traffic measurements from the Sprint IP backbone , 2003, IEEE Netw..

[38]  Jens B. Schmitt,et al.  Packet marking for integrated load control , 2005, 2005 9th IFIP/IEEE International Symposium on Integrated Network Management, 2005. IM 2005..

[39]  Michael Hicks,et al.  Passive aggressive measurement with MGRP , 2009, SIGCOMM '09.

[40]  A. L. Narasimha Reddy,et al.  NetViewer: A Network Traffic Visualization and Analysis Tool , 2005, LISA.

[41]  Radu State,et al.  SAFEM: Scalable analysis of flows with entropic measures and SVM , 2012, 2012 IEEE Network Operations and Management Symposium.

[42]  Vinicius Tavares Guimaraes,et al.  UAMA: a unified architecture for active measurements in IP networks; End-to-end objetive quality indicators , 2007, 2007 10th IFIP/IEEE International Symposium on Integrated Network Management.

[43]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[44]  Chadi Barakat,et al.  Ranking flows from sampled traffic , 2005, CoNEXT '05.

[45]  Toby J. Teorey,et al.  Using RMON Matrix Group Extensions to Analyze Internetworking Problems , 2004, Journal of Network and Systems Management.

[46]  Hyong S. Kim,et al.  Netpiler: detection of ineffective router configurations , 2009, IEEE Journal on Selected Areas in Communications.

[47]  Zhi-Li Zhang,et al.  Practical delay monitoring for ISPs , 2005, CoNEXT '05.

[48]  Nick Feamster,et al.  Detecting BGP configuration faults with static analysis , 2005 .

[49]  John Sellens Thresh--A Data-Directed SNMP Threshold Poller , 2000, LISA.

[50]  Taesang Choi,et al.  Content-aware Internet application traffic measurement and analysis , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[51]  Tobias Oetiker,et al.  MRTG: The Multi Router Traffic Grapher , 1998, LISA.

[52]  Chen-Nee Chuah,et al.  ProgME: Towards Programmable Network MEasurement , 2007, IEEE/ACM Transactions on Networking.

[53]  Marina Thottan,et al.  Distributed network monitoring with bounded link utilization in IP networks , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[54]  Ranveer Chandra,et al.  What's going on?: learning communication rules in edge networks , 2008, SIGCOMM '08.

[55]  Christopher J. Tengi,et al.  PatchMaker: A Physical Network Patch Manager Tool , 2004, LISA.

[56]  Oliver Spatscheck,et al.  Accurate, scalable in-network identification of p2p traffic using application signatures , 2004, WWW '04.

[57]  Michalis Faloutsos,et al.  Internet traffic classification demystified: myths, caveats, and the best practices , 2008, CoNEXT '08.

[58]  George Varghese,et al.  Building a better NetFlow , 2004, SIGCOMM.

[59]  Elena Baralis,et al.  Data mining techniques for effective and scalable traffic analysis , 2005, 2005 9th IFIP/IEEE International Symposium on Integrated Network Management, 2005. IM 2005..

[60]  Mun Choon Chan,et al.  A scalable monitoring approach based on aggregation and refinement , 2002, IEEE J. Sel. Areas Commun..

[61]  Danny Raz,et al.  Efficient reactive monitoring , 2001, Proceedings IEEE INFOCOM 2001. Conference on Computer Communications. Twentieth Annual Joint Conference of the IEEE Computer and Communications Society (Cat. No.01CH37213).

[62]  Daniel A. Keim,et al.  Large-Scale Network Monitoring for Visual Analysis of Attacks , 2008, VizSEC.

[63]  Konstantina Papagiannaki,et al.  Long-term forecasting of Internet backbone traffic: observations and initial models , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[64]  Ítalo S. Cunha,et al.  Measurement methods for fast and accurate blackhole identification with binary tomography , 2009, IMC '09.

[65]  Danny Raz,et al.  Toward efficient monitoring , 2000, IEEE Journal on Selected Areas in Communications.

[66]  Michalis Faloutsos,et al.  BLINC: multilevel traffic classification in the dark , 2005, SIGCOMM '05.

[67]  R. Rhinehart,et al.  Correlation , 2014, BMJ : British Medical Journal.

[68]  Ravishankar K. Iyer,et al.  An adaptive architecture for monitoring and failure analysis of high-speed networks , 2002, Proceedings International Conference on Dependable Systems and Networks.

[69]  Brian Trammell,et al.  Specification of the IP Flow Information Export (IPFIX) File Format , 2009, RFC.

[70]  Nick G. Duffield,et al.  Trajectory sampling for direct traffic observation , 2001, TNET.

[71]  Antonio Puliafito,et al.  Programmable agents for flexible QoS management in IP networks , 2000, IEEE Journal on Selected Areas in Communications.

[72]  Hervé Rivano,et al.  Optimal positioning of active and passive monitoring devices , 2005, CoNEXT '05.

[73]  Albert G. Greenberg,et al.  The cutting EDGE of IP router configuration , 2004, Comput. Commun. Rev..

[74]  Rolf Stadler,et al.  A-GAP: An Adaptive Protocol for Continuous Network Monitoring with Accuracy Objectives , 2007, IEEE Transactions on Network and Service Management.

[75]  Albert G. Greenberg,et al.  IP fault localization via risk modeling , 2005, NSDI.

[76]  Albert G. Greenberg,et al.  Detection and Localization of Network Black Holes , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[77]  Benoit Claise,et al.  Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information , 2008, RFC.

[78]  Brian Trammell,et al.  NAF: The NetSA Aggregated Flow Tool Suite , 2006, LISA.

[79]  Jia Wang,et al.  Locating internet bottlenecks: algorithms, measurements, and implications , 2004, SIGCOMM '04.

[80]  David Plonka,et al.  FlowScan: A Network Traffic Flow Reporting and Visualization Tool , 2000, LISA.

[81]  George Pavlou,et al.  Exploiting agent mobility for large-scale network monitoring , 2002, IEEE Netw..

[82]  Yong-Hoon Choi,et al.  In-service QoS monitoring of real-time applications using SM MIB , 2005, Int. J. Netw. Manag..

[83]  Dan Gunter,et al.  Scalable analysis of network measurements with Hadoop and Pig , 2012, 2012 IEEE Network Operations and Management Symposium.

[84]  Paramvir Bahl,et al.  Detailed diagnosis in enterprise networks , 2009, SIGCOMM '09.

[85]  Martin Bjorklund,et al.  YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF) , 2010 .

[86]  Nick G. Duffield,et al.  Trajectory Sampling With Unreliable Reporting , 2008, IEEE/ACM Transactions on Networking.

[87]  Jacques Labetoulle,et al.  An efficient polling layer for SNMP , 2000, NOMS 2000. 2000 IEEE/IFIP Network Operations and Management Symposium 'The Networked Planet: Management Beyond 2000' (Cat. No.00CB37074).

[88]  Leonardo Neumeyer,et al.  S4: Distributed Stream Computing Platform , 2010, 2010 IEEE International Conference on Data Mining Workshops.

[89]  Manish Karir,et al.  Flamingo: Visualizing Internet Traffic , 2006, 2006 IEEE/IFIP Network Operations and Management Symposium NOMS 2006.

[90]  Hyong S. Kim,et al.  Correlation, visualization, and usability analysis of routing policy configurations , 2010, IEEE Transactions on Network and Service Management.

[91]  Matthew J. Luckie,et al.  Towards improving packet probing techniques , 2001, IMW '01.

[92]  Chen-Nee Chuah,et al.  Measurement-Aware Monitor Placement and Routing: A Joint Optimization Approach for Network-Wide Measurements , 2012, IEEE Transactions on Network and Service Management.

[93]  George Varghese,et al.  Network monitoring using traffic dispersion graphs (tdgs) , 2007, IMC '07.

[94]  Walter Willinger,et al.  Spatio-temporal compressive sensing and internet traffic matrices , 2009, SIGCOMM '09.

[95]  Christopher J. Tengi,et al.  autoMAC: A Tool for Automating Network Moves, Adds, and Changes , 2004, LISA.

[96]  Martin A. Musicante,et al.  ANEMONA: a programming language for network monitoring applications , 2008, Int. J. Netw. Manag..

[97]  Giuseppe Di Battista,et al.  26 Computer Networks , 2004 .

[98]  Ratul Mahajan,et al.  Understanding BGP misconfiguration , 2002, SIGCOMM 2002.

[99]  Robert Beverly RTG: A Scalable SNMP Statistics Architecture for Service Providers , 2002, LISA.

[100]  David Wetherall,et al.  Scriptroute: A Public Internet Measurement Facility , 2003, USENIX Symposium on Internet Technologies and Systems.

[101]  Darryl Veitch,et al.  A measurement-friendly network (MFN) architecture , 2006, INM '06.

[102]  Long Wang,et al.  Towards an Understanding of Oversubscription in Cloud , 2012, Hot-ICE.

[103]  Mark Crovella,et al.  Graph wavelets for spatial traffic analysis , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[104]  T. S. Eugene Ng,et al.  Synergy2Cloud: Introducing Cross-Sharing of Application Experiences Into the Cloud Management Cycle , 2012, Hot-ICE.

[105]  Anja Feldmann,et al.  IP network configuration for intradomain traffic engineering , 2001, IEEE Netw..

[106]  Jürgen Schönwälder,et al.  Network Configuration Protocol (NETCONF) , 2011, RFC.

[107]  Tina Wong,et al.  Automatic discovery of relationships across multiple network layers , 2007, INM '07.

[108]  Hyong S. Kim,et al.  NetPolis: Modeling of Inter-Domain Routing Policies , 2008, IEEE GLOBECOM 2008 - 2008 IEEE Global Telecommunications Conference.

[109]  Mark Burgess,et al.  Key research challenges in network management , 2007, IEEE Communications Magazine.

[110]  Xiaohong Guan,et al.  Accurate Classification of the Internet Traffic Based on the SVM Method , 2007, 2007 IEEE International Conference on Communications.

[111]  Myungjin Lee,et al.  Enabling Flow-level Latency Measurements across Routers in Data Centers , 2011, Hot-ICE.

[112]  Zhi-Li Zhang,et al.  Profiling internet backbone traffic: behavior models and applications , 2005, SIGCOMM '05.

[113]  Saverio Niccolini,et al.  Stream-monitoring with blockmon: convergence of network measurements and data analytics platforms , 2013, CCRV.

[114]  HicksMichael,et al.  Passive aggressive measurement with MGRP , 2009 .

[115]  Krishna P. Gummadi,et al.  Sprobe: A fast technique for measuring bottleneck bandwidth in uncooperative environments , 2002, INFOCOM 2002.

[116]  Xin Wu,et al.  NetPilot: automating datacenter network failure mitigation , 2012, SIGCOMM '12.