Measurement Challenges for Cyber Cyber Digital Twins: Experiences from the Deployment of Facebook's WW Simulation System

A cyber cyber digital twin is a deployed software model that executes in tandem with the system it simulates, contributing to, and drawing from, the system's behaviour. This paper outlines Facebook's cyber cyber digital twin, dubbed WW, a twin of Facebook's WWW platform, built using web-enabled simulation. The paper focuses on the current research challenges and opportunities in the area of measurement. Measurement challenges lie at the heart of modern simulation. They directly impact how we use simulation outcomes for automated online and semi-automated offline decision making. Measurements also encompas how we verify and validate those outcomes. Modern simulation systems are increasingly becoming more like cyber cyber digital twins, effectively moving from manual to automated decision making, hence, these measurement challenges acquire ever greater significance.

[1]  Michael Fagan.  Design and Code Inspections to Reduce Errors in Program Development , 1976, IBM Syst. J..

[2]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[3]  J. Voas,et al.  Software Testability: The New Verification , 1995, IEEE Softw..

[4]  Clayton L. Hanson,et al.  Stochastic Weather Simulation: Overview and Analysis of Two Commonly Used Models , 1996 .

[5]  A. Müller.  Integral Probability Metrics and Their Generating Classes of Functions , 1997, Advances in Applied Probability.

[6]  K. Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP '00.

[7]  Prioritizing Test Cases For Regression Testing , 2001, IEEE Trans. Software Eng..

[8]  Tsong Yueh Chen,et al.  Metamorphic testing of programs on partial differential equations: a case study , 2002, Proceedings 26th Annual International Computer Software and Applications.

[9]  Sergio Cavalieri,et al.  Simulation in the supply chain context: a survey , 2004, Comput. Ind..

[10]  Christopher H. Bryant,et al.  Functional genomic hypothesis generation and experimentation by a robot scientist , 2004, Nature.

[11]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[12]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[13]  Hans-Peter Kriegel,et al.  Integrating structured biological data by Kernel Maximum Mean Discrepancy , 2006, ISMB.

[14]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[15]  S. Goodman.  A dirty dozen: twelve p-value misconceptions. , 2008, Seminars in hematology.

[16]  Joaquín Muñoz-García,et al.  A test for the two-sample problem based on empirical characteristic functions , 2008, Comput. Stat. Data Anal..

[17]  Mark Harman,et al.  Regression Testing Minimisation, Selection and Prioritisation - A Survey , 2009 .

[18]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[19]  Mark Harman,et al.  Automated web application testing using search based software engineering , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[20]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[21]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[22]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[23]  Kenji Fukumizu,et al.  Hypothesis testing using pairwise distances and associated kernels , 2012, ICML.

[24]  Tracy Hall,et al.  A Systematic Literature Review on Fault Prediction Performance in Software Engineering , 2012, IEEE Transactions on Software Engineering.

[25]  Alberto Bacchelli,et al.  Expectations, outcomes, and challenges of modern code review , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[26]  Dror G. Feitelson,et al.  Development and Deployment at Facebook , 2013, IEEE Internet Computing.

[27]  Myra B. Cohen,et al.  Automated testing of GUI applications: Models, tools, and controlling flakiness , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[28]  Aiko Fallas Yamashita,et al.  Do developers care about code smells? An exploratory survey , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[29]  Claire Le Goues,et al.  Current challenges in automatic software repair , 2013, Software Quality Journal.

[30]  Mark Harman,et al.  Babel Pidgin: SBSE Can Grow and Graft Entirely New Functionality into a Real World System , 2014, SSBSE.

[31]  Darko Marinov,et al.  An empirical analysis of flaky tests , 2014, SIGSOFT FSE.

[32]  Mark Harman,et al.  An analysis of the relationship between conditional entropy and failed error propagation in software testing , 2014, ICSE.

[33]  Hussein Zedan,et al.  A comprehensive survey on vehicular Ad Hoc network , 2014, J. Netw. Comput. Appl..

[34]  Lionel C. Briand,et al.  A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering , 2014, Softw. Test. Verification Reliab..

[35]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[36]  Mark Harman,et al.  Automated software transplantation , 2015, ISSTA.

[37]  Fan Wu,et al.  Mutation-aware fault prediction , 2016, ISSTA.

[38]  Margaret-Anne D. Storey,et al.  Disrupting developer productivity one bot at a time , 2016, SIGSOFT FSE.

[39]  Luke Church,et al.  Modern Code Review: A Case Study at Google , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[40]  Simon Urli,et al.  How to Design a Program Repair Bot? Insights from the Repairnator Project , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[41]  Miryung Kim,et al.  Automated Transplantation and Differential Testing for Clones , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[42]  David Lopez-Paz,et al.  Revisiting Classifier Two-Sample Tests , 2016, ICLR.

[43]  John Micco,et al.  Taming Google-Scale Continuous Testing , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[44]  Ron Kohavi,et al.  Online Controlled Experiments and A/B Testing , 2017, Encyclopedia of Machine Learning and Data Mining.

[45]  Peter W. O'Hearn,et al.  From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis , 2018, 2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[46]  Mark Harman,et al.  Genetic Improvement of Software: A Comprehensive Survey , 2018, IEEE Transactions on Evolutionary Computation.

[47]  Zoubin Ghahramani,et al.  The Automatic Statistician , 2019, Automated Machine Learning.

[48]  Mark Harman,et al.  SapFix: Automated End-to-End Repair at Scale , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[49]  Ralf Lämmel,et al.  WES: Agent-based User Interaction Simulation on Real Infrastructure , 2020, ICSE.

[50]  D. Adam.  Special report: The simulations driving the world’s response to COVID-19 , 2020, Nature.

[51]  Qingtang Jiang,et al.  Two‐sample test based on classification probability , 2019, Stat. Anal. Data Min..

[52]  Mark Harman,et al.  Testing Web Enabled Simulation at Scale Using Metamorphic Testing , 2021, 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[53]  Larry A. Wasserman,et al.  Classification Accuracy as a Proxy for Two Sample Testing , 2016, The Annals of Statistics.

[54]  E. Meijer,et al.  Facebook’s Cyber–Cyber and Cyber–Physical Digital Twins , 2021, EASE.

[55]  Xing Qiu,et al.  Hypothesis Testing for Two Sample Comparison of Network Data , 2021, 2106.13931.