Automatically Assessing and Extending Code Coverage for NPM Packages

Typical Node.js applications extensively rely on packages hosted in the npm registry. As such packages may be used by thousands of other packages or applications, it is important to assess their code coverage. Moreover, increasing code coverage may help detect previously unknown issues. In this paper, we introduce TESA, a new tool that automatically assembles a test suite for any package in the npm registry. The test suite includes 1) tests written for the target package and usually hosted in its development repository, and 2) tests selected from dependent packages. The former tests allow assessing the code coverage of the target package, while the latter ones can increase code coverage by exploiting third-party tests that also exercise code in the target package. We use TESA to assess the code coverage of 500 popular npm packages. Then, we demonstrate that TESA can significantly increase code coverage by including tests from dependent packages. Finally, we show that the test suites assembled by TESA increase the effectiveness of existing dynamic program analyses to identify performance issues that are not detectable when only executing the developer’s tests.

[1]  Philippe Suter,et al.  A Look at the Dynamics of the JavaScript Package Ecosystem , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[2]  Haiyang Sun,et al.  Reasoning about the Node.js Event Loop using Async Graphs , 2019, 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO).

[3]  Anders Møller,et al.  Systematic approaches for increasing soundness and precision of static analyzers , 2017, SOAP@PLDI.

[4]  Frank Tip,et al.  Finding broken promises in asynchronous JavaScript programs , 2018, Proc. ACM Program. Lang..

[5]  Markus Zimmermann,et al.  Small World with High Risks: A Study of Security Threats in the npm Ecosystem , 2019, USENIX Security Symposium.

[6]  Arie van Deursen,et al.  Software Ecosystem Call Graph for Dependency Management , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER).

[7]  Hitesh Sajnani,et al.  A Study on the Lifecycle of Flaky Tests , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[8]  Chris F. Kemerer,et al.  Cyclomatic Complexity Density and Software Maintenance Productivity , 1991, IEEE Trans. Software Eng..

[9]  Ali Mesbah,et al.  Efficient JavaScript Mutation Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[10]  Mario Linares Vásquez,et al.  Mutode: generic JavaScript and Node.js mutation testing tool , 2018, ISSTA.

[11]  Pedro de Alcântara dos Santos Neto,et al.  A machine learning approach to generate test oracles , 2018, SBES.

[12]  Andres Ojamaa,et al.  Assessing the security of Node.js platform , 2012, 2012 International Conference for Internet Technology and Secured Transactions.

[13]  Kyriakos C. Chatzidimitriou,et al.  npm-Miner: An Infrastructure for Measuring the Quality of the npm Registry , 2018, 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR).

[14]  Manu Sridharan,et al.  MemInsight: platform-independent memory debugging for JavaScript , 2015, ESEC/SIGSOFT FSE.

[15]  Koushik Sen,et al.  A Survey of Dynamic Analysis and Test Generation for JavaScript , 2017, ACM Comput. Surv..

[16]  Darko Marinov,et al.  DeFlaker: Automatically Detecting Flaky Tests , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[17]  Lin Chen,et al.  Impact Analysis of Cross-Project Bugs on Software Ecosystems , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[18]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[19]  Haiyang Sun,et al.  AutoBench: Finding Workloads That You Need Using Pluggable Hybrid Analyses , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[20]  Michael Pradel,et al.  Extracting Taint Specifications for JavaScript Libraries , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[21]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[22]  Richard McNally,et al.  Fuzzing: The State of the Art , 2012 .

[23]  Amin Milani Fard,et al.  JavaScript: The (Un)Covered Parts , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[24]  Darko Marinov,et al.  An empirical analysis of flaky tests , 2014, SIGSOFT FSE.

[25]  Frank Tip,et al.  A framework for automated testing of javascript web applications , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[26]  Haiyang Sun,et al.  Efficient dynamic analysis for Node.js , 2018, CC.

[27]  Ying Wang,et al.  Watchman: Monitoring Dependency Conflicts for Python Library Ecosystem , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[28]  Gordon Fraser,et al.  Code coverage at Google , 2019, ESEC/SIGSOFT FSE.

[29]  Xiaoyin Wang,et al.  Taming Behavioral Backward Incompatibilities via Cross-Project Testing and Analysis , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[30]  Gabriele Bavota,et al.  Why Developers Refactor Source Code: A Mining-based Study , 2021, ArXiv.

[31]  E. L. Lawler,et al.  Branch-and-Bound Methods: A Survey , 1966, Oper. Res..

[32]  Ali Mesbah,et al.  Generating Fixtures for JavaScript Unit Testing , 2015 .

[33]  Koushik Sen,et al.  Jalangi: a selective record-replay and dynamic analysis framework for JavaScript , 2013, ESEC/FSE 2013.

[34]  Steve Vinoski,et al.  Node.js: Using JavaScript to Build High-Performance Network Programs , 2010, IEEE Internet Comput..

[35]  Amin Milani Fard,et al.  Leveraging existing tests in automated test generation for web applications , 2014, ASE.

[36]  Yves Le Traon,et al.  Chapter Six - Mutation Testing Advances: An Analysis and Survey , 2019, Adv. Comput..

[37]  Esben Andreasen,et al.  Determinacy in static analysis for jQuery , 2014, OOPSLA 2014.

[38]  Arie van Deursen,et al.  Crawling Ajax-Based Web Applications through Dynamic Analysis of User Interface State Changes , 2012, TWEB.

[39]  Ali Mesbah,et al.  PYTHIA: Generating test cases with oracles for JavaScript applications , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[40]  Christian Kästner,et al.  Adding Sparkle to Social Coding: An Empirical Study of Repository Badges in the npm Ecosystem , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion).

[41]  Koushik Sen,et al.  JITProf: pinpointing JIT-unfriendly JavaScript code , 2015, ESEC/SIGSOFT FSE.

[42]  Haiyang Sun,et al.  Automated Large-Scale Multi-Language Dynamic Program Analysis in the Wild (Tool Insights Paper) , 2019, ECOOP.

[43]  Koushik Sen,et al.  FuzzFactory: domain-specific fuzzing with waypoints , 2019, Proc. ACM Program. Lang..