SoftMon: A Tool to Compare Similar Open-source Software from a Performance Perspective

Over the past two decades, a rich ecosystem of open-source software has evolved. For every type of application, there are a wide variety of alternatives. We observed that even if different applications that perform similar tasks and compiled with the same versions of the compiler and the libraries, they perform very differently while running on the same system. Sadly prior work in this area that compares two code bases for similarities does not help us in finding the reasons for the differences in performance. In this paper, we develop a tool, SoftMon, that can compare the codebases of two separate applications and pinpoint the exact set of functions that are disproportionately responsible for differences in performance. Our tool uses machine learning and NLP techniques to analyze why a given open-source application has a lower performance as compared to its peers, design bespoke applications that can incorporate specific innovations (identified by SoftMon) in competing applications, and diagnose performance bugs. In this paper, we compare a wide variety of large open-source programs such as image editors, audio players, text editors, PDF readers, mail clients and even full-fledged operating systems (OSs). In all cases, our tool was able to pinpoint a set of at the most 10-15 functions that are responsible for the differences within 200 seconds. A subsequent manual analysis assisted by our graph visualization engine helps us find the reasons. We were able to validate most of the reasons by correlating them with subsequent observations made by developers or from existing technical literature. The manual phase of our analysis is limited to 30 minutes (tested with human subjects).

[1]  Juanru Li,et al.  Cross-Architecture Binary Semantics Understanding via Similar Code Comparison , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[2]  David Brumley,et al.  Blanket Execution: Dynamic Similarity Testing for Program Binaries and Components , 2014, USENIX Security Symposium.

[3]  Camil Demetrescu,et al.  Input-Sensitive Profiling , 2012, IEEE Transactions on Software Engineering.

[4]  Prathmesh Kallurkar,et al.  pTask: A smart prefetching scheme for OS intensive applications , 2016, 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[5]  David Brumley,et al.  BYTEWEIGHT: Learning to Recognize Functions in Binary Code , 2014, USENIX Security Symposium.

[6]  Chanchal Kumar Roy,et al.  Towards a Big Data Curated Benchmark of Inter-project Code Clones , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[7]  Alexander Aiken,et al.  Stochastic superoptimization , 2012, ASPLOS '13.

[8]  Xiangyu Zhang,et al.  Matching execution histories of program versions , 2005, ESEC/FSE-13.

[9]  Alvin Cheung,et al.  Summarizing Source Code using a Neural Attention Model , 2016, ACL.

[10]  Emden R. Gansner,et al.  An open graph visualization system and its applications to software engineering , 2000, Softw. Pract. Exp..

[11]  Uri Alon,et al.  code2vec: learning distributed representations of code , 2018, Proc. ACM Program. Lang..

[12]  Mireille Régnier,et al.  Knuth-Morris-Pratt Algorithm: An Analysis , 1989, MFCS.

[13]  Michael D. Smith,et al.  The measured performance of personal computer operating systems , 1995, SOSP.

[14]  Mark Johnson,et al.  An Improved Non-monotonic Transition System for Dependency Parsing , 2015, EMNLP.

[15]  Mary Baker,et al.  A Performance Comparison of UNIX Operating Systems on the Pentium , 1996, USENIX Annual Technical Conference.

[16]  Dongsoo Han,et al.  Microbenchmarks of operating system performance on the multiprocessor platform , 1999, Proceedings of IEEE. IEEE Region 10 Conference. TENCON 99. 'Multimedia Technology for Asia-Pacific Information Infrastructure' (Cat. No.99CH37030).

[17]  Khaled Yakdan,et al.  discovRE: Efficient Cross-Architecture Identification of Bugs in Binary Code , 2016, NDSS.

[18]  Dhriti Khanna,et al.  POLLUX: safely upgrading dependent application libraries , 2016, SIGSOFT FSE.

[19]  Uri Alon,et al.  A general path-based representation for predicting program properties , 2018, PLDI.

[20]  Vipin Kumar,et al.  Chameleon: Hierarchical Clustering Using Dynamic Modeling , 1999, Computer.

[21]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[22]  Susan L. Graham,et al.  Gprof: A call graph execution profiler , 1982, SIGPLAN '82.

[23]  Michael Stumm,et al.  FlexSC: Flexible System Call Scheduling with Exception-Less System Calls , 2010, OSDI.

[24]  Debin Gao,et al.  BinHunt: Automatically Finding Semantic Differences in Binary Programs , 2008, ICICS.

[25]  Halvar Flake,et al.  Structural Comparison of Executable Objects , 2004, DIMVA.

[26]  Babak Falsafi,et al.  Confluence: Unified instruction supply for scale-out servers , 2015, 2015 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[27]  Anastasia Ailamaki,et al.  SLICC: Self-Assembly of Instruction Cache Collectives for OLTP Workloads , 2012, 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture.

[28]  Josef Weidendorfer,et al.  Argument Controlled Profiling , 2009, Euro-Par Workshops.

[29]  LuShan,et al.  Understanding and detecting real-world performance bugs , 2012 .

[30]  Yang Liu,et al.  Accurate and Scalable Cross-Architecture Cross-OS Binary Code Search with Emulation , 2019, IEEE Transactions on Software Engineering.

[31]  Robert S. Boyer,et al.  A fast string searching algorithm , 1977, CACM.

[32]  Shan Lu,et al.  Statistical debugging for real-world performance problems , 2014, OOPSLA.

[33]  Arnaldo Carvalho de Melo,et al.  The New Linux ’ perf ’ Tools , 2010 .

[34]  Prathmesh Kallurkar,et al.  SchedTask: A Hardware-Assisted Task Scheduler , 2017, 2017 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[35]  Yuanyuan Zhou,et al.  CP-Miner: finding copy-paste and related bugs in large-scale software code , 2006, IEEE Transactions on Software Engineering.

[36]  Y. Charlie Hu,et al.  Differential Energy Profiling: Energy Optimization via Diffing Similar Apps , 2018, OSDI.

[37]  Yan Cui,et al.  Experience on Comparison of Operating Systems Scalability on the Multi-core Architecture , 2011, 2011 IEEE International Conference on Cluster Computing.

[38]  Yang Liu,et al.  BinGo: cross-architecture cross-OS binary search , 2016, SIGSOFT FSE.

[39]  Jitendra Bajpai,et al.  Graph isomorphisms in quasi-polynomial time , 2017, 1710.04574.

[40]  Emery D. Berger,et al.  Coz: finding code that counts with causal profiling , 2015, USENIX Annual Technical Conference.

[41]  Barbara G. Ryder,et al.  A scalable technique for characterizing the usage of temporaries in framework-intensive Java applications , 2008, SIGSOFT '08/FSE-16.

[42]  Charles A. Sutton,et al.  A Convolutional Attention Network for Extreme Summarization of Source Code , 2016, ICML.

[43]  Zhaofang Wen,et al.  Automatic Algorithm Recognition and Replacement: A New Approach to Program Optimization , 2000 .

[44]  R TallentNathan,et al.  Effective performance measurement and analysis of multithreaded applications , 2009 .

[45]  Johan A. Pouwelse,et al.  Understanding software performance regressions using differential flame graphs , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[46]  Christian Rossow,et al.  Cross-Architecture Bug Search in Binary Executables , 2015, 2015 IEEE Symposium on Security and Privacy.

[47]  P. Jaccard.  THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1 , 1912 .

[48]  Denis Barthou,et al.  Algorithm recognition based on demand-driven data-.ow analysis , 2003, 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings..

[49]  Marco Cesati,et al.  Understanding the Linux Kernel - from I / O ports to process management: covers Linux Kernel version 2.4 (2. ed.) , 2005 .

[50]  Diomidis Spinellis.  A tale of four kernels , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[51]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.

[52]  Nathan R. Tallent,et al.  Effective performance measurement and analysis of multithreaded applications , 2009, PPoPP '09.

[53]  Babak Falsafi,et al.  Proactive instruction fetch , 2011, 2011 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[54]  Anastasia Ailamaki,et al.  STREX: boosting instruction cache reuse in OLTP workloads through stratified transaction execution , 2013, ISCA.

[55]  Balaji Vasan Srinivasan,et al.  Environment Specific Content Rendering & Transformation , 2016, IUI Companion.

[56]  Sencun Zhu,et al.  Semantics-based obfuscation-resilient binary code similarity comparison with applications to software plagiarism detection , 2014, SIGSOFT FSE.

[57]  Dongmei Zhang,et al.  Context-sensitive delta inference for identifying workload-dependent performance bottlenecks , 2013, ISSTA.