Deep Data Flow Analysis

Compiler architects increasingly look to machine learning when building heuristics for compiler optimization. The promise of automatic heuristic design, freeing the compiler engineer from the complex interactions of program, architecture, and other optimizations, is alluring. However, most machine learning methods cannot replicate even the simplest of the abstract interpretations of data flow analysis that are critical to making good optimization decisions. This must change for machine learning to become the dominant technology in compiler heuristics. To this end, we propose ProGraML - Program Graphs for Machine Learning - a language-independent, portable representation of whole-program semantics for deep learning. To benchmark current and future learning techniques for compiler analyses we introduce an open dataset of 461k Intermediate Representation (IR) files for LLVM, covering five source programming languages, and 15.4M corresponding data flow results. We formulate data flow analysis as an MPNN and show that, using ProGraML, standard analyses can be learned, yielding improved performance on downstream compiler optimization tasks.

[1]  Jeffrey D. Ullman,et al.  Global Data Flow Analysis and Iterative Algorithms , 1976, J. ACM.

[2]  Jeronimo Castrillon,et al.  Compiler-based graph representations for deep learning models of code , 2020, CC.

[3]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[4]  Jeffrey D. Ullman,et al.  Monotone data flow analysis frameworks , 1977, Acta Informatica.

[5]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[6]  Christopher C. Cummins,et al.  Synthesizing benchmarks for predictive modeling , 2017, 2017 IEEE/ACM International Symposium on Code Generation and Optimization (CGO).

[7]  Chris Cummins,et al.  End-to-End Deep Learning of Optimization Heuristics , 2017, 2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT).

[8]  Gary A. Kildall,et al.  A unified approach to global program optimization , 1973, POPL.

[9]  Charles A. Sutton,et al.  Learning natural coding conventions , 2014, SIGSOFT FSE.

[10]  Andreas Krause,et al.  Predicting Program Properties from "Big Code" , 2015, POPL.

[11]  Tao Wang,et al.  Convolutional Neural Networks over Tree Structures for Programming Language Processing , 2014, AAAI.

[12]  Gianluca Palermo,et al.  A Survey on Compiler Autotuning using Machine Learning , 2018, ACM Comput. Surv..

[13]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[14]  David Pichardie,et al.  Validating Dominator Trees for a Fast, Verified Dominance Test , 2015, ITP.

[15]  David H. Bailey,et al.  The Nas Parallel Benchmarks , 1991, Int. J. High Perform. Comput. Appl..

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Uri Alon,et al.  code2vec: learning distributed representations of code , 2018, Proc. ACM Program. Lang..

[18]  Torsten Hoefler,et al.  Neural Code Comprehension: A Learnable Representation of Code Semantics , 2018, NeurIPS.

[19]  Marc Brockschmidt,et al.  Learning to Represent Programs with Graphs , 2017, ICLR.

[20]  Christopher D. Manning,et al.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks , 2015, ACL.

[21]  Maunendra Sankar Desarkar,et al.  IR2Vec: LLVM IR based Scalable Program Embeddings , 2019 .

[22]  Michael F. P. O'Boyle,et al.  Portable mapping of data parallel programs to OpenCL for heterogeneous systems , 2013, Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization (CGO).

[23]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[24]  Enrico Macii,et al.  Code Mapping in Heterogeneous Platforms Using Deep Learning and LLVM-IR , 2019, 2019 56th ACM/IEEE Design Automation Conference (DAC).

[25]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[26]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[27]  Graham Neubig,et al.  Learning to Represent Edits , 2018, ICLR.

[28]  Maunendra Sankar Desarkar,et al.  IR2Vec: A Flow Analysis based Scalable Infrastructure for Program Encodings , 2019, ArXiv.

[29]  Robert E. Tarjan,et al.  A fast algorithm for finding dominators in a flowgraph , 1979, TOPL.

[30]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[31]  Joe D. Warren,et al.  The program dependence graph and its use in optimization , 1987, TOPL.

[32]  Yann Dauphin,et al.  Convolutional Sequence to Sequence Learning , 2017, ICML.

[33]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[34]  Aditya Kanade,et al.  Learning and Evaluating Contextual Embedding of Source Code , 2019, ICML.

[35]  Martin T. Vechev,et al.  Learning a Static Analyzer from Data , 2016, CAV.

[36]  Wenwu Zhu,et al.  Deep Learning on Graphs: A Survey , 2018, IEEE Transactions on Knowledge and Data Engineering.

[37]  Torsten Hoefler,et al.  ProGraML: Graph-based Deep Learning for Program Optimization and Analysis , 2020, ArXiv.

[38]  Samy Bengio,et al.  Device Placement Optimization with Reinforcement Learning , 2017, ICML.

[39]  Zheng Wang,et al.  Machine Learning in Compiler Optimization , 2018, Proceedings of the IEEE.

[40]  Daniel Tarlow,et al.  Learning Execution through Neural Code Fusion , 2019, ICLR.

[41]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[42]  Michael Carbin,et al.  Compiler Auto-Vectorization with Imitation Learning , 2019, NeurIPS.

[43]  Premkumar T. Devanbu,et al.  A Survey of Machine Learning for Big Code and Naturalness , 2017, ACM Comput. Surv..

[44]  Uri Alon,et al.  A general path-based representation for predicting program properties , 2018, PLDI.

[45]  Pushmeet Kohli,et al.  Graph Matching Networks for Learning the Similarity of Graph Structured Objects , 2019, ICML.

[46]  Aditya K. Ghose,et al.  A deep tree-based model for software defect prediction , 2018, ArXiv.

[47]  Le Song,et al.  Learning Loop Invariants for Program Verification , 2018, NeurIPS.

[48]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[49]  Ke Wang,et al.  Blended, precise semantic program embeddings , 2020, PLDI.

[50]  Keith D. Cooper,et al.  Engineering a Compiler , 2003 .

[51]  Ken Kennedy,et al.  Iterative Data-flow Analysis , Revisited , 2003 .