C2S: translating natural language comments to formal program specifications

Formal program specifications are essential for various software engineering tasks, such as program verification, program synthesis, code debugging and software testing. However, manually inferring formal program specifications is not only time-consuming but also error-prone. In addition, it requires substantial expertise. Natural language comments contain rich semantics about behaviors of code, making it feasible to infer program specifications from comments. Inspired by this, we develop a tool, named C2S, to automate the specification synthesis task by translating natural language comments into formal program specifications. Our approach firstly constructs alignments between natural language word and specification tokens from existing comments and their corresponding specifications. Then for a given method comment, our approach assembles tokens that are associated with words in the comment from the alignments into specifications guided by specification syntax and the context of the target method. Our tool successfully synthesizes 1,145 specifications for 511 methods of 64 classes in 5 different projects, substantially outperforming the state-of-the-art. The generated specifications are also used to improve a number of software engineering tasks like static taint analysis, which demonstrates the high quality of the specifications.

[1]  Alessandra Gorla,et al.  Translating code comments to procedure specifications , 2018, ISSTA.

[2]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[3]  Juan Zhai,et al.  Assertion-Directed Precondition Synthesis for Loops over Data Structures , 2015, SETTA.

[4]  Yuanyuan Zhou,et al.  /*icomment: bugs or bad comments?*/ , 2007, SOSP.

[5]  Bin Li,et al.  Precondition Calculation for Loops Iterating over Data Structures , 2016, 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[6]  Jacques Klein,et al.  FlowDroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps , 2014, PLDI.

[7]  Hridesh Rajan,et al.  Statistical Learning for Inference between Implementations and Documentation , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER).

[8]  Hridesh Rajan,et al.  Mining preconditions of APIs in large-scale code corpus , 2014, FSE 2014.

[9]  Ruzica Piskac,et al.  Complete functional synthesis , 2010, PLDI '10.

[10]  Tao Xie,et al.  PreInfer: Automatic Inference of Preconditions via Symbolic Analysis , 2018, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[11]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[12]  Alessandra Gorla,et al.  Automatic generation of oracles for exceptional behaviors , 2016, ISSTA.

[13]  Yuanyuan Zhou,et al.  aComment: mining annotations from comments and code to detect interrupt related concurrency bugs , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[14]  Douglas Kramer,et al.  API documentation from source code comments: a case study of Javadoc , 1999, SIGDOC '99.

[15]  Daniel Kroening,et al.  Counterexample-Guided Precondition Inference , 2013, ESOP.

[16]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[17]  Suresh Jagannathan,et al.  Static specification inference using predicate mining , 2007, PLDI '07.

[18]  Tao Xie,et al.  Inferring Resource Specifications from Natural Language API Documentation , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[19]  Yuanyuan Zhou,et al.  Listening to programmers — Taxonomies and characteristics of comments in operating system code , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[20]  Jingyi Su,et al.  Poster: Using Consensus to Automatically Infer Post-conditions , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion).

[21]  Martin P. Robillard,et al.  Patterns of Knowledge in API Reference Documentation , 2013, IEEE Transactions on Software Engineering.

[22]  Patrick Cousot,et al.  Automatic Inference of Necessary Preconditions , 2013, VMCAI.

[23]  Xiangyu Zhang,et al.  Automatic Model Generation from Documentation for Java API Functions , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[24]  Juan Zhai,et al.  Post-condition-Directed Invariant Inference for Loops over Data Structures , 2014, 2014 IEEE Eighth International Conference on Software Security and Reliability-Companion.

[25]  Lin Tan.  Leveraging code comments to improve software reliability , 2009 .

[26]  Alexander Aiken,et al.  Specification Inference Using Context-Free Language Reachability , 2015, POPL.

[27]  Andrew W. Appel,et al.  Modern Compiler Implementation in Java , 1997 .

[28]  Bernardete Ribeiro,et al.  The importance of stop word removal on recall values in text categorization , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[29]  Ranjit Jhala,et al.  Refinement types for Haskell , 2014, ICFP.

[30]  AikenAlex,et al.  Specification Inference Using Context-Free Language Reachability , 2015 .

[31]  Linzhang Wang,et al.  Automated Generation of LTL Specifications For Smart Home IoT Using Natural Language , 2020, 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[32]  Xiangyu Zhang,et al.  Detecting sensitive data disclosure via bi-directional text correlation analysis , 2016, SIGSOFT FSE.

[33]  Weifeng Zhang,et al.  CPC: Automatically Classifying and Propagating Natural Language Comments via Program Analysis , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[34]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[35]  Manuel Costa,et al.  Bouncer: securing software by blocking bad input , 2008, WRAITS '08.

[36]  Gary T. Leavens,et al.  @tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[37]  Michael D. Ernst,et al.  Automatic generation of program specifications , 2002, ISSTA '02.

[38]  Sumit Gulwani,et al.  From program verification to program synthesis , 2010, POPL '10.

[39]  Rajeev Alur,et al.  Syntax-guided synthesis , 2013, 2013 Formal Methods in Computer-Aided Design.

[40]  Bertrand Meyer,et al.  Contract driven development = test driven development - writing test cases , 2007, ESEC-FSE '07.

[41]  Bertrand Meyer,et al.  Automatic Testing of Object-Oriented Software , 2007, SOFSEM.

[42]  Nikolai Tillmann,et al.  DySy: dynamic symbolic execution for invariant inference , 2008, ICSE.

[43]  Yu Zhou,et al.  Analyzing APIs Documentation and Code to Detect Directive Defects , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[44]  D ErnstMichael,et al.  Automatic generation of program specifications , 2002 .

[45]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[46]  Yuriy Brun,et al.  Automatically Generating Precise Oracles from Structured Natural Language Specifications , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[47]  Tao Xie,et al.  Inferring method specifications from natural language API descriptions , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[48]  JhalaRanjit,et al.  Refinement types for Haskell , 2014 .