Repairing Decision-Making Programs Under Uncertainty

The world is uncertain. Programs can be wrong. We address the problem of repairing a program under uncertainty, where program inputs are drawn from a probability distribution. The goal of the repair is to construct a new program that satisfies a probabilistic Boolean expression. Our work focuses on loop-free decision-making programs, e.g., classifiers, that return a Boolean- or finite-valued result. Specifically, we propose distribution-guided inductive synthesis, a novel program repair technique that iteratively (i) samples a finite set of inputs from a probability distribution defining the precondition, (ii) synthesizes a minimal repair to the program over the sampled inputs using an smt-based encoding, and (iii) verifies that the resulting program is correct and is semantically close to the original program. We formalize our algorithm and prove its correctness by rooting it in computational learning theory. For evaluation, we focus on repairing machine learning classifiers with the goal of making them unbiased (fair). Our implementation and evaluation demonstrate our approach’s ability to repair a range of programs.

[1]  Abhik Roychoudhury,et al.  DirectFix: Looking for Simple Program Repairs , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[2]  Yair Zick,et al.  Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[3]  Christos H. Papadimitriou,et al.  Games against nature , 1983, 24th Annual Symposium on Foundations of Computer Science (sfcs 1983).

[4]  Michael L. Littman,et al.  The Computational Complexity of Probabilistic Planning , 1998, J. Artif. Intell. Res..

[5]  Christian von Essen,et al.  Program Repair without Regret , 2013, CAV.

[6]  Roderick Bloem,et al.  Program Repair as a Game , 2005, CAV.

[7]  Alexander Aiken,et al.  Bias-variance tradeoffs in program analysis , 2014, POPL.

[8]  Yair Zick,et al.  Algorithmic Transparency via Quantitative Input Influence , 2017 .

[9]  Armando Solar-Lezama,et al.  Program synthesis by sketching , 2008 .

[10]  Toniann Pitassi,et al.  Fairness through awareness , 2011, ITCS '12.

[11]  Swarat Chaudhuri,et al.  Bridging boolean and quantitative synthesis using smoothed proof search , 2014, POPL.

[12]  Rishabh Singh,et al.  Qlose: Program Repair with Quantitative Objectives , 2016, CAV.

[13]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[14]  Nathan Srebro,et al.  Equality of Opportunity in Supervised Learning , 2016, NIPS.

[15]  Carlos Eduardo Scheidegger,et al.  Certifying and Removing Disparate Impact , 2014, KDD.

[16]  Lijun Zhang,et al.  Model Repair for Markov Decision Processes , 2013, 2013 International Symposium on Theoretical Aspects of Software Engineering.

[17]  C. R. Ramakrishnan,et al.  Model Repair for Probabilistic Systems , 2011, TACAS.

[18]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[19]  Suresh Venkatasubramanian,et al.  On the (im)possibility of fairness , 2016, ArXiv.

[20]  Roderick Bloem,et al.  Automated error localization and correction for imperative programs , 2011, 2011 Formal Methods in Computer-Aided Design (FMCAD).

[21]  Timon Gehr,et al.  PSI: Exact Symbolic Inference for Probabilistic Programs , 2016, CAV.

[22]  Aws Albarghouthi,et al.  Fairness as a Program Property , 2016, ArXiv.