Towards answering “Am I on the right track?” automatically using program synthesis

Students learning to program often need help completing assignments and understanding why their code does not work as they expect it to. One common place where they seek such help is at teaching assistant office hours. We found that teaching assistants in introductory programming (CS1) courses frequently answer some variant of the question ``Am I on the Right Track?''. The goal of this work is to develop an automated tool that provides similar feedback for students in real-time from within an IDE as they are writing their program. Existing automated tools lack the generality that we seek, often assuming a single approach to a problem, using hand-coded error models, or applying sample fixes from other students. In this paper, we explore the use of program synthesis to provide less constrained automated answers to ``Am I on the Right Track'' (AIORT) questions. We describe an observational study of TA-student interactions that supports targeting AIORT questions, as well as the development of and design considerations behind a prototype integrated development environment (IDE). The IDE uses an existing program synthesis engine to determine if a student is on the right track and we present pilot user studies of its use.