DRAW: A Challenging and Diverse Algebra Word Problem Set

We present DRAW, a dataset consisting of 1000 algebra word problems, semiautomatically annotated for the evaluation of automatic solvers.1 Details of the annotation process are described, which involves a novel template reconciliation procedure for reducing equivalent templates. DRAW also consists of richer annotations, including gold coefficient alignments and equation system templates, which were absent in existing benchmarks. We present a quantitative comparison of DRAW to existing benchmarks, showing that DRAW consists a wide variety of problems, both in terms of narrative diversity and problem types. We provide a strong baseline for DRAW using a simple yet powerful solver. We also experimentally verify that the additional annotations indeed improves the performance for our automatic solver.