Predicting Question Effectiveness

We develop a method for predicting the quality of the passage retrieval component in question answering systems. Since high-quality passages form the basis for accurate answer extraction, our method naturally extends to prediction of an entire system’s effectiveness at extracting a correct answer for a given question. Such prediction of question performance may lead to ways of guiding users in improving questions unlikely to succeed. Our metric is also a necessary research step towards systems that automatically tailor their methods to suit each individual question. Building on previous work on predicting the performance of queries in retrieving documents, we show how to compute the clarity score for questions using passage-based collections. We show that this score is correlated with the average precision in a TREC-9 based system, breakdown the correlation by question type, and discuss example questions. We also study a more general set of questions extracted from a Web log to help make the case for the general usefulness of performance prediction based on question clarity score.