Item Clusters and Computerized Adaptive Testing: A Case for Testlets

It is observed that many sorts of difficulties may preclude the uneventful construction of tests by a computerized algorithm, such as those currently in favor in Computerized Adaptive Testing (CAT). In this essay we discuss a number of these problems, as well as some possible avenues of solution. We conclude with the development of the "testlet," a bundle of items that can be arranged either hierarchically or linearly, thus maintaining the efficiency of an adaptive test while keeping the quality control of test construction that is possible currently only with careful expert scrutiny. Performance on the separate testlets is aggregated to yield ability estimates. In the old days of testing a wise examiner confronted a nervous examinee and asked questions. After getting some initial idea of the examinee's ability or knowledge level, the examiner would not waste time asking questions that were too difficult or too easy. Instead, he or she would focus questions on the area around the examinee's proficiency level. This made for a challenging and efficient test. It was also subjective and expensive. This state of affairs changed during the First World War with the Army Alpha. The mass processing of recruits into the armed services demonstrated that standardized tests could make the training and accessioning process more efficient. Substituting for the wise examiner was a broad-range test made up of many multiple-choice questions. This had several advantages, among which were its vastly reduced cost per unit and its objectivity-everyone took the same test, and it was scored in the same way for everyone. It also had some disadvantages. As a written document, a test form could be stolen, and because of its broad audience it had to contain items that would tax the least able of the prospective