The In-Training Examination in Internal Medicine

The In-Training Examination in Internal Medicine (ITE-IM) is a written examination that was developed to enable medical housestaff and residency programs to compare themselves with their national peer groups. Since 1988, the examination has been administered annually to internal medicine residents on a voluntary basis. More than 90% of medicine training programs in the United States have participated each year. We review the first 6 years of the ITE-IM, including its preparation, content, administration, and results. We also provide insights into the changing profile of internal medicine residents and training programs from 1988 to 1993. The Examination Background In the 1970s and 1980s, several medical professional organizations, including the American College of Surgeons, the American Board of Family Practice, the American Board of Anesthesiology and the American Society of Anesthesiologists; subspecialty surgical societies; and a coalition of internal medicine residency programs in the Midwest and Northeast, developed written examinations to test the knowledge base of their residents-in-training [1-9]. These initiatives provided momentum to develop a standardized, national in-training examination for internal medicine residents. In the mid-1980s, the American College of Physicians, the Association of Program Directors in Internal Medicine, and the Association of Professors of Medicine collaborated to prepare and administer an annual voluntary, written examination for internal medicine residency programs in the United States. The ITE-IM was targeted to residents in their second year of postgraduate training, the midpoint of their clinical training, although residents in their first and third years of postgraduate training were also encouraged to take the examination. The examination was intended to be a primarily educational instrument for self-evaluation [10]. It was specifically noted that the examination was not to be used as a pretest for determining eligibility for certifying examinations, as a substitute for clinical competency examinations, for promotion or termination within residency programs, or by any outside regulatory agency to assess the knowledge of a particular resident or the quality of a particular training program. Preparation Each edition of the ITE-IM is prepared by a committee composed of 10 authors representing the American College of Physicians, the Association of Program Directors in Internal Medicine, and the Association of Professors of Medicine. Questions are written by committee members according to a test blueprint that defines the major content of the examination and the proportion of questions that will be included for each organ system and related disciplines. To ensure reliability, between 25% and 50% of questions in each examination are items that were used in previous examinations. A pilot examination containing only new questions is critiqued by a group of chief medical residents and program directors before each examination. Content The ITE-IM is intended to test the knowledge base of residents in their second year of postgraduate training in general medicine. The subject matter of questions is selected to reflect the experience of residents at this level of training. The test also includes items to assess the resident's understanding of the clinical examination and practical applications of basic medical concepts. In recent examinations, questions that relate to ambulatory care and case vignettes that require clinical decision making have been emphasized. Each question is characterized by its setting (ambulatory, inpatient, or critical care), content (physical examination skills, diagnosis, or treatment), and process (judgment, synthesis, or recall). Each year, the examination consists of between 375 and 450 items distributed among the organ systems and related areas according to the blueprint of that year. The entire examination consists of test items that are relevant to the practice of internal medicine. Subjects for questions include topics in primary care, critical care, neurology, psychiatry, dermatology, geriatrics, preventive medicine, medical ethics, and epidemiology, as well as the medical specialties. Examination scores are grouped by organ system to facilitate analysis and provide targeted feedback to residents and program directors. Approximately 20% of questions test core knowledge in internal medicine and related disciplines; the percentage of questions allotted to each organ system ranges from 7% to 15%, with most between 9% and 10%. Administration The ITE-IM is administered each January at internal medicine training program sites in the United States, Puerto Rico, and Canada. All participating programs offer the examination to residents in their second year of postgraduate training; many programs also make it available to residents in their first and third years. Residents are not encouraged to study for the examination. If the examination is not given to all housestaff at a site on a single day, the residents in their second year of training are scheduled on the first day to maintain the security of the testing material for this cohort. Program directors are responsible for the administration and security of the examination at each site. Scores Individual examinations are scored and analyzed statistically by the National Board of Medical Examiners. Before final scoring, test items that have poor statistical performance are reviewed by staff from the Board and the American College of Physicians, as well as the committee chairperson. In general, items that fewer than 30% of residents answered correctly or items that failed to discriminate high-scoring from low-scoring examinees are identified for review; if found defective in content or structure, these questions are excluded from the final scoring and analysis. The results of the January examination are mailed to program directors for distribution to residents in late March or early April. Each resident receives a report that includes his or her total score of the percentage of correct responses and the distribution of scores for the nationwide cohort of persons taking the test from that peer group. Each resident also receives a list of question numbers that were answered incorrectly and their educational objectives; this information is organized by organ system and enables the resident to identify specific topic areas that need review. Program directors receive copies of the individual reports that are provided to their residents, including the list of items that were answered incorrectly and the educational objectives. The program directors also receive a composite of the average scores of the residents in their program by year of training, accompanied by scores from the national cohorts of residents taking the examination in each level of postgraduate training. The average scores of each group in the program are further subdivided by organ system area for comparison with scores of the national peer group. Statistical Analysis Analysis of resident performance on the ITE-IM from each year has shown the total test scores to be reliable. The Kuder-Richardson reliability coefficient is used to measure the internal consistency of the test and to provide an estimate of the accuracy of scores [11]. The reliability coefficient is a statistic that can range from 0.0 to 1.0. A high value indicates that the scores of individual questions correlate highly with one another; a low value indicates that questions are heterogeneous or nonconsistent. The reliability of the examination increases with the total number of test questions, as well as with the homogeneity and quality of individual test items. The reliability of the test is also increased by the inclusion of questions that have done well in previous years. Examinations with reliability coefficients of 0.70 and higher are considered precise enough to provide reliable educational feedback to persons taking the test. The reliability coefficient of the examination has been consistently greater than 0.91 for residents in their second year of postgraduate training each year; in 1992 and 1993, it was 0.95. This means that 95% of the variance in scores among test-takers is caused by true differences in proficiency. The consistently high reliability coefficient for the examination is similar to that of examinations used for the purposes of licensure or certification, including the examinations of the National Board of Medical Examiners or the United States Medical Licensing Examiners and the certifying examination of the American Board of Internal Medicine. The reliability coefficients for the individual organ system areas or medical specialty scores are high, although not as high as that for the total examination score. Lower reliability coefficients can be expected because of the smaller number of test items in each of the component areas. Reliability coefficients for organ system scores and the total score for the 1993 examination are provided in Table 1, which shows how the reliability increases as the number of test items increases. Scores of the average percentage of correct responses and average item discrimination indices are also listed in Table 1. The discrimination index, or point-biserial correlation coefficient, is an indication of how well, on average, individual questions distinguish examinees with high total scores from those with low total scores [12]. Values greater than 0.10 are considered acceptable by the In-Training Examination Committee. All subtests in the 1993 examination have average values between 0.20 and 0.25. Table 1. 1993 Internal Medicine In-Training Examination Summary Statistics for Second-Year Residents Results Program Participation The number of housestaff at each level of training who have taken the ITE-IM during the last 6 years has increased steadily (Table 2). In 1993, more than 11 000 of the approximately 18 000 residents in t