The present study reports the results of an evaluation of three types of multiple choice questions—the five choice completion, multiple completion, and assertion‐reason. Fifty‐four questions, eighteen of each type and measuring the candidate on the same scientific principle and classified as either factual or comprehension, were developed and included in the General Surgery certifying examination of the Royal College of Physicians and Surgeons of Canada. In addition to using descriptive statistics, the multitrait‐multi‐method technique was used to investigate whether the item types measured different aspects of examinee capabilities. Results indicated that performance on the five choice completion and the multiple completion type questions was roughly the same, whereas performance on the assertion‐reason type was lower. The results of the multitrait‐multimethod validation revealed that the three item types were unable to discriminate between the two traits of factual and comprehension.