The Association Between Confidence and Accuracy Among Users of a Mobile Web Platform for Medical Education

Background: Recent literature suggests that physicians' diagnostic confidence tends to exceed accuracy, with confidence being relatively inflexible regardless of case difficulty (1). Overconfidence could lead to physicians not asking for help when they need it. Underconfidence may expose patients to defensive medicine. Those who lack confidence may misrepresent their actual level of knowledge, resulting in missed opportunities for professional development (2). Research suggests that female medical students perform as well as their male peers but report less confidence in their abilities and are actually perceived to be less confident (3). The future of medical education may involve training students to better align confidence and accuracy. One strategy may include providing students with immediate, actionable performance feedback through technology-enabled education platforms (4). Osmosis is a mobile Web platform designed to help medical students learn by answering questions related to their curricula. Its database includes information from more than 14000 users. We developed a user-facing Calibration Index to show students their propensity for under- or overconfidence when answering questions. Objective: To measure medical students' confidence in their answers, compare confidence with accuracy, and examine differences between men and women. Methods: To answer a question in the platform, a user must first select 1 of 3 confidence ratings: I'm sure, Feeling lucky, or No clue. We analyzed the responses of 1021 users who answered more than 50 random multiple-choice and true-or-false questions. An automated software package (SexMachine, version 0.1.1, based on Gender.c, version 1.2) that evaluates data on how frequently a name correlates with male or female gender in the United States was used to determine a user's sex from his or her first name. Users with ambiguous namesex pairings were omitted. The final list was checked to confirm reasonable assignment. Users were assumed to be predominantly medical students in their preclinical years because the platform is designed for and marketed to this audience. Results are presented as the means of user-level response averages and compared with MannWhitney U tests using the Bonferroni adjustment (Stata, version 12.0, StataCorp). Findings: The study included 617 men and 404 women. Women's answers were more accurate than men's (61.4% vs. 60.3% correct; P= 0.040); however, they were less confident than men. For example, women selected I'm sure significantly less often than men (39.5% vs. 44.4% of responses; P= 0.001) (Table). Average accuracy for questions rated as I'm sure and Feeling lucky was higher in women (80.5% vs. 78.3% of I'm sure responses were correct [P= 0.002]; 53.5% vs. 49.8% of Feeling lucky responses were correct [P= 0.001]). The average number of questions answered by female and male users did not differ (647 vs. 579; P= 0.40). When results were stratified into quartiles based on the number of questions answered, differences by sex remained. Table. Correlation Between Confidence and Accuracy in Men and Women Discussion: The data suggest that, despite performing at the same level or higher, women lacked confidence relative to men. To understand the meaning and magnitude of this observed effect, it will be important to relate question-answering behavior to actual behavior on the wards. One limitation is that sex was determined from first name. Also, we could not verify that users were medical students or stratify results according to school and academic year. Notwithstanding these limitations, the data are a reminder that less confidence might not indicate a lack of knowledge and that confidence should not be mistaken for correctness. Insights gained from understanding the relationship between confidence and accuracy in medical trainees may ultimately prove valuable in reducing diagnostic errors caused by overconfidence and sex disparities caused by lack of confidence. Going forward, new educational interventions like the Calibration Index may help future physicians align confidence with accuracy to improve patient care and promote career advancement (5).