Hawks and Doves: Adjusting for Bias in Residency Interview Scoring

Purpose: Individual interviews are an important part of the residency interview process. Programs may use these scores to calculate rankings used in the match process. Individual interviewers can introduce bias by consistently scoring candidates higher or lower than their peers. The order of interview or year of interview also has the potential to introduce bias. This study seeks to determine if interviewers or timing introduces bias into interview scores and to provide a method to adjust for this bias. Methods: Interview scores at a single general surgery residency program were obtained over 3 years. The mean interview score and standard error were calculated for each interviewer. The difference in average score between years and by order of interview was evaluated with a linear mixed model. Individual interviewer mean scores were ranked from lowest scores to highest scores. Each candidate's interview score was then plotted against the combined rank of their respective interviewers and significance was calculated for linear regression. The average deviation of each interviewer was calculated to obtain an adjustment score individualized for each interviewer. Results: One thousand three hundred and five interview scores from 91 interviewers were included in the analysis. Interview scoring ranged from 1 (lowest) to 4 (highest). The average score was 3.35 (standard deviation [SD] 0.56). The interviewers conducted an average of 14 (SD 11.4) interviews during the study period. Each interviewer averaged 8.25 interviews per year (SD 5). There was no difference in scores by year (p = 0.20) or by order of interview (p = 0.33). Plotting average applicant score against rank of interviewers revealed a variance between the lowest scoring and highest scoring interviewers revealing a progressive bias (see Figure 3; p < 0.0001). The calculated adjustment factor was added or subtracted to each interviewer's score and the linear model was recalculated. The new plot revealed a lack of bias between interviewer ranking and scores (p = 0.32). Conclusions: In a large cohort of residency interviewers, interview scores demonstrated significant interviewer bias. This bias has the potential to significantly alter an applicant's rank list position. An adjustment score can be calculated to reduce this bias in interview scores. Prospective validation of this adjustment will be helpful in determining its utility in candidate ranking.