Evaluation using simulated students for reliability of multiple standardized-patients scoring in clinical performance examinations

Background: The consistency of rating among different standardized patients (SPs) is very important in clinical performance examinations (CPX). Aims: We evaluated the usefulness of simulated students to assess the reliability of a group of standardized patients in a CPX. Methods: Five SPs in each case were trained to simulate the same patient in a CPX. Ten fourth-year medical student volunteers were selected to act as simulated students. After the simulated students encountered the first SP, they reviewed a video of their performance and sequentially repeated the performance with the other 4 SPs. Results: The average maximum difference (Δmax ± SD) of multiple SPs' percent scores on a simulated student was 18 ± 7. In 10 pairs of SPs who gave largest Δmax on a given student, the median Δmax was 30 (range, 26-38) while median difference of percent score in the faculty scoring on the same simulated student was 3 (range, 0-7). Five problematic SPs whose ratings were significantly different from 3 or more other SPs' ratings in each case were identified. Conclusions: Simulated students may be a useful means to check the consistency of rating among different SPs in a CPX.