The surgical efficiency score: a feasible, reliable, and valid method of skills assessment

Background Technical skills assessments are being increasingly used in surgical residency programs, with the objectivity and validity of several techniques well established. However, many of these methods are labor and time intensive, limiting their feasibility. This study aims to compare more efficient techniques of skills appraisals with an established gold standard. Methods Thirty surgeons completed 2 previously validated laboratory-based surgical models: small bowel anastomosis and vein patch insertion. Gold standard evaluation was the Objective Structured Assessment of Technical Skills (OSATS) method. “Efficient” techniques used were (1) quality of final product (FP); (2) snapshot assessment (SS), in which task performance was edited to a 2-minute sound bite and scored with OSATS; and (3) the surgical efficiency score (SES), a combination of final product quality and hand-motion analysis. All human observer evaluations used retrospective video analysis with 3 trained observers. Nonparametric tests were used to analyze the results. Results With respect to small bowel anastomosis, correlations with OSATS were as follows: FP 0.341 (P = .07), SS 0.577 (P< .001), and SES 0.842 (P< .001). For vein patch insertion, the correlations were as follows: FP 0.545 (P = .001), SS 0.609 (P< .001), and SES 0.700 (P< .001). Interobserver concordance was high for both models with respect to FP (Cronbach’s alpha 0.80 for small bowel anastomosis and 0.84 for vein patch insertion). With respect to SS, interobserver reliability was high for vein patch insertion (Cronbach’s alpha 0.80) but only moderate for small bowel anastomosis (0.59). Conclusions The surgical efficiency score and snap shot assessments both show significant correlations with the traditional OSATS appraisals and suggest that skills assessment can be made more feasible. Correlations were closer with the former and interobserver concordance more variable with the latter, suggesting the surgical efficiency score as the most reliable of the methods evaluated.