Munz Y, Moorthy K, Bann S, Shah J, Ivanova S, Darzi S. Ceiling effect in technical skills of surgical residents. Am J Surg. 2004 Sep; 188(3):294-300

PMID: 15450837

Abstract

BACKGROUND:
Objective evaluation of technical skills is now possible. The aim of this study was to evaluate whether a previously validated assessment tool can discriminate between junior and senior higher surgical trainees (HST) with regard to their technical skills.
METHODS:
Fifteen junior trainees (years 1 to 3) and 15 senior (years 4 to 6) were asked to perform a series of bench-model tasks, which included knot-tie at depth, vertical mattress suturing, sebaceous cyst excision, small bowel anastomosis, and saphenofemoral junction dissection. Analysis of performance included the Imperial College Surgical Assessment Device, which analyzed positional data such as total number of movements, total distance traveled, and time taken, and video-based qualitative evaluation of performance using global rating scales. In addition, all participants were asked to complete a self-assessment using exactly the same scoring system used by the observers. Statistical analysis included nonparametric tests and Cronbach's alpha was used for interrater reliability (IRR) A P value less than 0.05 was deemed significant.
RESULTS:
There were no significant differences in performance between junior and senior trainees for most of the tasks (IRR was 0.8 to 0.9). However, significant differences in performance were noted, in favor of the junior group, for knot-tie at depth, for two of the three parameters measured (total number of movements, P = 0.004; time taken, P = 0.01).
CONCLUSIONS:
In the absence of any significant differences between the groups, we can conclude that there is a ceiling effect (years 3 to 4) in performance beyond which pure technical skills, as measured by this tool, can no longer discriminate between levels of experience. Increasing the sensitivity of future assessment tools is necessary for better evaluation of performance levels. This finding also may imply that for the evaluation of senior trainees, other aspects of surgical competence should be assessed.



Author's contacts: yaron.munz@sheba.health.gov.il