Criterion Validity for Assessment of Laparoscopic Skills; What do We Actually Assess?
EAES Academy. Hardon S. 07/05/22; 363126; P171
Mr. Sem Hardon
Contributions
Contributions
Abstract
Aims:
Technical innovations in simulation training for minimally invasive surgery have evolved rapidly over the past few decades. However, trainers still traditionally assess skills and readiness for surgery in the operating room using assessment forms. This study aimed to reassess the validity of the gold standard when compared to novel innovative assessment tools.
Methods:
Laparoscopically naive surgical residents were enrolled in a three-week fundamental laparoscopic skills (FLS) course. Laparoscopic suturing was performed for pre- and post-course assessments. Performances were assessed using the Objective Structured Assessment of Technical Skills (OSATS) form, the ForceSense measurement system, and video analyses. Pre- and post-course comparisons for each tool were performed to determine the acquisition of technical competence. Moreover, these outcomes were compared and evaluated to assess the criterion validity.
Results:
Twenty-four trainees performed pre-course and post-course suturing tasks. The post-course OSATS score increased by 7.7 point (p ≤ 0.001). Furthermore, a significant improvement was observed in the post-course task time (-41.5%, p = 0.001), left-hand depth perception (+11%, p=0.020), and path length (-32.8%, p = 0.001), compared to the pre-course task. Trainees dropped their needle (-77.8%, p ≤ 0.001) or thread (-50.0%, p = 0.021) less often during the post-course task and were less likely to manipulate the tissue unnecessarily (-56.9%, p = 0.001). No significant correlations were found between post-course OSATS scores and neither post-course ForceSense parameters nor post-course cumulative errors obtained from video analysis.
Conclusion:
OSATS, the ForceSense system and video analyses can accurately assess the extent to which FLS training prepares trainees for laparoscopic surgery. However, this study shows major differences among these assessment tools, which are commonly used to determine technical competence, according to criterion and construct validity assessment. Surgical trainers should be aware of these outcomes, and carefully choose according to the purpose of the assessment.
Technical innovations in simulation training for minimally invasive surgery have evolved rapidly over the past few decades. However, trainers still traditionally assess skills and readiness for surgery in the operating room using assessment forms. This study aimed to reassess the validity of the gold standard when compared to novel innovative assessment tools.
Methods:
Laparoscopically naive surgical residents were enrolled in a three-week fundamental laparoscopic skills (FLS) course. Laparoscopic suturing was performed for pre- and post-course assessments. Performances were assessed using the Objective Structured Assessment of Technical Skills (OSATS) form, the ForceSense measurement system, and video analyses. Pre- and post-course comparisons for each tool were performed to determine the acquisition of technical competence. Moreover, these outcomes were compared and evaluated to assess the criterion validity.
Results:
Twenty-four trainees performed pre-course and post-course suturing tasks. The post-course OSATS score increased by 7.7 point (p ≤ 0.001). Furthermore, a significant improvement was observed in the post-course task time (-41.5%, p = 0.001), left-hand depth perception (+11%, p=0.020), and path length (-32.8%, p = 0.001), compared to the pre-course task. Trainees dropped their needle (-77.8%, p ≤ 0.001) or thread (-50.0%, p = 0.021) less often during the post-course task and were less likely to manipulate the tissue unnecessarily (-56.9%, p = 0.001). No significant correlations were found between post-course OSATS scores and neither post-course ForceSense parameters nor post-course cumulative errors obtained from video analysis.
Conclusion:
OSATS, the ForceSense system and video analyses can accurately assess the extent to which FLS training prepares trainees for laparoscopic surgery. However, this study shows major differences among these assessment tools, which are commonly used to determine technical competence, according to criterion and construct validity assessment. Surgical trainers should be aware of these outcomes, and carefully choose according to the purpose of the assessment.
Aims:
Technical innovations in simulation training for minimally invasive surgery have evolved rapidly over the past few decades. However, trainers still traditionally assess skills and readiness for surgery in the operating room using assessment forms. This study aimed to reassess the validity of the gold standard when compared to novel innovative assessment tools.
Methods:
Laparoscopically naive surgical residents were enrolled in a three-week fundamental laparoscopic skills (FLS) course. Laparoscopic suturing was performed for pre- and post-course assessments. Performances were assessed using the Objective Structured Assessment of Technical Skills (OSATS) form, the ForceSense measurement system, and video analyses. Pre- and post-course comparisons for each tool were performed to determine the acquisition of technical competence. Moreover, these outcomes were compared and evaluated to assess the criterion validity.
Results:
Twenty-four trainees performed pre-course and post-course suturing tasks. The post-course OSATS score increased by 7.7 point (p ≤ 0.001). Furthermore, a significant improvement was observed in the post-course task time (-41.5%, p = 0.001), left-hand depth perception (+11%, p=0.020), and path length (-32.8%, p = 0.001), compared to the pre-course task. Trainees dropped their needle (-77.8%, p ≤ 0.001) or thread (-50.0%, p = 0.021) less often during the post-course task and were less likely to manipulate the tissue unnecessarily (-56.9%, p = 0.001). No significant correlations were found between post-course OSATS scores and neither post-course ForceSense parameters nor post-course cumulative errors obtained from video analysis.
Conclusion:
OSATS, the ForceSense system and video analyses can accurately assess the extent to which FLS training prepares trainees for laparoscopic surgery. However, this study shows major differences among these assessment tools, which are commonly used to determine technical competence, according to criterion and construct validity assessment. Surgical trainers should be aware of these outcomes, and carefully choose according to the purpose of the assessment.
Technical innovations in simulation training for minimally invasive surgery have evolved rapidly over the past few decades. However, trainers still traditionally assess skills and readiness for surgery in the operating room using assessment forms. This study aimed to reassess the validity of the gold standard when compared to novel innovative assessment tools.
Methods:
Laparoscopically naive surgical residents were enrolled in a three-week fundamental laparoscopic skills (FLS) course. Laparoscopic suturing was performed for pre- and post-course assessments. Performances were assessed using the Objective Structured Assessment of Technical Skills (OSATS) form, the ForceSense measurement system, and video analyses. Pre- and post-course comparisons for each tool were performed to determine the acquisition of technical competence. Moreover, these outcomes were compared and evaluated to assess the criterion validity.
Results:
Twenty-four trainees performed pre-course and post-course suturing tasks. The post-course OSATS score increased by 7.7 point (p ≤ 0.001). Furthermore, a significant improvement was observed in the post-course task time (-41.5%, p = 0.001), left-hand depth perception (+11%, p=0.020), and path length (-32.8%, p = 0.001), compared to the pre-course task. Trainees dropped their needle (-77.8%, p ≤ 0.001) or thread (-50.0%, p = 0.021) less often during the post-course task and were less likely to manipulate the tissue unnecessarily (-56.9%, p = 0.001). No significant correlations were found between post-course OSATS scores and neither post-course ForceSense parameters nor post-course cumulative errors obtained from video analysis.
Conclusion:
OSATS, the ForceSense system and video analyses can accurately assess the extent to which FLS training prepares trainees for laparoscopic surgery. However, this study shows major differences among these assessment tools, which are commonly used to determine technical competence, according to criterion and construct validity assessment. Surgical trainers should be aware of these outcomes, and carefully choose according to the purpose of the assessment.
{{ help_message }}
{{filter}}