+ Site Statistics
+ Search Articles
+ PDF Full Text Service
How our service works
Request PDF Full Text
+ Follow Us
Follow on Facebook
Follow on Twitter
Follow on LinkedIn
+ Subscribe to Site Feeds
Most Shared
PDF Full Text
+ Translate
+ Recently Requested

Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively



Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively



Clinical Orthopaedics and Related Research 474(4): 874-881



Performance assessment in skills training is ideally based on objective, reliable, and clinically relevant indicators of success. The Objective Structured Assessment of Technical Skill (OSATS) is a reliable and valid tool that has been increasingly used in orthopaedic skills training. It uses a global rating approach to structure expert evaluation of technical skills with the experts working from a list of operative competencies that are each rated on a 5-point Likert scale anchored by behavioral descriptors. Given the observational nature of its scoring, the OSATS might not effectively assess the quality of surgical results. (1) Does OSATS scoring in an intraarticular fracture reduction training exercise correlate with the quality of the reduction? (2) Does OSATS scoring in a cadaveric extraarticular fracture fixation exercise correlate with the mechanical integrity of the fixation? Orthopaedic residents at the University of Iowa (six postgraduate year [PGY]-1s) and at the University of Minnesota (seven PGY-1s and eight PGY-2s) undertook a skills training exercise that involved reducing a simulated intraarticular fracture under fluoroscopic guidance. Iowa residents participated three times during 1 month, and Minnesota residents participated twice with 1 month between their two sessions. A fellowship-trained orthopaedic traumatologist rated each performance using a modified OSATS scoring scheme. The quality of the articular reduction obtained was then directly measured. Regression analysis was performed between OSATS scores and two metrics of articular reduction quality: articular surface deviation and estimated contact stress. Another skills training exercise involved fixing a simulated distal radius fracture in a cadaveric specimen. Thirty residents, distributed across four PGY classes (PGY-2 and PGY-3, n = 8 each; PGY-4 and PGY-5, n = 7 each), simultaneously completed the exercise at individual stations. One of three faculty hand surgeons independently scored each performance using a validated OSATS scoring system. The mechanical integrity of each fixation construct was then assessed in a materials testing machine. Regression analysis was performed between OSATS scores and two metrics of fixation integrity: stiffness and failure load. In the intraarticular fracture model, OSATS scores did not correlate with articular reduction quality (maximum surface deviations: R = 0.17, p = 0.25; maximum contact stress: R = 0.22, p = 0.13). Similarly in the cadaveric extraarticular fracture model, OSATS scores did not correlate with the integrity of the mechanical fixation (stiffness: R = 0.10, p = 0.60; failure load: R = 0.30, p = 0.10). OSATS scoring methods do not effectively assess the quality of the surgical result. Efforts must be made to incorporate assessment metrics that reflect the quality of the surgical result. New objective, reliable, and clinically relevant measures of the quality of the surgical result obtained by a trainee are urgently needed. For intraarticular fracture reduction and extraarticular fracture fixation, direct physical measurement of reduction quality and of mechanical integrity of fixation, respectively, meet this need.

Please choose payment method:






(PDF emailed within 0-6 h: $19.90)

Accession: 058444972

Download citation: RISBibTeXText

PMID: 26502107

DOI: 10.1007/s11999-015-4603-4


Related references

Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room. Surgery Today 43(3): 271-275, 2013

Value and face validity of objective structured assessment of technical skills (OSATS) for work based assessment of surgical skills in obstetrics and gynaecology. Medical Teacher 30(2): 212-216, 2008

A new curriculum for hysteroscopy training as demonstrated by an objective structured assessment of technical skills (OSATS). American Journal of Obstetrics and Gynecology 193(5): 1856-1865, 2005

Objective Structured Assessment of Technical Skills (OSATS) evaluation of hysteroscopy training: a prospective study. European Journal of Obstetrics, Gynecology, and Reproductive Biology 178(): 1-5, 2015

Comparing the Effects of Objective Structured Assessment of Technical Skills (OSATS) and Traditional Method on Learning of Students. Nursing and Midwifery Studies 4(2): E27714-E27714, 2015

Objective structured assessment of technical skill (OSATS) for surgical residents. British Journal of Surgery 84(2): 273-278, 1997

A randomized comparison of video demonstration versus hands-on training of medical students for vacuum delivery using Objective Structured Assessment of Technical Skills (OSATS). Medicine 96(11): E6355-E6355, 2017

Objective Structured Assessment of Technical Skills (OSATS) evaluation of theoretical versus hands-on training of vaginal breech delivery management: a randomized trial. European Journal of Obstetrics, Gynecology, and Reproductive Biology 171(2): 252-256, 2014

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence. Advances in Health Sciences Education 20(5): 1149-1175, 2015

Mersey deanery ophthalmology trainees' views of the objective assessment of surgical and technical skills (OSATS) workplace-based assessment tool. Perspectives on Medical Education 2(1): 21-27, 2013

Objective structured assessment of technical skills and checklist scales reliability compared for high stakes assessments. Anz Journal of Surgery 84(7-8): 568-573, 2015

Using objective structured assessment of technical skills to evaluate a basic skills simulation curriculum for first-year surgical residents. Journal of the American College of Surgeons 209(3): 364-370.E2, 2009

Passing the Objective Structured Assessment of Technical Skill Exam Predicts Future Technical Skills Performance in Surgical Trainees. Journal of the American College of Surgeons 221(4): S53-S54, 2015

Early, intermediate, and late effects of a surgical skills "boot camp" on an objective structured assessment of technical skills: a randomized controlled study. Journal of the American College of Surgeons 210(6): 984-989, 2010

Reliability of results produced through objectively structured assessment of technical skills (OSATS) for endotracheal intubation (ETI). Journal of the College of Physicians and Surgeons--Pakistan 23(1): 51-55, 2013