The Development of a Workplace-Based Surgical Clinic Assessment Tool

Abstract

Purpose of Study: Workplace-based assessment is an opportunity for a learner to be assessed in their community of practice by an expert rater. The challenges and biases brought into this assessment relationship are complex. A shift towards Competency Based Medical Education in post-graduate residency education has triggered consideration of how to implement feasible assessment tools for the operating room, the in-patient ward, and the outpatient clinic. Competent performance in outpatient clinic is vital to surgical practice, yet no assessment tool currently exists to assess daily performance of technical and nontechnical skills of surgery residents. This project describes the development of a competency-based assessment tool, the Ottawa Clinic Assessment Tool (OCAT).

Research Question: How does the OCAT demonstrate validity for measurement of surgical resident performance in clinic?

Method: A consensus group of experts was gathered to generate ideas reflective of a competent ‘generalist’ surgeon in clinic. An entrustability anchor scale was developed. A six-month pilot study of the OCAT was conducted in orthopedics, general surgery and obstetrics and gynecology with quantitative and qualitative evidence of validity collected. Two subsequent feedback sessions, and a survey for staff and residents evaluated the OCAT for clarity and utility.

Results: The OCAT was developed as a 13-item tool, with a global assessment item and 2 short answers questions. 44 staff surgeons completed 132 OCAT assessments of 79 residents. Psychometric data was collected as evidence of internal structure validity and relations with other variables. Analysis of feedback indicated the rating scale was practical and useful for surgeons and residents.

Conclusions & Contribution to the Research Field: Surgical programs will require a daily clinic assessment tool to help define resident competency progression. Multiple sources of validity evidence collected in this pilot project demonstrate that the OCAT can measure resident clinic competency in a valid and feasible manner.