Measuring non-technical skills of anesthesiologists in the operating room: A systematic review of assessment tools and their measurement properties

Abstract

Background and Rationale: Clinical competence in anesthesia requires proficiency in non-technical skills (e.g. communication, leadership, situation awareness). Comprehensive assessment of non-technical skills performance in clinical practice requires robust tool(s) to be identified. It is currently unknown which assessment tools are the most robust for assessing non-technical skills in anesthesia, with inconsistency in the tools and settings used in evaluations of anesthesiologists’ non-technical skills. Objectives: This systematic review will: (1) summarize the tools used to measure the intraoperative non-technical performance of anesthesiologists; and (2) synthesize the psychometric properties of these tools. Search strategy and information sources: Literature searches will be conducted by an experienced librarian collaborating closely with the team of investigators. Medline and Medline in Process (via OVID), PsycINFO, CINAHL, Embase (via OVID), and ERIC will be searched with no date or language restrictions. The Medline search strategy will be peer-reviewed by a second information specialist using the PRESS tool. Adjustments will be made to the search for each database to optimize search results. Reference lists of previously published systematic reviews and of included articles will also be searched for additional relevant references. Eligibility criteria: Inclusion Criteria: Examine the psychometric properties (i.e. validity and reliability) of tools specifically intended to assess the non-technical skills of anesthesiologists (either trainee or graduated); Tools are evaluated within a clinical or simulation intraoperative environment; Study must include a quantitative analysis of psychometric properties or qualitative assessment of forms of validity; Tools must be developed for objective assessment of skills. Exclusion Criteria: Tools that assess anesthesia assistants, nurse anesthetists, and interprofessional teams; Tools that include technical skills items; Studies where psychometric assessment is not the primary outcome; Studies that evaluate anesthesiologists’ performance but not the assessment tool itself; Tools that are developed for subjective (i.e. self-reported) assessment of skills. Study selection: Titles and abstracts will be screened in duplicate for eligibility by two independent reviewers. Full-texts of included studies will then be reviewed. Disagreements at each level of screening will be resolved by consensus discussion or assistance from another reviewer if needed. Data extraction: Data extraction will be conducted by one reviewer using an electronic data collection form for all included articles. Extracted information will them be verified by a second reviewer. The final list of included tools will be reviewed by a group of anesthesiologists to determine accuracy and completeness. The data extraction form will collect general article information (e.g. year and study location), demographics of learners (e.g. trainee status), tool design (e.g. name and number of items), and psychometric outcomes (e.g. properties assessed and validation values). Study quality: Screeners will assess the methodological quality of included studies in duplicate using The COSMIN checklist.22 Disagreements will be resolved through consensus or a third reviewer as required. Synthesis: A narrative summary of the types of reliability and validity, psychometric coefficients, validation context (i.e. simulation or clinical), and level of psychometric evidence for each assessment tool (i.e. minimal, moderate, or extensive) will be completed for each included assessment tool.