In this two-year project funded through the Stemmler Fund, investigators aim to develop and validate a machine learning (ML) model for automated workplace-based assessment (WBA) of clinical reasoning (CR) documentation. Previously, we developed an innovative WBA using ML and natural language processing for formative feedback on medicine residents’ CR documentation.

This grant supports two subsequent aims:

• improving the ML model to provide more specific feedback
• generating additional validity evidence to support use of the WBA in summative assessment

To demonstrate generalizability, we will conduct these aims in parallel at two institutions: NYU Grossman School of Medicine and the University of Cincinnati. Our current ML model has high performance but provides only dichotomous output, rating notes as low- or high-quality CR documentation. To generate more specific feedback, we will label a dataset of notes using a human assessment rubric we developed and validated in prior phases of this work, and train new ML models to provide feedback. We will gather validity evidence for the new ML model using Messick’s framework. After sufficient validity evidence is collected, this novel WBA can be integrated into CR competency–based assessment programs to help facilitate achievement of the sub-competency “appropriate utilization and completion of health records.” Our multi-site collaboration will also generate a process for implementation to facilitate dissemination of this tool across electronic health records.

Principal Investigators

Verity E. Schaye, MD, MHPE, principal investigator
Jesse Burk Rafel, MD, MRes, co-principal investigator
Sally Santen, MD, PhD, co-principal investigator

NYU Langone Co-Investigators

Yindalon Aphinyanaphongs, MD, PhD
Benedict Guzman, MS
Marina Marin, MSc
Daniel Sartori, MD

Collaborative Co-Investigators

Larry Gruppen, PhD
Eric Warm, MD
Danielle Weber, MD, MEd
Danny T. Y. Wu, PhD, MSI