Virginia Research Day 2025

Faculty Research Education and Simulation

Daniels T.; Perera IR.; Moriarty S.; Noor S.; Rastogi S.; Crouse C.; Kharel A.; Januchowski R.; Looney J.; Gittings K.; Rawlins II F. Corresponding author: frawlins@vcom.edu tdaniels@vcom.edu 01 Comparison Of Novel Voice Recognition Models In Medical Student Decision Making During Case Presentation

Center for Simulation and Educational Technology Edward Via College of Osteopathic Medicine - Virginia Campus

text (STT) in medical student MDM during clinical case presentations. Methods: Approximately 154 previously de identified Integrated Clinical Case (ICC) recordings were transcribed by student researchers and validated by an attending physician. These initial transcripts were considered True Transcriptions. Additionally, recordings were transcribed by Deepgram Nova-2 and Whisper V3, for comparison of STT. The textual output was cleaned for abbreviations, punctuation, grammar, and other textual impurities. This process was universally applied to both Deepgram Nova-2 and Whisper V3. Next, the textual output for the models was analyzed using Term Frequency Inverse Document Frequency (TFIDF). The textual output of the two Large Language Models (LLM) was then paired against the true transcription, generating standard LLM error metrics: 1. Jaccard Similarity, 2. Jaro-Winkler Distance, 3. Leveshtein Distance, 4. Word Error Rate, 5. Word Information Loss, 6. Match

Error Rate, and 7. Character Error Rate. The metrics of the two LLMs were analyzed and compared utilizing paired t-tests. Results: Comparison of Deepgram Nova-2 and Whisper V3 demonstrated a statistically significant difference in all seven LLM error metrics (P < 0.05). Conclusion: Deepgram Nova-2 outperformed Whisper V3 in all STT LLM evaluation metrics, demonstrating its ability to produce an accurate medical transcript in medical student MDM during clinical case presentations. Innovative educational technology leveraging this model could be implemented into the medical student curriculum to elevate both VCOM and our students to a higher tier of academic excellence.

Context: Medical schools are required by accrediting bodies to ensure students attain mastery of medical knowledge, patient care, and appropriate communication within an interdisciplinary healthcare team to collaborate in medical decision making (MDM). VCOM meets the accrediting core competency of communication through its Integrated Clinical Cases (ICC) course. Medical students are required to make an oral presentation of their simulated patient encounter and complete a self-assessment of their performance. However, for medical schools to produce students of the highest caliber we must provide instantaneous high-quality expert feedback related to student MDM, this is an area in which our current curricular practices are inadequate. The introduction of a novel voice recognition software could bridge this gap. Objective: To compare two novel voice recognition models, Deepgram Nova-2 and Whisper V3, on their ability to accurately convert speech-to

IRB #2112436-4.

Table of Contents

19

2025 Research Recognition Day