Collaboration Stories

The purpose of this series is to tell the story of long-lasting, as well as emergent, stories of collaboration among clinicians and methodologists, content experts, and engineers, who have successfully worked on joint projects at the intersection of AI and health. Such collaborations are at the center of e-HAIL’s mission, and we offer these stories as a way of inspiring collaboration in other researchers.

“Pieces of a Puzzle”: Bringing Together Learning Sciences and Multimodal AI to Improve Medical Education

Drs. Vitaliy Popov and Mohamed Abouelenien first heard about each other in 2021, although it took a couple more years before they realized their research interests aligned. In 2024, they received e-HAIL supplemental award funding for their project Leveraging Multimodal Sensing and Machine Learning for Shared Situational Awareness in Acute Care Teams in 2024/25. This allowed them to successfully apply for a follow-up project on Advancing Intelligent Cognitive Load Sensing and Adaptive Scaffolding to Support Collaborative Simulation-based Learning in High-Stakes Environments, awarded in September 2025.

Popov and Abouelenien’s joint research revolves around developing new education methods for medical students that utilize virtual reality (VR) and AI. These cutting-edge methods overcome challenges such as instructors having limited time to run practice sessions and not being able to focus on each trainee individually throughout the practice sessions. Ultimately, Popov and Abouelenien are most passionate about the impact their work can have: “We are trying to improve medical education so that clinicians can perform optimally, and through that save lives.”

Popov is an assistant professor in the Department of Learning Health Sciences with a courtesy appointment at the U-M School of Information. He is also the Director of Learning Sciences and Technology for the Clinical Simulation Center at the U-M Medical School. His recent research focuses on developing educational methods through clinical simulation training and learning analytics.

Abouelenien is an associate professor of Computer and Information Science in the U-M Dearborn College of Engineering and Computer Science. Much of his research focuses on multimodal interaction and modeling human behavior using different fields such as natural language processing, computer vision, and applied machine learning.

Early on in his research, Popov realized the complexity of many clinical tasks that healthcare professionals need to master, and how the ability to capture and display multiple forms of data was necessary for improved medical education. This is especially important in training learners to be able to operate effectively in the context of shared decision making, team settings, and emergency scenarios. What was needed was a system that could capture that complexity and distill it into a digestible format for instructors and trainees. Learn more about Popov’s work here.

Abouelenien turned out to be the perfect partner for the effort to build such a system due to his focus on multimodal data. Together, they are developing analytics pipelines for trainees in medical settings. Their system simulates emergency scenarios where a team of clinicians must work together to quickly discover the patient’s problem. This type of scenario requires teams to collaborate under pressure, and members may not be familiar with each other prior to the emergency.

When completed, the posy-simulation debriefing analytics that Popov and Abouelenien are developing will enable several advantages over more traditional teaching methods. Their system captures activities and physiological parameters such as gaze, electrodermal activity, and speech, allowing for more individualized feedback to each participant. “We are even able to track cognitive load through wearable sensors,” says Popov. “That way, trainees learn when they are at risk of cognitive overload and hence more likely to miss a key event.” The system also allows for complex, varied cases. In one scenario, for instance, the trainees are attending to a patient going into cardiac arrest. While some team members focus on the monitor to track heart rhythm disturbances, they may neglect other key equipment in the room, such as an oxygen tubing that has become disconnected.

A final important aspect is cost: their VR simulations are more cost-effective, scalable, and demand less physical and mental strain on the facilitators, while also allowing more training opportunities for trainees.

Popov and Abouelenien’s ultimate goal is to improve healthcare delivery by developing systems to support healthcare professionals to work better in teams and when communicating with patients. Through what they learn from the simulated trainings, together with their clinical collaborators Drs. Michael Cole (Emergency Medicine) and James Cooke (Learning Health Sciences and Family Medicine) also want to improve teaching curricula to help healthcare professionals strengthen their performance, coordinate better, and deal with cognitive load in high stress situations.

“The data collection process is very tedious,” noted Abouelenien, requiring “multiple parties to be present at the same time as well as annotating the data.” e-HAIL has been able to support their work with student funding, allowing them to speed up the data collection. Although their data collection process has been going on for over a year now, they still require more data. “We’re still in the early stages,” continued Abouelenien, “We’re focusing mainly on the dialogue that happens between the physicians and trainees, simulated physiological signals of the patient manikins, and how the team operates from two camera angles.” They recently published a paper discussing how the multimodal data “poses significant challenges to existing models.”

While Abouelenien provides the computational muscle, Popov provides the knowledge of education. “We need the correct labeling schemes, the right systems, and the knowledge of how to interpret results. Vitaliy provides this knowledge and experience,” said Abouelenien, adding, “It’s all pieces of a puzzle that work together, and everyone is responsible for certain pieces.”

The kind of work Popov and Abouelenien are doing requires extensive interdisciplinary collaboration, which comes with its own challenges. One issue is the different approaches to publishing in their respective disciplines. In computer science, conference proceedings have become a common way for researchers to publish their work. As Abouelenien notes, “this makes sense given how rigorous some conferences are, and how difficult they are to get into.” On the other hand, the medical field still expects more traditional publishing via journals and may not look as highly on research published in computer science conference proceedings.

Another common challenge that research teams in this space face is being able to frame proposals so that they speak to the priorities of the National Institutes of Health (NIH) or those of the National Science Foundation (NSF) respectively. When writing to NIH, the focus and impact revolve around the disease, explained Popov. In contrast, for NSF the disease is just the use-case or context. “For NSF, we’re advancing the science, but the medical aspect is just the setting where we study it,” he added, highlighting this framing is something e-HAIL’s grant writer Hossam Abouzahr helped with in their proposal.