"To one, it is a wall; to another, a rope; to a third, a spear."
In the old story of the blind men and the elephant, each man touches a single part and is certain he knows the whole. No one can picture the whole animal, because no one has felt more than his piece of it. Yet this is exactly how we train medical AI today. In valvular heart disease (VHD) especially, one model sees only the echocardiogram, another only the electrocardiogram (ECG), a third only the electronic health records (EHR). Each is a blind man certain of his single part. Your cardiologist, in contrast, steps back, and always takes in the whole elephant.
Valvular heart disease management is inherently multimodal
Meet Herr Schmidt. He is a 75-year-old retiree living in the suburbs of Munich. At the LMU University Hospital in Munich, a structural heart team is evaluating Herr Schmidt for severe tricuspid regurgitation to see if he qualifies for a transcatheter valve intervention.
To make this important decision, clinicians never rely on a single test. Instead, they naturally combine multiple data sources: Herr Schmidt's reported symptoms (text), his heart murmurs (audio), his ECG traces (waveforms), and his 3D echocardiograms (imaging).
This isn't just unique to the hospital; the 2025 ESC/EACTS Guidelines1 explicitly calls for an integrated, multimodal approach to manage valvular heart disease. In the clinic, doctors operate as multimodal agents by default. Yet, the AI models we build to assist them remain trapped in single-modality silos2.
Multimodal VHD models are scarce
Before counting how many models are multimodal, we have to agree on what the word means. A truly multimodal model learns from the raw, native representations of different data types. The text, audio and imaging data each pass through its own encoder so the model sees the signal the way the modality actually presents it.
However, a far more common pattern exists in machine learning VHD research. A clinician or researcher manually reads the different data sources and extracts the findings into a single structured table, and feeds that table to a tabular model. The information comes from many sources, but the model is unimodal: it only ever sees numbers in rows (See Figure 1). This unimodal-but-multisource approach is common in the field as an acknowledgement of the importance of different data sources in clinical care but it also sidesteps the hardest problems of fusing raw signals.
With this distinction in hand, the scarcity is glaring. In a recent systematic review2, we found that only 8% of predictive models for valvular heart disease integrated more than a single modality. The overwhelming majority were built on structured EHR data alone with a smaller cluster trained on heart sounds or echocardiography in isolation. None combined the clinical history, laboratory findings and longitudinal imaging that a cardiologist considers when managing these patients.
This gap is profound because it diverges from how we actually practice medicine. Though the guidelines favor an inherently integrative approach, our models seem to reproduce the single-organ, single-signal approach from an earlier period of medical computing. We are building siloed specialists to solve holistic clinical problems, and then wondering why their solutions feel so narrow.
This scarcity is not simply a matter of underinvestment or lack of interest. Multimodal medical models are genuinely harder to build than their unimodal counterparts. Medical data are misaligned, frequently incomplete, and unevenly informative, and the methods for fusing them are still being developed.
Figure 1: Multimodal machine learning model vs unimodal with multisource learning model
Why medicine breaks multimodal AI
To build an excellent multimodal model, one needs abundant, complete, well-aligned, and independently informative multimodal data. This doesn't exist in medicine. While other fields seem to have solved the multimodality conundrum, medicine, and VHD in particular, struggles to do so for this very reason
Outside of medicine, it is possible to either input missing data modalities or drop samples with missing data modalities. This doesn't necessarily apply in VHD care. For these patients, the test the doctor orders is itself a signal. A clinician would order a transesophageal echocardiography because they already suspect something. This is informative missingness. Hence a model that ignores this is discarding information vital for clinical translation. Robustness to missing modalities at inference time is therefore crucial for any multimodal model developed for VHD management.
Additionally, alignment and ground truth in VHD management are extremely tricky. Medical data is rarely collected at the same time. An ECG from January and an echo from June may describe 2 different hearts from the same patient. Multimodal models have to account for the longitudinal nature of the different modalities involved. Beyond their longitudinal complexity, the modalities share a deeper problem: the "ground truth" they are labelled against is itself partly subjective. Several cardiovascular studies report substantial physician agreement on outcome labels, often near 90%3,4. But 90% is not the same as ground truth. It means that for roughly one patient in ten, the correct label depends on which expert you ask, and a model trained on a single adjudicated answer inherits that disagreement as invisible noise.
Finally, the deployment bar is higher for medical AI. A model trained on one hospital's devices, populations, and ordering habits will degrade at the next. Distribution shift is commonplace in medicine. Added to this, medical AI faces regulatory demands, reliability thresholds, and prospective validation that are absent in other fields. Meeting these strict benchmarks remains an uphill battle for simpler unimodal models, let alone highly complex multimodal networks. Multimodal models that do not meet these demands, regardless of the complexity of the data sources, will not be deployed in clinical practice.
From Benchmark to Bedside: Building Models Clinicians Will Actually Use
So far, we've argued that VHD management demands multimodal models and that medicine makes them hard to build. However, dealing with these technical issues is not the final hurdle. Some models clear the technical bar and still never reach a patient. To get a multimodal model to the bedside, developers, and the field at large need to address three main barriers to translation.
- Measure what matters at the bedside: A model can top a benchmark on AUROC and still be unsafe, because discrimination says nothing about whether its probabilities are trustworthy. Calibration, decision-curve analysis, and performance within prospectively defined subgroups matter more here than a single headline metric. This is even more vital given the existent label noise. Can a clinician trust the model's outputs in a world it wasn't trained on? A model validated only on historical data has been tested on the world as it was, not the world it will be deployed into. Hence, models deployed on retrospective datasets are less likely to be adopted into clinical practice.
- Make the model explainable: A model is rarely used if the clinician can't see why it decided what it did. Explainability is what lets a cardiologist integrate the model's recommendation into their own understanding of the case. Importantly, it also allows them to override the model when its judgement differs. Explainability is challenging even in unimodal models but it is highly important in multimodal models where clinical value lies not only in the modalities that drive the output but in how the signals were combined.
- Build the shared infrastructure: Multimodal models are scarce partially because the infrastructure to support them is lacking. There are almost no shared multimodal VHD datasets, no agreed reporting standards, and, critically, no benchmark that tests robustness to missing modalities. Creating this infrastructure is hard work, but its existence is what turns isolated proofs-of-concept into clinically usable products.
Multimodality is not a feature of good VHD management, it is the basis of it. Yet the models we build to assist VHD care remain largely unimodal. Closing this gap means learning to handle missing modalities, contested labels, and the hard work of fusing raw signals rather than flattening them into a table.
Herr Schmidt's heart team already works as a multimodal agent, integrating his symptoms, his murmur, his ECG, and his echo into a single judgment. A viable ML model has to learn to function in a similar manner. The cardiologist is already multimodal. It is time our models were too.
References
1. Praz, F., Borger, M. A., Lanz, J., Marin-Cuartas, M., Abreu, A., Adamo, M., Ajmone Marsan, N., Barili, F., Bonaros, N., Cosyns, B., De Paulis, R., Gamra, H., Jahangiri, M., Jeppsson, A., Klautz, R. J. M., Mores, B., Pérez-David, E., Pöss, J., Prendergast, B. D., Rocca, B., Rossello, X., Suzuki, M., Thiele, H., Tribouilloy, C. M., Wojakowski, W., & ESC/EACTS Scientific Document Group. (2025). 2025 ESC/EACTS guidelines for the management of valvular heart disease: Developed by the task force for the management of valvular heart disease of the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS). European Heart Journal, 46(44), 4635–4736. https://doi.org/10.1093/eurheartj/ehaf194
2. Idakwo, V. O., Strote, C., Goelz, C., Shafei, Q., Stocker, T. J., Hausleiter, J., & Vieluf, S.. Unimodal to multimodal: A systematic review of predictive machine learning models for valvular heart diseases. Frontiers in Cardiovascular Medicine. https://doi.org/10.3389/fcvm.2026.1855775
3. Kamyszek, R. W., Newman, N., Ragheb, J. W., Sjoding, M. W., Joo, H., Maile, M. D., Cassidy, R. B., Golbus, J. R., Engoren, M. C., Mathis, M. R., & Michigan Congestive Heart Failure Investigators. (2023). Differences between patients in whom physicians agree versus disagree about the preoperative diagnosis of heart failure. Journal of Clinical Anesthesia, 90, Article 111226. https://doi.org/10.1016/j.jclinane.2023.111226
4. Mathis, M. R., Ghadimi, K., Benner, A., Jewell, E. S., Janda, A. M., Joo, H., Maile, M. D., Golbus, J. R., Aaronson, K. D., Engoren, M. C., & Michigan Congestive Heart Failure Investigators. (2025). Heart failure diagnostic accuracy, intraoperative fluid management, and postoperative acute kidney injury: A single-centre prospective observational study. British Journal of Anaesthesia, 134(1), 32–44. https://doi.org/10.1016/j.bja.2024.08.020. (Erratum published 2025, British Journal of Anaesthesia, 135(3), 828. https://doi.org/10.1016/j.bja.2025.06.012)
About the Author
Valentine Idakwo, MD, MSc is a physician and doctoral researcher specializing in Machine Learning for Valvular Heart Diseases at Dr. Solveig Vieluf's AI-Based Telemonitoring research group (LMU University Hospital). As a member of relAI, his work focuses on the intersection of clinical cardiovascular management and reliable AI systems.
