ERC Starting Grant for relAI Fellow Vincent Fortuin
People
/ mc
🎉Congratulations!
We are very excited to announce that relAI Fellow Vincent Fortuin has been awarded a European Research Council (ERC) Starting Grant for his project "AutoBayes", with the title "Unlocking Reliable Small-data AI through Bayesian Deep Learning".
Most current deep learning models require large amounts of training data and tend to be overconfident in their predictions, undermining their reliability. Over the next five years, Vincent and his team will develop Bayesian deep learning methods to create AI models that are confident using limited data while also improving the quantification of the uncertainty in their predictions.
The ERC Starting Grant is one of the most competitive sources of funding available to early-career researchers; it enables them to pursue ambitious projects, to form their own teams and to gain independence at a crucial point in their careers. This year, the funding—amounting to €705 million in total—has been awarded to 421 early-career researchers throughout Europe.
relAI is proud to support the Philosophy of Machine Learning Conference (PhilML'26), which will take place in Munich from October 6 to 9. Tom Sterkenburg, a relAI Fellow, is one of the main organizers of this event.
The conference will address fundamental epistemological, ethical, and social questions related to machine learning through the lens of analytic philosophy. We welcome the following types of work:
1. Research that applies philosophical concepts and methods to gain insights into machine learning.
2. Research that critically reflects on the philosophical and ethical implications of machine learning.
To promote close and productive dialogue, PhilML brings together philosophers and machine learning researchers with a philosophical inclination, encouraging direct engagement with scientific and mathematical details.
In this work, the authors derive the neural network Gaussian process (NNGP) limits of graph transformers, demonstrating the structural advantages of attention mechanisms compared to standard graph convolutions.
As an expert on social biases in AI models, Valentin was asked to comment on the controversial AI chatbot of state Senator Scott Wiener, which targeted Supervisor Connie Chan, Wiener's opponent in the 2026 California congressional election. The chatbot was accused of producing racist and sexist responses of Hong Kong-born Chan, who is a non-native English speaker.
In the interview, Hofmann explained that chatbots like Wiener’s are trained on large-scale internet data, which often contains racist and offensive stereotypes related to accents and citizenship. After reviewing the chatbot’s responses, Hofmann suggested that the model likely inferred from biographical and other available information that Connie Chan speaks with an accent. He noted that, in a failed attempt at satire, the chatbot exploited this information.
Hofmann's research demonstrates that linguistic stereotypes, particularly bias related to dialect and accent, are difficult to eliminate from AI models because they are encoded in subtle ways, in contrast to more explicit forms of racial prejudice. He also commented on the difficulties of eliminating problematic behaviour in AI chatbots and on current approaches to addressing it.
Join us for a lively meetup featuring leading researchers from MDSI and relAI at the SAP Labs Munich Campus Auditorium, where we will connect, discuss, and explore new ideas with our industry partner SAP.
🗓️ September 22, 2026, 4:00 - 5:30 pm (doors open at 3:30 pm, talks start at 16:00)
📍 SAP Labs Munich Campus (MUE03), Friedrich-Ludwig-Bauer-Straße 5, 85748 Garching bei München, Auditorium (AE.76)
Bhavatarini Kumaravel(MDSI): A Graph-Based Deep Q-Learning Agent for Grammar-Guided Structural Form-Finding Bhavatarini's supervisor is the MDSI Core Member Prof. Pierluigi D'Acunto, TUM Chair for Structural Design, TUM School of Engineering and Design
The work investigates how graph-based deep reinforcement learning can be combined with a structural grammar in finding novel and materially efficient structural design solutions.
Annika Schneider(relAI): Decision-aligned Evaluation of Uncertainty Quantification Annika is a relAI PhD student at Helmholtz Munich and TUM, supervised by relAI Fellow Dr. Vincent Fortuin, TUM Associate Professorship of AI for Scientific Modelling
This work investigates the question of how probabilistic models should be evaluated to ensure they perform in downstream decisions.
Sameer Ambekar(relAI & MDSI): Adressing Distribution Shifts: The Shift from Static to Thinking-based Vision-Language Models Sameer is a relAI PhD student at Helmholtz Munich and TUM, supervised by relAI Fellow and MDSI Core Member Prof. Julia Schnabel, TUM Chair for Computational Imaging and AI in Medicine
Distribution shifts have remained a persistent challenge over the years, with solutions spanning from domain adaptation to test-time training and, most recently, post-training mechanisms for vision-language models. This is because models inevitably encounter unseen data at inference time, data they were never trained for.
This talk at SAP by Sameer will focus on addressing this problem, tracing it from convolutional architectures to modern vision-language models and reasoning models, and connecting these stages through a consistent pattern: models that allocate additional computation at test time, whether through adaptation or reasoning, generalize more reliably than those relying solely on fixed, pre-trained parameters. Sketching through these topics from his PhD research, the talk will provide an overview of addressing unseen data at test time through training, reasoning, and post-training mechanisms.
/ mc
From July 28 to July 31, 2026, relAI hosted the second relAI International Summer School at LMU. This event focused on promoting educational exchange aimed at developing reliable AI and exploring current trends in the field.
Last week, our industry partner, QuantCo, hosted a networking event with relAI at their new Munich office.
The relAI students had a wonderful evening filled with engaging conversations and delicious food. They had the opportunity to meet colleagues from QuantCo's AI, and engineering teams and learned about how the company develops scalable systems to transform messy, real-world data into impactful decisions.
The summer school took place in Bologna from June 9 to June 17, 2026, and was a fantastic opportunity for young PhD students and researchers to explore the exciting connections between imaging sciences and machine/deep learning.
In her poster, Inés introduced CHEM (Conformal Hallucination Estimation Metric), a method for detecting and quantifying hallucinations in image reconstruction models. By combining wavelet and shearlet representations with conformalized quantile regression, CHEM identifies realistic-looking artifacts that aren’t present in the ground truth in a way that doesn’t depend on distribution.
We are happy to announce that relAI will be represented in Seoul, South Korea, from July 6 to 11 at the Forty-Third International Conference on Machine Learning (ICML2026)!. ICML is widely recognized as one of the top three most influential conferences in machine learning and artificial intelligence research.
📖 About relAI Publications at ICML
relAI contributes fourteen publications to the conference, including nine in the main track.
relAI PhD student Emre Kavakwill present a Spotlight paper (ranked in the top 2.2%).
Five relAI papers will be presented at ICML Workshops, with contributions from three relAI MSc students.
🤝Meet relAI Students
If you are attending ICML we encourage you to engage in discussions about relAI research with the following relAI students present at the conference:
relAI is proud to announce an outstanding achievement – a first author publication of Moritz Knolle, one of relAI’s PhD students, in Nature journal.
Medical AI models are increasingly utilized in applications such as diagnosing and remotely treating patients. While these models have proven valuable to both practitioners and patients, concerns remain about the privacy of patients whose information is used to train these AI systems. This issue has been examined by relAI PhD student Moritz Knolle, in a study published this week in Nature. The analysis revealed privacy vulnerabilities in medical AI models, emphasizing the importance of reliable AI research in medicine and healthcare 🩺.
🔍 Check the summary below and the article to learn more about the study!.
Summary of the article
Individuals whose data are used to train medical AI models may be at risk of being identified in cyber-attacks, according to a Nature paper published this week. Underrepresented groups may face disproportionately higher risks of having their data compromised, the study indicates. The researchers find these individuals are not accounted for in current risk assessments and call for further mitigation and strict access control.
Medical AI models may improve global health outcomes, especially in areas in which specialized expertise is not available. Yet, the sensitive data used to train these models may be exposed to privacy attacks. Membership inference attacks (MIAs) are used by attackers to determine whether an individual’s data were used to train a model. From these attacks, a patient’s medical data and private information can be determined. Previous research on data risk has been determined by whole datasets, and does not take an individual’s risk into account.
Moritz Knolle and colleagues conduct a privacy audit to focus on individual privacy risk, finding that medical AI models may pose a privacy risk to individual data contributors. Using seven large datasets made up of real-world clinical data, including medical images, electrocardiograms and electronic health records, the authors determine the most vulnerable among data-contributing patients. They find that at an individual level, those targeted by the MIAs were successfully done so with almost no error. At a group level, those identified as underrepresented in datasets include people with rare diseases, people from a minority racial group or, socioeconomic status, or those having the less-common gender. With more distinctive data that are encoded by AI models, these groups and individuals are found to be more vulnerable and disproportionately exposed to privacy attacks. The authors find the instances of successful MIAs attacks rise with model capacity and size.
These findings show privacy attacks, such as MIA, are more effective at successfully targeting on an individual level than currently thought. The authors conclude that privacy risk assessment must now take individual risk into account, and vulnerable models be further protected."
