The summer school took place in Bologna from June 9 to June 17, 2026, and was a fantastic opportunity for young PhD students and researchers to explore the exciting connections between imaging sciences and machine/deep learning.
In her poster, Inés introduced CHEM (Conformal Hallucination Estimation Metric), a method for detecting and quantifying hallucinations in image reconstruction models. By combining wavelet and shearlet representations with conformalized quantile regression, CHEM identifies realistic-looking artifacts that aren’t present in the ground truth in a way that doesn’t depend on distribution.
relAI is proud to announce an outstanding achievement – a first author publication of Moritz Knolle, one of relAI’s PhD students, in Nature journal.
Medical AI models are increasingly utilized in applications such as diagnosing and remotely treating patients. While these models have proven valuable to both practitioners and patients, concerns remain about the privacy of patients whose information is used to train these AI systems. This issue has been examined by relAI PhD student Moritz Knolle, in a study published this week in Nature. The analysis revealed privacy vulnerabilities in medical AI models, emphasizing the importance of reliable AI research in medicine and healthcare 🩺.
🔍 Check the summary below and the article to learn more about the study!.
Summary of the article
Individuals whose data are used to train medical AI models may be at risk of being identified in cyber-attacks, according to a Nature paper published this week. Underrepresented groups may face disproportionately higher risks of having their data compromised, the study indicates. The researchers find these individuals are not accounted for in current risk assessments and call for further mitigation and strict access control.
Medical AI models may improve global health outcomes, especially in areas in which specialized expertise is not available. Yet, the sensitive data used to train these models may be exposed to privacy attacks. Membership inference attacks (MIAs) are used by attackers to determine whether an individual’s data were used to train a model. From these attacks, a patient’s medical data and private information can be determined. Previous research on data risk has been determined by whole datasets, and does not take an individual’s risk into account.
Moritz Knolle and colleagues conduct a privacy audit to focus on individual privacy risk, finding that medical AI models may pose a privacy risk to individual data contributors. Using seven large datasets made up of real-world clinical data, including medical images, electrocardiograms and electronic health records, the authors determine the most vulnerable among data-contributing patients. They find that at an individual level, those targeted by the MIAs were successfully done so with almost no error. At a group level, those identified as underrepresented in datasets include people with rare diseases, people from a minority racial group or, socioeconomic status, or those having the less-common gender. With more distinctive data that are encoded by AI models, these groups and individuals are found to be more vulnerable and disproportionately exposed to privacy attacks. The authors find the instances of successful MIAs attacks rise with model capacity and size.
These findings show privacy attacks, such as MIA, are more effective at successfully targeting on an individual level than currently thought. The authors conclude that privacy risk assessment must now take individual risk into account, and vulnerable models be further protected."
We are excited to announce that the call for applications to the PhD program 2026 of our Konrad Zuse School of Excellence in Reliable AI (relAI) is now open!
The novel, innovative PhD relAI program offers a cross-sectional training for successful education in AI including scientific knowledge, professional development courses and industrial exposure, providing a coherent, yet flexible and personalised training.
Funded applicants will receive a full salary for three years, including social benefits (TV-L E13 of the German public sector). They may receive additional support through travel grants for conference attendance, research stays, or home travel. Doctoral students are hosted by a relAI Fellow who helps them to define their research project. Depending on the affiliation of this hosting fellow, they enrol at TUM or LMU.
We highly encourage you to apply if you have:
an excellent master’s degree (or equivalent) in computer science, mathematics, engineering, natural sciences or other data science/machine learning/AI related disciplines;
a genuine interest to work on a topic of reliable AI covering aspects such as safety, security, privacy and responsibility in one relAI’s research areas Mathematical & Algorithmic foundations, Algorithmic Decision-Making, Medicine & Healthcare, Robotics & Interacting Systems, or Learning and Education;
Please help us in spreading the word, especially to excellent international candidates.
/ mc
🎉 Congratulations!
We are excited to announce that a team consisting of relAI PhD students Shuo Chen, Bailan He, and Jingpei Wu, along with relAI Fellow Volker Tresp and members of the Torr Vision Group from the University of Oxford and TU Berlin, received the Honorable Mention Award at OpenAI Red-Teaming Challenge on Kaggle. They ranked among the top 20 teams (Top 3%) out of 5,911 participants and over 600 teams.
The Red Teaming Challenge, initiated by OpenAI, tasked participants with probing its newly released open-weight model, gpt-oss-20b. The objective was to identify previously undetected vulnerabilities and harmful behaviors, such as lying, deceptive alignment, and reward-hacking exploits.
Would you like to learn more about the awarded work?
The write-up of the hackathon and the accompanying paper, “Bag of Tricks for Subverting Reasoning-Based Safety Guardrails,” detail the findings of the study, revealing systemic vulnerabilities in recent reasoning-based safety guardrails like Deliberative Alignment.
We are proud to share that relAI PhD student Yusuf Sale has been honored with one of the IJAR Young Researcher Awards. The prestigious prize, funded by the International Journal of Approximate Reasoning (IJAR), recognizes students who demonstrate excellence in research at an early stage of their scientific careers.
Yusuf received the award at ISIPTA 25, the 14th International Symposium on Imprecise Probabilities: Theories and Applications, organized by ISIPTA, the leading international forum for theories and applications of imprecise probabilities.
This is an exceptional outcome, considering that only six papers have received this recognition out of more than 12000 submitted this year.
relAI has been instrumental in fostering the collaboration that led to this significant outcome by funding Unai Fisher Abaigar's research stay at Harvard University. Visits to international centres are one of the components of the relAI PhD curriculum, designed to support collaborations with international researchers and gain international research experience on the topic of the reliability of artificial intelligence (AI).
The paper tackles aspects of the Algorithmic Decision-Making relAI research area and the relAI central theme Responsibility, exploring how predictive models, particularly those using machine learning, can be used in government programs to identify and support the most vulnerable individuals.
/ mc
On the latest TV episode of “Neuland” by BR - Bayerischer Rundfunk, relAI PhD student Sarah Ball shares her insights about fundamental issues surrounding a central theme of relAI: “responsibility in AI systems.” She addresses topics such as when AI might reinforce discrimination and how to ensure that AI systems align with human values.
The recent work of relAI PhD student Lukas Gosch has won the Best Paper Award at the 3rd AdvML-Frontiers workshop at the 38th Annual Conference on Neural Information Processing Systems (NeurIPS 2024). The workshop and paper presentation took place at the Vancouver Convention Center in Canada on December 14th, 2024.
Lukas is a PhD student at relAI, advised by the relAI Co-Director Prof. Dr. Stephan Günnemann. His research focuses on robust and reliable machine learning, as well as machine learning on graphs.
The award-winning paper „Provable Robustness of (Graph) Neural Networks Against Data Poisoning and Backdoor Attacks“, that Lukas authored together with Mahalakshmi Sabanayagam and relAI Fellows Debarghya Ghoshdastidar and Stephan Günnemann, develops the first architecture-aware certification technique for common neural networks against poisoning and backdoor attacks.
Our sincerest congratulations to Lukas and his co-authors on this achievement!
/ mc
We are excited to announce that the call for applications to the PhD program 2025 of our Konrad Zuse School of Excellence in Reliable AI (relAI) is now open!
The novel, innovative PhD relAI program offers a cross-sectional training for successful education in AI including scientific knowledge, professional development courses and industrial exposure, providing a coherent, yet flexible and personalised training.
Funded applicants will receive a full salary for three years, including social benefits (TV-L E13 of the German public sector). They are further supported by travel grants, e.g. for conference attendance, research stays or home travel. Doctoral students are hosted by a relAI Fellow who helps them to define their research project. Depending on the affiliation of this hosting fellow they enrol at TUM or LMU.
We highly encourage you to apply if you have:
an excellent master’s degree (or equivalent) in computer science, mathematics, engineering, natural sciences or other data science/machine learning/AI related disciplines;
a genuine interest to work on a topic of reliable AI covering aspects such as safety, security, privacy and responsibility in one relAI’s research areas Mathematical & Algorithmic foundations, Algorithmic Decision-Making, Medicine & Healthcare or Robotics & Interacting Systems;
Please help us in spreading the word, especially to excellent international candidates.
/ mc
Congratulations! relAI student Sameer Ambekar wins the best paper award at the MICCAI Workshop on Advancing Data Solutions in Medical Imaging AI (ADSMI).
Sameer is a PhD student at relAI, advised by the relAI Fellow Julia A. Schnabel. His research focusses on test-time adaptation and domain generalization for medical imaging.
His award-winning paper “Selective Test-Time Adaptation for Unsupervised Anomaly Detection using Neural Implicit Representations”, co-authored with Julia A. Schnabel and Cosmin Bereca, presents a novel zero-shot methodology to adapt models in real time to test images from new domains using deep pre-trained features. The approach is validated on brain anomaly detection data.
This work addresses domain shift at test-time, which Sameer explains in more detail in his recently published relAI blog post. In the post, you can also learn about the importance of handling domain shifts to make AI more reliable: https://zuseschoolrelai.de/blog/mitigating-domain-shifts/
