Erfolgreich eingeworben

MusteR-FM: AI foundation models for the analysis of surgical procedures

How can artificial intelligence gain a better understanding of surgical procedures? The ‘MusteR-FM’ project is developing methods to enhance foundational AI models for the analysis of surgical and endoscopic videos. The aim is to create versatile AI systems that can recognise and analyse medical procedures and, in the long term, enable intelligent assistance systems in the operating theatre.

What is the title of your research project?

MusteR-FM

Who is involved in the project?

Project leader: Prof. Dr Christoph Palm, Research assistant: Raphaela Märkl

What is your funded project?

MusteR-FM is developing methods to extend existing AI base models for the multimodal analysis of surgical and endoscopic videos. Such base models learn widely applicable representations and form the basis for a wide range of clinical applications, rather than being limited to individual tasks. In the current phase of the project, the focus is on minimally invasive surgery in collaboration with the Technical University of Munich. In a planned follow-up phase, the work is to be extended to endoscopic procedures in gastroenterology, in collaboration with Augsburg University Hospital.

What excites you about the project?

The most exciting aspect is the cross-domain approach. To date, AI systems in surgical image analysis have mostly been specialised for narrowly defined individual tasks. MusteR-FM, on the other hand, is developing methods designed to enable a model to generalise across different types of procedures and, in doing so, to understand not only individual images but also temporal sequences. This is only possible when methodology and clinical relevance come together, and that is precisely what the close collaboration with TUM and Augsburg University Hospital achieves. Another exciting aspect is the collaboration within the framework of the Bavarian AI Base Model Initiative, where the methods developed are set to contribute to the initiative’s joint health base model.

How can findings from the project be applied in practice?

A trained base model can serve as a starting point for a wide variety of applications. These could include, for example, the automatic detection of surgical phases, the segmentation of anatomical structures, the prediction of subsequent steps in a procedure, or support in the training of trainee doctors. In the long term, such functions could lead to intelligent assistance systems in the operating theatre. The methods developed in MusteR-FM are intended to be so widely applicable that they can serve as a basis for further research and application projects. As the Bavarian AI Base Model Initiative is committed to open access, research groups and application partners can build upon them.

 

On our website, you will find an overview of the current funded projects at OTH Regensburg.

Schematic illustration of AI-assisted analysis of surgical and endoscopic videos. Photo: Christoph Palm, created using ChatGPT/DALLE-E, OpenAI, 2026