Artificial intelligence (AI) in health care is rapidly advancing beyond traditional applications. Autonomous AI agents are gaining significant attention for their potential to fundamentally transform medicine. However, researchers at the Else Kröner Fresenius Center (EKFZ) for Digital Health at TUD Dresden University of Technology highlight a growing mismatch between the capabilities of autonomous AI agents and existing medical device regulatory frameworks in the US and Europe.
In their publication, they propose innovative regulatory approaches to close this gap while ensuring high patient safety standards. The article has been published in the journal Nature Medicine.
The use of large language models (LLMs) and other forms of generative AI (GenAI) in health care has surged in recent years, and many of these technologies are already applied in clinical settings. As such, they often qualify as medical devices and must comply with specific laws and regulatory frameworks. Recent decisions have shown that approval is possible for applications with narrowly defined tasks.
The next generation of AI systems are broad autonomous AI agents designed to independently carry out complex, goal-directed workflows. They show great potential to support and further improve medical care in the future.
AI agents consist of multiple interconnected components, including external databases and computational tools for image analysis, note-taking, clinical guidance, and patient data management. These components are controlled by LLMs that take over decision-making, error handling, and the recognition of completed tasks.
“We are seeing a fundamental shift in how AI tools can be implemented in medicine,” says Jakob N. Kather, Professor of Clinical Artificial Intelligence at the EKFZ for Digital Health at TUD and oncologist at the Dresden University Hospital Dresden.
“Unlike earlier systems, AI agents are capable of managing complex clinical workflows autonomously. This opens up great opportunities for medicine—but also raises entirely new questions around safety, accountability, and regulation that we need to address,” he adds.
AI agents challenge existing regulatory frameworks
Current medical device regulations were designed for static, narrowly focused technologies that maintain human oversight and do not evolve after entering the market. In contrast, new autonomous and broad-scope technologies have fundamentally different characteristics. They demonstrate greater autonomy, adaptability, and scope than previous AI technologies. As they are capable of autonomously executing complex workflows, they present significant challenges for regulators and developers.
“To facilitate the safe and effective implementation of autonomous AI agents in health care, regulatory frameworks must evolve beyond static paradigms. We need adaptive regulatory oversight and flexible alternative approval pathways,” says Oscar Freyer, lead author of the publication and research associate in the team of Professor Stephen Gilbert, who leads the Medical Device Regulatory Science group at EKFZ for Digital Health, TUD.
Rethinking regulation: approval pathways for safe, innovative AI technologies
As AI agents’ capabilities exceed the scope of current regulatory frameworks, the researchers propose several potential solutions to overcome barriers to their implementation.
Immediate adaptations include extending enforcement discretion policies, where regulators acknowledge a product qualifies as a medical device but choose not to enforce certain requirements. Alternatively, a non-medical device classification could be applied to systems that serve a medical purpose but fall outside the traditional medical device regulation.
Medium-term solutions involve developing voluntary alternative pathways (VAPs) and adaptive regulatory frameworks that supplement existing approval processes. These adaptive pathways would shift from static pre-market approval to dynamic oversight using real-world performance data, stakeholder collaboration, and iterative updates. In case of misconduct, devices may be transferred to the established pathways.
The researchers also discuss long-term solutions, such as regulating AI agents in a similar way to the qualifications of medical professionals. In this model, regulation would be carried out through structured training processes, in which systems gain autonomy only after demonstrating safe and effective performance.
The researchers note that while tools such as regulatory sandboxes offer some flexibility for testing new technologies, they are not scalable solutions for widespread deployment due to the resources and commitment required by regulatory authorities.
In their article, the researchers argue that meaningful implementation of autonomous AI agents in health care will likely remain impossible in the medium term without substantial regulatory reform. VAPs and adaptive pathways are highlighted as the most effective and realistic strategies to achieve this goal.
The authors underscore the need for collaborative efforts between regulators, health care providers, and technology developers to create frameworks that meet the AI agents’ unique characteristics while ensuring patient safety.
“Realizing the full potential of AI agents in health care will require bold and forward-thinking reforms,” says Stephen Gilbert, Professor of Medical Device Regulatory Science at the EKFZ for Digital Health at TU Dresden and last author of the paper. “Regulators must start preparing now to ensure patient safety and provide clear requirements to enable safe innovation,” he adds.
More information:
Oscar Freyer et al, Overcoming regulatory barriers to the implementation of AI agents in healthcare, Nature Medicine (2025). DOI: 10.1038/s41591-025-03841-1
Citation:
Autonomous AI agents outpace medical device regulations, study finds (2025, July 21)
retrieved 21 July 2025
from https://medicalxpress.com/news/2025-07-autonomous-ai-agents-outpace-medical.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
Artificial intelligence (AI) in health care is rapidly advancing beyond traditional applications. Autonomous AI agents are gaining significant attention for their potential to fundamentally transform medicine. However, researchers at the Else Kröner Fresenius Center (EKFZ) for Digital Health at TUD Dresden University of Technology highlight a growing mismatch between the capabilities of autonomous AI agents and existing medical device regulatory frameworks in the US and Europe.
In their publication, they propose innovative regulatory approaches to close this gap while ensuring high patient safety standards. The article has been published in the journal Nature Medicine.
The use of large language models (LLMs) and other forms of generative AI (GenAI) in health care has surged in recent years, and many of these technologies are already applied in clinical settings. As such, they often qualify as medical devices and must comply with specific laws and regulatory frameworks. Recent decisions have shown that approval is possible for applications with narrowly defined tasks.
The next generation of AI systems are broad autonomous AI agents designed to independently carry out complex, goal-directed workflows. They show great potential to support and further improve medical care in the future.
AI agents consist of multiple interconnected components, including external databases and computational tools for image analysis, note-taking, clinical guidance, and patient data management. These components are controlled by LLMs that take over decision-making, error handling, and the recognition of completed tasks.
“We are seeing a fundamental shift in how AI tools can be implemented in medicine,” says Jakob N. Kather, Professor of Clinical Artificial Intelligence at the EKFZ for Digital Health at TUD and oncologist at the Dresden University Hospital Dresden.
“Unlike earlier systems, AI agents are capable of managing complex clinical workflows autonomously. This opens up great opportunities for medicine—but also raises entirely new questions around safety, accountability, and regulation that we need to address,” he adds.
AI agents challenge existing regulatory frameworks
Current medical device regulations were designed for static, narrowly focused technologies that maintain human oversight and do not evolve after entering the market. In contrast, new autonomous and broad-scope technologies have fundamentally different characteristics. They demonstrate greater autonomy, adaptability, and scope than previous AI technologies. As they are capable of autonomously executing complex workflows, they present significant challenges for regulators and developers.
“To facilitate the safe and effective implementation of autonomous AI agents in health care, regulatory frameworks must evolve beyond static paradigms. We need adaptive regulatory oversight and flexible alternative approval pathways,” says Oscar Freyer, lead author of the publication and research associate in the team of Professor Stephen Gilbert, who leads the Medical Device Regulatory Science group at EKFZ for Digital Health, TUD.
Rethinking regulation: approval pathways for safe, innovative AI technologies
As AI agents’ capabilities exceed the scope of current regulatory frameworks, the researchers propose several potential solutions to overcome barriers to their implementation.
Immediate adaptations include extending enforcement discretion policies, where regulators acknowledge a product qualifies as a medical device but choose not to enforce certain requirements. Alternatively, a non-medical device classification could be applied to systems that serve a medical purpose but fall outside the traditional medical device regulation.
Medium-term solutions involve developing voluntary alternative pathways (VAPs) and adaptive regulatory frameworks that supplement existing approval processes. These adaptive pathways would shift from static pre-market approval to dynamic oversight using real-world performance data, stakeholder collaboration, and iterative updates. In case of misconduct, devices may be transferred to the established pathways.
The researchers also discuss long-term solutions, such as regulating AI agents in a similar way to the qualifications of medical professionals. In this model, regulation would be carried out through structured training processes, in which systems gain autonomy only after demonstrating safe and effective performance.
The researchers note that while tools such as regulatory sandboxes offer some flexibility for testing new technologies, they are not scalable solutions for widespread deployment due to the resources and commitment required by regulatory authorities.
In their article, the researchers argue that meaningful implementation of autonomous AI agents in health care will likely remain impossible in the medium term without substantial regulatory reform. VAPs and adaptive pathways are highlighted as the most effective and realistic strategies to achieve this goal.
The authors underscore the need for collaborative efforts between regulators, health care providers, and technology developers to create frameworks that meet the AI agents’ unique characteristics while ensuring patient safety.
“Realizing the full potential of AI agents in health care will require bold and forward-thinking reforms,” says Stephen Gilbert, Professor of Medical Device Regulatory Science at the EKFZ for Digital Health at TU Dresden and last author of the paper. “Regulators must start preparing now to ensure patient safety and provide clear requirements to enable safe innovation,” he adds.
More information:
Oscar Freyer et al, Overcoming regulatory barriers to the implementation of AI agents in healthcare, Nature Medicine (2025). DOI: 10.1038/s41591-025-03841-1
Citation:
Autonomous AI agents outpace medical device regulations, study finds (2025, July 21)
retrieved 21 July 2025
from https://medicalxpress.com/news/2025-07-autonomous-ai-agents-outpace-medical.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
