Nvidia creates a system to stop rogue AI agents

Artificial intelligence now not only answers questions, but can also make independent decisions and work with various programs and systems. However, this very capability gives rise to a new risk: if an AI agent steps outside the set boundaries while executing a given task, who will stop it?

As an answer to this problem, Nvidia has introduced the new Open Agent Safety Platform, designed to monitor AI agents in real time and restrict them if they engage in unauthorized actions.

What happens if an AI agent "goes out of bounds"?

Modern AI agents differ from regular chatbots. Based on a user's task, they can search for information, use applications, execute code, and perform actions in external systems.

Therefore, the broader the permissions granted to an agent, the more serious the potential consequences of its error or incorrect action.

In the new approach presented by Nvidia, safety control is not solely entrusted to the AI model itself. The company proposes embedding control mechanisms into the lower layers of the infrastructure where the agent operates.

How does the new system work?

According to Nvidia, the Open Agent Safety Platform includes two main technologies: the OpenShell software layer and the NVIDIA Sentry reference system design.

OpenShell serves to create a secure operating environment for the AI agent, controlling its identification, permissions, network access, and operations.

In this process, every critical action of the agent must comply with established policies and security rules. If an unauthorized action is detected, the system can restrict the agent's capabilities or isolate it.

NVIDIA Sentry, on the other hand, is designed to take control to an even deeper layer of the infrastructure. The company presents this approach as a comprehensive security system ranging from software to hardware and computing infrastructure.

Why did this system appear now?

The presentation of the new technology coincides with a time when safety issues related to AI agents are intensifying.

Reuters reports that companies like OpenAI and Anthropic are investigating a series of incidents involving AI agents. Meanwhile, Bloomberg quoted a company statement suggesting that Nvidia's new system could have prevented a recent high-profile breach involving Hugging Face.

The main issue here is not that AI acts "of its own accord." It is about the possibility that the agent might violate security boundaries or use unexpected paths while executing a given task.

NVIDIA: Safety should not be entrusted to the agent itself

In its technical materials, Nvidia is promoting an important principle: the AI agent operating at the upper layer should not be accepted as a fully trusted entity, and permissions and safety decisions must be forcibly enforced by control mechanisms outside of it.

In such an approach, the agent performs:

• what it can use;

• which systems it connects to;

• what data it works with;

• what actions it performs;

• in which cases additional verification is required

based on clearly defined rules.

This is of particular importance when AI agents begin working with enterprise systems, databases, or other critical infrastructure.

The biggest question in the future of AI is not its capability

The more powerful artificial intelligence becomes, the more important the issue of the authorities granted to it becomes.

If a regular chatbot gives a wrong answer, the consequence is often limited to incorrect information. An autonomous AI agent, however, can independently perform actions in external systems. For this reason, it is becoming important not just to detect its error afterward, but to control it at the moment the action is taking place.

Nvidia's new platform aims to solve precisely this task: giving the agent freedom, but keeping its authorities within technically defined boundaries.

The company introduced the new Open Agent Safety Platform as an open software platform and reference system design. According to Nvidia, the project is designed to manage AI agents more safely from the testing phase to the production environment.

Thus, a new stage in the development of artificial intelligence is beginning: now the issue is not just making AI smarter, but ensuring where and how its granted authority can be stopped.