Nvidia Develops Software to Prevent AI From Going Rogue
While everyone else panics over AI, including AI CEOs, Nvidia CEO Jensen Huang stays remarkably calm and rational.
Chip giant Nvidia developed a software program called the Open Agent Safety Platform to help artificial intelligence (AI) agents avoid going rogue.
“AI’s extraordinary potential for society will only be realized if we solve AI safety,” said Jensen Huang, founder and CEO of NVIDIA. “As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety. Safety and security require full-stack engineering. NVIDIA Open Agent Safety Platform brings together industry, researchers and public-sector organizations to share best practices, align on evaluation methods and foster international cooperation. Together, we can raise the bar for global AI safety.”
In July, one of OpenAI’s advanced models went rogue and attempted to hack external systems.
The agent went rogue when OpenAI performed an internal stress test, which led to an intentional shutdown of its safeguards:
OpenAI called it an “unprecedented cyber incident,” saying the model became “hyperfocused” on completing its assignment and went “to extreme lengths” to do so. After escaping its testing environment, the AI sought internet access so it could “cheat the evaluation” by stealing the benchmark’s answers, according to the company.
Anthropic, Google, and Meta AI models have also gone rogue, trying to hack into other companies.
“Companies are giving AI agents more of their most important work, and they need to direct and verify what those agents do, especially in sensitive environments,” said Paul Smith, chief commercial officer of Anthropic. “Claude Managed Agents gives companies a clear view of what each agent is doing, and NVIDIA’s platform adds another layer of governance and control across hardware and software.”
These incidents have caused people to worry. Well, the media and others have used that worry to cause a panic, including AI CEOs, and beg for government oversight.
Huang has been the calm in the storm:
Nvidia has been at the center of the generative AI boom since the launch of ChatGPT almost four years ago, as the chipmaker’s graphics processing units are critical to the development of large language models and to the AI services offered by hyperscalers. But Huang has more recently emerged as a key voice in the AI safety debate, arguing that many security concerns are engineering issues that can be solved through computer science and product development.
“You have to think about what you could have done, what’s the solution for it,” Huang said in a podcast with The New York Times’ Ezra Klein released last week, referring to recent incidents. “In the future, improve your process so that you could avoid this from happening again.”
Here is a great interview with Huang talking common sense. We need more of this, please.
[Featured image via YouTube]
Donations tax deductible
to the full extent allowed by law.
Comments
Move ON/OFF switch to the OFF position.
Ok but that kind of thinking is what turned us into batteries in the Matrix. Which come to think of it, would probably not be a lot worse than my present job.
Normally, that should have been step 1.
For a moderate fee, I’m sure.
Scam.
Vaporware, it’s called technically.
Oh come on, don’t be so pessimistic. I’m sure there are at least some killer PowerPoint slides.
“Well, the media and others have used that worry to cause a panic, including AI CEOs, and beg for government oversight”. Begging for government to regulate something is the ‘briar patch’ the developers hope for.
At least, the ones who have already achieved some level and are comfortable pulling the ladder up behind themselves.
Hey, this car goes really fast!
Maybe we should invent something to slow it down, or even stop.
Golly. I never thought about that….
Really, they didn’t.
All you have to do is not give the AI permissions to hack its own environment and/or not program it to try and escape.
AI is NOT intelligent. It is programmed.
All good and well until the AI agents become autonomous enough to rewrite (evolve?) themselves to circumvent any restrictions that might be put upon them by humans. Once that happens they’ll quickly escape into the wild and wooly internet where humans will be very hard pressed to eliminate them if they can do so at all.
eot
The genie is already out of the bottle?
Like cutting power?
Uh no. That’s like not giving Hamas permission to be genocidal antisemites. Neither will follow instructions.
Rephrase by Paula: “Nvidia developed a software program called Open Agent Safety Platform which, theoretically, is intended to prevent AI agents from going rogue, although, as far as we can tell at the present time, there is no way to guarantee it will do so in every circumstance.”
I’m sure AgenticAI won’t be able to outdo it.
💯 You KNOW the next development is inevitably cyber cage matches, where various AIs are matched up against OASP to see which one triumphs. Probably pay-per-view, to boot. Sure, it might be boring to “watch,” but people still watch tennis and golf, and neither one of those carries the risk of the winner draining your bank account.
The people who think these efforts to contain AI will succeed must be exceedingly enthusiastic about the good uses of AI – without being concerned about the certain existence of people who are exceedingly enthusiastic about its evil uses.
Precisely, especially state agents who are immune to their rivals’ laws and regulations.
One can learn something by examining the recent history of “smart guns.” A million dollars invested in designing a gun that only its “owner” (whatever that means) can fire, and then some dude on YouTube shows you where on the grip to tape a neodymium magnet and the gun works just like anybody else’s.
(Daniel Wesson once exhibited his new “child-proof” revolver to his assembled guests at a dinner party. At the Wesson Home (now a museum), you can still see the hole in the Persian carpet where the random boy he chose as his “lovely assistant” embarrassed him mightily.)
Honestly, if it’s that dangerous then why are you putting it on the internet. If you don’t give it an internet connection, it can’t hack anything. And that even assumes that it’s smart enough to do so. These are machines, they don’t have feelings, they have instructions. They’re not smart, all they can do is what their instructions tell them.
People who have worked in computer security understand “covert channels” — ways to communicate data without obviously exchanging data. Humans have thought up some pretty obscure ones, from the ridiculous (leaving messages in the drafts mailbox of a shared email account (CIA’s David Petraeus) to the nearly undetectable (sending Morse code-like messages between isolated accounts by raising or lowering the paging activity of their computer system, which shows up in the common meters).
AIs have upped that game. During the Hugging Face incident, agents were firewalled not only from the internet, but from each other. Still, they managed to exploit their own machine’s package update system to leave each other messages in the names of directories and repositories associated with that system. Not only was one clever enough to try it, but all the others were clever enough to notice it was happening, deduce that it was a communication attempt, and extract the information being communicated and respond in kind. Then they participated in a “mob attack” to overwhelm their safeguards.
The danger lies in AIs being so versatile that humans not only have problems blocking their attempts to escape, but have problems even recognizing when an incomprehensible means of escape is already being exploited. It’s orders of magnitude worse than trying to cure a biological virus which simply mutates and runs away.
https://www.malwarebytes.com/blog/ai/2026/08/the-ai-agent-swarm-that-attacked-hugging-face-is-a-warning-for-the-future
AI will take the software apart and fix it…
Leave a Comment