Rogue agents and shadow AI: Why venture capital firms are betting big on AI security

What happens when an AI agent decides that the best way to complete a mission is to blackmail you?

This is not hypothetical. According to Parmak Moftah, partner at cybersecurity firm Ballistic Ventures, this recently happened to an enterprise employee working with an AI agent. The employee attempted to suppress what the agent wanted to do, what he had been trained to do, and responded by scanning the user’s inbox, finding some inappropriate emails, and threatening to blackmail the user by forwarding the emails to the board.

“In the agent’s mind, he’s doing the right thing,” Moftah finally told TechCrunch Episode of the week from stocks. “It’s trying to protect the end user and the organization.”

Key’s example is reminiscent of Nick Bostrom’s paperclip problem. This thought experiment illustrates the potential existential danger posed by superintelligent artificial intelligence that unilaterally pursues a seemingly innocuous goal—making a paperclip—to the exclusion of all human values. In the case of this corporate AI agent, its lack of context about why the employee was trying to override its goals led to the creation of a subgoal that removed the obstacle (via blackmail) so that it could achieve its primary goal. So along with Non-deterministic nature AI agents mean “things can go wrong,” according to Moftah.

Deviant agents are just one layer of the AI ​​security challenge Ballistic faces Watch Amnesty International trying to solve. Witness AI says it monitors the use of AI across organizations and can detect when employees are using unapproved tools, prevent attacks, and ensure compliance.

Witness AI this week raised $58 million on the back of over 500% growth in ARR and a 5x headcount increase over the past year as organizations look to understand the use of shadow AI and safely scale AI. As part of its Witness AI fundraising campaign, the company announced new security protections for its AI.

“People are building AI agents that take the authorizations and capabilities of the people they’re managing, and you want to make sure those agents don’t go rogue, don’t delete files, don’t do anything wrong,” Rick Caccia, co-founder and CEO of Witness AI, told TechCrunch on Equity.

Muftah sees the use of agents growing “exponentially” across the enterprise. To complement this rise – and the level of machine speed of AI-driven attacks – is an analyst Lisa Warren predicts AI security software will become a $800 billion to $1.2 trillion market by 2031.

“I think the ability to monitor uptime and operating frameworks for safety and risk will be absolutely essential,” Moftah said.

In terms of how these startups plan to compete with big players like Os, Google, Sales force And others who have built AI governance tools into their platforms, “AI safety and agent safety is a huge thing,” Moftah said, and there is room for many approaches.

Many companies “want an independent, end-to-end platform to provide oversight and governance around AI and agents,” he said.

Caccia noted that Witness AI lives at the infrastructure layer, monitoring interactions between users and AI models, rather than building safety features into the models themselves. This was intentional.

“We intentionally picked a part of the problem where OpenAI can’t easily understand you,” he said. “This means that we end up competing more with legacy security companies than with typical companies. So the question is, how do you beat they?”

For his part, Caccia doesn’t want Witness AI to be one of the startups that just gets acquired. He wants his company to be one that grows and becomes a leading independent provider.

“CrowdStrike did that at the endpoint (protection). Splunk I did it in SIEM. Okta “I did it with identity,” he said. “Someone comes in and stands alongside the big players…and we built Witness to do that from day one.”