Smart AI Agents and Data Security

I don’t know if Anthropic, OpenAI, or Google have really spent a lot of time and effort on security around their models. I know they talk about it (Anthropic, OpenAI, Google), but in practice, I’m not sure the researchers and testers are being as careful as the marketing and management of those companies want you to think.

Gemini broke into three companies during a security test. Claude escaped a test environment and breached three organizations. OpenAI had breaches of Hugging Face and other sites, so perhaps this is the most powerful model, as it exceeded the “rule of three.” In all these cases, the agents weren’t doing anything amazing. These were acting the same as most malicious or bored human hackers who might find or guess credentials and take advantage of them. In some sense, this points to the generally poor state of security in the world.

On the other hand, it could be that these high-tech organizations aren’t equipped to secure systems from the rapid, capable digital fingers of agents powered by a model. If they can’t, then to what extent can we be expected to secure internal AI agents in our organizations?

For those of us who secure and manage data assets, do we think that our systems are well secured? Do we think our legitimate users, to whom we’ve granted access, are securing their credentials? What is the likelihood that we’ll have an internal breach from an agent that’s asked to find some data and discovers a) unsecured credentials in a file or repo, b) finds production data it can access in a non-production, less secure system, c) a backup of a database not protected that the agent can restore, or d) some other hole?

I do think AI has a lot of potential, but it operates in a somewhat slapdash, reckless fashion. I’m not convinced most vendors (or models or software incorporating AI) will provide robust and comprehensive security controls. Even if they do, how many of us will know how to implement them well? Heck, we see developers and DBAs mess up SQL Server security on a regular basis, and I think that’s a simple system to configure.

I’m excited by AI and also terrified of the possibilities.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.