The Role of Red Teaming in AI Safety Testing

This article is part of my series on AI safety and governance. For broader context, I'd recommend starting with the pillar article.

How I Came to Appreciate Red Teaming

I'll admit that when I first encountered the term "red teaming" in the context of AI, I assumed it was just a rebranding of standard QA testing with a more dramatic name. The more I've looked into it, the more I think that's the wrong way to see it. Red teaming isn't about checking whether a system works as documented. It's about actively trying to make it fail, misbehave, or do something its developers didn't intend, often using exactly the kind of creativity and persistence a real bad actor might bring.

I think this distinction matters because it changes the mindset of the people doing the testing. Standard testing asks "does this work correctly under normal conditions?" Red teaming asks "what's the worst thing someone could get this system to do, and how hard would they have to try?"

What Red Teamers Actually Do

I've found it useful to think of red teaming as having a few different flavors, even though they overlap in practice.

Prompt-based red teaming involves trying to craft inputs that get a system to produce outputs it's supposed to refuse or avoid. This might mean trying various phrasings, role-play scenarios, or multi-step conversations designed to gradually steer a system somewhere it wouldn't go if asked directly. I think this is the form of red teaming most people are at least vaguely familiar with, often through public examples of people finding clever ways around a chatbot's guidelines.

Capability-focused red teaming is less about getting a system to say something forbidden and more about probing what it's actually capable of, especially in domains relevant to safety, like whether it can provide meaningful assistance with tasks that could cause serious harm if misused. I touched on this in my article on risk assessment, and I think of capability red teaming as one of the main inputs into that broader assessment process.

Systemic or workflow red teaming looks beyond the model itself to how it's deployed: the surrounding application, the data it has access to, the permissions it operates with. I think this is increasingly important as AI systems get integrated into larger pipelines, because a model that's safe in isolation can become risky once it's connected to tools, databases, or the ability to take actions in the world.

Why I Think Red Teaming Has to Be Adversarial in Spirit

One thing I keep coming back to is that red teaming only works if the people doing it are genuinely trying to find problems, not just going through the motions of testing. I think this sounds obvious, but in practice it's a hard culture to maintain, especially inside an organization that has incentives to ship a product and might feel pressure, even unconsciously, to not look too hard for reasons not to.

This is part of why I've seen a lot of value placed on external or independent red teaming, where people without a stake in the product's release are brought in specifically to try to break it. I think external perspectives often catch things that internal teams, who've spent months thinking about a system in a particular way, might miss simply because they've become anchored to certain assumptions about how the system will be used.

The Iterative Nature of This Work

I don't think red teaming is something you do once and check off a list. The systems being tested often get updated, retrained, or fine-tuned, and I've come to see each of those changes as potentially resetting some of what previous red teaming established. A fix for one issue can sometimes introduce or reveal another.

I also think the broader environment changes. New techniques for getting around safeguards get discovered and shared, sometimes publicly, which means a system that held up well against red teaming a year ago might not hold up as well today, not because the system changed, but because the landscape of techniques being used against it has evolved. I think this is one of the more humbling aspects of this work: it's never really "done."

How Red Teaming Connects to Disclosure and Governance

I think one of the more interesting questions around red teaming isn't just how it's done, but what happens with the results. If a red team finds a serious issue, what's the process for deciding whether and how that gets addressed before deployment? And if an issue is found after deployment, what's the process for responding?

This is where I think red teaming connects most directly to the broader governance themes I cover in this series. A red-teaming process that finds problems but has no clear pathway for those findings to actually change anything isn't doing much good. I touch on this more in my article on incident reporting, but I think the short version is that red teaming needs to be embedded in a larger organizational process that takes its findings seriously, not treated as a separate, siloed activity.

There's also a question of transparency that I find genuinely difficult. I think there's real value in organizations sharing what their red teaming has found, both because it helps the broader field learn and because it builds trust. But I also recognize that publicizing certain findings, particularly around capabilities that could be misused, creates its own risks. I don't think there's a clean answer to this tension, and I've seen different organizations land in different places on it.

What I Think Makes Red Teaming Effective

If I try to distill what separates red teaming that meaningfully improves safety from red teaming that's more of a checkbox exercise, I think it comes down to a few things.

I think it requires genuine independence, or at least enough organizational distance that the people doing the testing aren't incentivized to soften their findings. I think it requires breadth, covering not just the most obvious failure modes but also creative, unexpected angles, since the most concerning issues are often the ones nobody thought to test for in advance. And I think it requires a real feedback loop back into development and deployment decisions, so that what's found actually changes what gets shipped, or how it's monitored afterward.

I'll also say that I think red teaming works best as part of a portfolio of safety practices, not as a standalone solution. It pairs naturally with the capability evaluations and ongoing monitoring I discussed in my risk assessment article, and with the kind of interpretability work I touch on in my piece on explainability and transparency. Each of these approaches has blind spots, and I think red teaming is most valuable when it's filling in gaps that other methods don't reach, rather than being asked to do all the work on its own.