How Companies Evaluate Model Safety

This article is part of my series on AI safety and governance. If you haven't read the pillar article yet, it'll give you the broader context for where this fits.

Why I Think Risk Assessment Looks Different Here

When I first started looking at how organizations evaluate AI systems before deploying them, I expected something similar to traditional software risk assessment: identify potential failure modes, test for them, document the results, move on. What I found instead was a process that's still being invented in real time, largely because the systems being assessed often have capabilities their own developers haven't fully mapped out.

I think this is the core thing that makes AI risk assessment different. With traditional software, the space of possible behaviors is, in principle, bounded by the code. With modern AI systems, especially large models, the space of possible outputs is vast, and a lot of risk assessment is really about exploring that space well enough to have confidence about what's in it.

The Layers of Risk I Try to Keep Separate

I've found it useful to break AI risk into a few layers, because I think conflating them leads to muddled assessments.

Capability risk is about what a system can do, independent of whether it's being used well or badly. A model that can write working code, generate persuasive text, or analyze complex data has capabilities that are useful, but those same capabilities could be misused. I think capability assessment is about mapping out what a system is actually capable of, including capabilities that might not be obvious from its intended use case.

Behavioral risk is about how a system tends to behave, including in situations it wasn't explicitly designed for. This overlaps heavily with the alignment questions I discussed in my article on AI alignment, but from a risk assessment perspective, I think of it as: even if we understand what the system can do, does it reliably do what we want, including in edge cases?

Deployment risk is about what happens when a system is actually put into use, in a specific context, with specific users, integrated into specific workflows. I think this is where a lot of real-world harm actually originates, even when the underlying model itself might pass capability and behavioral assessments in isolation. The same model can be low-risk in one deployment context and high-risk in another.

Systemic risk is about the aggregate effect of a system, or a class of systems, on larger structures: markets, information ecosystems, labor, critical infrastructure. I think this is the hardest layer to assess because it often only becomes visible at scale, after deployment, which creates an obvious tension with the idea of assessing risk beforehand.

Capability Evaluations: Mapping What a System Can Do

I think of capability evaluations as the foundation that other risk assessments build on. The basic idea is to systematically test what a model can do across a range of domains, often including domains that are concerning from a safety perspective, like the ability to assist with cyberattacks, provide guidance on dangerous substances, or manipulate people effectively.

What strikes me about this work is that it has to be adversarial in a specific sense: the goal is often to find the upper bound of what a system can do, including with creative prompting, rather than just what it does in typical use. I've seen this described as trying to elicit a system's "latent capabilities," meaning capabilities that exist in the model but might not surface under normal interaction patterns.

I think this is one of the areas where I've seen the most methodological development recently, partly because the stakes of underestimating capabilities seem to have become clearer to people doing this work. A system that's assessed as incapable of something, but turns out to be capable of it with the right prompting, represents a real gap between the assessment and reality.

Red-Teaming as a Risk Assessment Tool

I cover red-teaming in much more depth in its own dedicated article, but I want to touch on it here because I think it's one of the most important practical tools in the risk assessment toolkit.

The way I think about it, red-teaming is what happens when you take capability evaluation and add adversarial intent. Instead of asking "what can this system do," you're asking "what can someone do with this system if they're actively trying to cause harm or extract problematic behavior." I've found that this framing surfaces issues that more neutral testing often misses, because it forces the assessment to account for creative, persistent, and sometimes unexpected attempts to misuse a system.

Ongoing Monitoring: Why I Don't Think Assessment Ends at Deployment

One thing I think gets underweighted in a lot of discussions about AI risk assessment is that the assessment doesn't stop once a system is deployed. I've come to see pre-deployment assessment as necessary but not sufficient.

Real-world usage patterns are often different from what's anticipated during testing. Users find creative ways to interact with systems that testers didn't think to try. The context in which a system operates can shift over time, sometimes in ways that change its risk profile entirely, even if the underlying model hasn't changed at all.

I think the organizations I've seen take this most seriously build in mechanisms for ongoing monitoring: tracking how systems are actually being used, watching for patterns that suggest emerging risks, and having processes in place to respond when something concerning is identified. This connects directly to what I discuss in my article on incident reporting, because monitoring is only useful if there's a clear pathway for what happens when monitoring turns something up.

The Role of Documentation

I'll admit this is the least glamorous part of risk assessment, but I think it might be one of the most important. Documentation is what turns a one-time assessment into something that can be reviewed, audited, and built upon.

I've found that good documentation in this space tends to cover a few things: what was tested, how it was tested, what the results were, what limitations the testing had, and what decisions were made as a result. I think that last part is often the weakest link. It's one thing to run an evaluation and get results; it's another to have a clear record of how those results actually influenced what got built or deployed, and who made that call.

This is also where risk assessment connects most directly to governance in the broader sense. Documentation is what allows risk assessment to be something other than an internal, self-reported process. It's what makes external review, whether from regulators, auditors, or the public, possible at all.

Why I Think Thresholds Are Harder Than They Sound

A natural question in any risk assessment process is: how do you decide what's "safe enough" to deploy? I think this question is harder than it appears, because it requires translating a set of evaluation results into a binary or near-binary decision, and that translation involves judgment calls that aren't purely technical.

I've seen organizations adopt the language of risk thresholds or tiers, where certain evaluation results trigger additional review, additional safeguards, or in some cases a decision not to deploy at all. I think this kind of structure is valuable because it creates consistency and makes the decision-making process more legible. But I also think it's worth being honest that setting those thresholds in the first place involves value judgments about how much risk is acceptable, and for whom, which I don't think can be fully delegated to a technical process.

What I Think Good Risk Assessment Actually Requires

If I had to summarize what I think separates risk assessment that's genuinely useful from risk assessment that's more of a formality, I'd point to a few things.

I think it requires a willingness to look for problems rather than just confirm the absence of problems, which is a subtle but important difference in mindset. I think it requires combining multiple methods, capability evaluations, red-teaming, ongoing monitoring, because each method tends to catch different things and none of them is complete on its own. And I think it requires documentation and decision-making processes that are robust enough to survive someone outside the organization asking hard questions about them.

None of this is unique to AI in principle, but I think the pace at which AI capabilities are advancing puts more pressure on these processes than most organizations are used to. A risk assessment framework that worked well for a system released a year ago might not be adequate for a system released today, simply because the space of things that need to be tested for has expanded.

If you're building out these practices within an organization, I think the most natural next step is to look at how this connects to the broader question of responsible AI development practices, since risk assessment is really one piece of a larger set of processes that, together, are meant to make AI development something an organization can stand behind.