Explainability and Transparency in AI Systems

This article is part of my series on AI safety and governance. If you're new here, the pillar article lays out how this topic fits into the bigger picture.

Why This Topic Sits Close to My Heart

Of everything I've written in this series, I think explainability is the topic where I feel the gap between "what we can do" and "what people expect" most acutely. People interact with AI systems that produce confident, fluent outputs, and I think there's a natural human assumption that something producing such coherent responses must also be able to explain itself in the same way a person would. I've come to believe that assumption is often wrong, and I think that gap is at the root of a lot of the trust issues this field grapples with.

Separating Explainability from Transparency

I find it useful to draw a line between these two terms, even though I see them used interchangeably a lot.

Explainability, as I think about it, is about whether we can understand why a specific system produced a specific output. If a model denies someone a loan, or flags a piece of content, or makes a particular recommendation, explainability is about whether there's a meaningful answer to "why did it do that, in this case."

Transparency is broader. It's about whether the people affected by a system, or overseeing it, have visibility into how the system works in general: what it was trained on, what its known limitations are, how it was tested, what safeguards exist. I think of transparency as the conditions under which explainability becomes possible and meaningful, rather than explainability itself.

I think both matter, but I've noticed they tend to get conflated in ways that aren't helpful. A system can be transparent about its general design while still being very hard to explain in any individual case. And a system can produce plausible-sounding explanations for individual outputs that don't actually reflect transparency about how it really works.

Why I Think This Is Technically Hard

I want to be honest about why explainability is difficult, because I think a lot of public discussion treats it as though it's mostly a matter of corporate willingness to disclose information, when I think there's a real technical problem underneath that too.

Modern AI systems, particularly large neural networks, don't operate in a way that maps cleanly onto human-readable reasoning. The internal representations a model uses to arrive at an output are mathematical structures distributed across enormous numbers of parameters, and I don't think there's currently a reliable way to translate those structures into the kind of step-by-step reasoning a person would recognize as an "explanation," at least not one that's guaranteed to be accurate.

This creates a problem I find genuinely uncomfortable: if you ask a model to explain its own reasoning, what you get back is itself a generated output, produced by the same kind of process that produced the original answer. It might be a reasonably good description of what influenced the output. It might also be a plausible-sounding story that doesn't actually correspond to what happened internally. I don't think there's always a way to tell the difference from the outside, which is part of why I see interpretability research as distinct from, but closely related to, explainability.

Interpretability Research: Looking Inside

I think of interpretability research as the more technically ambitious cousin of explainability. Rather than asking a system to explain itself, interpretability research tries to directly study what's happening inside a model: which internal structures correspond to which concepts, how information flows through the system, and what's actually driving a given output.

I find this work genuinely exciting, because I think it represents one of the few paths toward explanations that are verifiably accurate, rather than just plausible. But I also think it's important to be realistic about where this research stands. My understanding is that it's made real progress on understanding pieces of how these systems work, but a full, reliable account of why a large model produces any given output, the kind of account you could confidently present in, say, a legal or regulatory context, isn't something I think exists yet for the most capable systems.

Why I Think This Matters for Governance, Not Just Engineering

I think it would be a mistake to treat explainability as purely a technical research problem, separate from the governance conversations I cover elsewhere in this series. The reason is that a lot of governance frameworks, including some of the regulatory approaches I discuss in my piece on global governance frameworks, include requirements around transparency and, in some cases, explanation.

But I think there's a tension here that doesn't get acknowledged often enough. If a regulation requires that an AI system's decisions be explainable, and the current state of the technology doesn't allow for explanations that are both human-readable and verifiably accurate, what does compliance actually look like? I've seen a few different responses to this tension.

One response is to require explanations that are useful and meaningful, even if they're not complete accounts of the model's internal processing. I think this is roughly the approach embedded in a lot of existing transparency requirements: not "explain exactly how the model arrived at this," but "provide enough information that an affected person can understand, contest, or seek review of a decision."

Another response is to restrict the use of less interpretable systems in certain high-stakes contexts altogether, on the theory that if you can't adequately explain a decision, you shouldn't be using that kind of system to make it. I think this is part of the logic behind some of the high-risk categorizations I discussed in my governance frameworks piece.

A third response, which I find most honest but also most uncomfortable, is to acknowledge that we're operating with an explainability gap and to build other safeguards, like human review, audit trails, and the ability to appeal decisions, that don't depend on the system itself being fully explainable.

What I Think "Good" Transparency Looks Like in Practice

Setting aside the harder open questions about explainability at the model level, I think there's a lot that can be done on the transparency side that doesn't require solving interpretability first.

I think documentation about what a system was trained on, what its intended uses are, what its known limitations are, and what testing it underwent is valuable transparency, even if it doesn't explain any individual output. I've found that this kind of documentation is often what makes the difference between an organization that can answer hard questions about its systems and one that can't.

I also think clear communication to users about when they're interacting with an AI system, and what that system's role is in a larger decision-making process, is a meaningful form of transparency. If someone knows a decision involved an AI system, and knows there's a human reviewing or able to override that decision, I think that changes the stakes of the explainability gap, even if it doesn't close it.

And I think being honest about uncertainty matters more than people sometimes give it credit for. A system that can communicate "I'm not confident about this" is, in a sense, providing a kind of transparency that a system which always sounds equally confident isn't, even if neither system can fully explain its reasoning.

Where I Land on This

I don't think explainability and transparency are problems that get "solved" in the way a bug gets fixed. I think of them more as an ongoing relationship between what the technology can currently support and what people reasonably need in order to trust and contest decisions that affect them.

What I hope comes through in this piece is that I don't think the answer is to wait for interpretability research to fully mature before taking transparency seriously. There's a lot that can be done now, through documentation, disclosure, human oversight, and honest communication about limitations, that doesn't depend on first solving the harder technical problem. But I also don't think it's honest to pretend that documentation and disclosure are a substitute for genuine explainability when the stakes are high enough. I think both tracks need to keep moving, and I'd rather be upfront about the gap between them than pretend it's smaller than it is.

If this topic interests you, I think it pairs naturally with my article on bias and fairness, since a lot of the work involved in identifying and addressing bias depends on the same kind of visibility into how systems actually behave that I've been describing here.