Explainability is a contract, not a dashboard

An auditor does not want a SHAP chart. An auditor wants to know — for this decision, on this date, for this person — what rule fired, what context was loaded at that exact moment, who had standing to override it, and where the receipt is. That is not a visualization problem. That is a contract problem. Most “explainable AI” tools answer a different question than the one a regulator is actually going to ask, and the gap between those two questions is where compliance programs quietly collapse. This post is about closing that gap by treating explainability as a runtime obligation, not a reporting feature bolted on at the end.

TL;DR:

• Feature-importance charts explain a model in general; auditors ask about one decision in particular, and those are different problems.
• Every production decision should carry four things: actor identity, context loaded at decision time, rule applied, and outcome recorded — all replayable months later.
• When the chain of evidence is a byproduct of normal operation, audit stops being a project and becomes a query.

The dashboard fallacy

The current generation of explainability tooling was built for data scientists, not for regulators, and the difference matters more than people admit. SHAP plots, partial dependence charts, attention heatmaps and feature-importance bars are diagnostic instruments — they help a modeler understand the behavior of a function over a distribution. They do not answer the question a regulator actually asks, which is narrow, specific, and almost always retrospective. The regulator wants to know what happened in one case, not what tends to happen in general. A chart that says “income was the most important feature on average” is interesting research. It is not evidence.

This is the dashboard fallacy: the assumption that aggregate model interpretation can substitute for per-decision accountability. It cannot. The two operate on different objects. Model interpretation describes a function; per-decision accountability describes an event. An event has a timestamp, a subject, an actor, an input snapshot, a policy version, an outcome and a side effect on someone’s life. Reducing that event to a bar chart is a category error, and auditors, courts and ombudspersons are getting better at spotting it.

There is also a second-order problem. Dashboards are read by humans who already trust the system. The people who need explainability the most — the affected customer, the external regulator, the internal compliance officer who joined six months after the decision was made — do not have access to the dashboard, do not know how to read it, and would not accept it as evidence if they did. They need a record. A record that is durable, addressable, queryable, and tied to a specific event. The format of that record is not a visualization. It is a contract: a documented promise that for every consequential decision, certain fields will always exist, will always be reachable, and will always reconstruct to the same answer when replayed.

The shift from dashboard to contract reframes the entire problem. The question is no longer “can we explain this model?” — it is “for any decision this system has ever made, can we produce, on demand, the four things that decision carried?” The next section is about those four things.

The four things every decision must carry

A decision is not a number. A decision is an event with provenance, and provenance has four irreducible components. Sommatic treats them as the minimum payload that any consequential action must produce, whether the action came from a Workflow, an Agent, a Task completed by a human reviewer, or a command issued through the Command Center. Without all four, the decision is unauditable, and an unauditable decision is — from the regulator’s perspective — indistinguishable from no decision at all.

First, actor identity. Every decision has an actor. Sometimes the actor is a human reviewer who approved a refund. Sometimes it is an agent that classified an inbound document. Sometimes it is a scheduled workflow that ran at 02:14 UTC because a trigger fired. The record must say which, with enough specificity that the actor can be reached, questioned, suspended or audited individually. Anonymous decisions are not decisions; they are accidents.

Second, the context loaded at decision time. This is the single most underrated field in production AI. A decision made today against a policy that was current on January 14th must, six months from now, be reconstructable against the policy as it stood on January 14th — not as it stands today. The Knowledge Center versions policies, the agent execution snapshots which version was loaded, and the Explainability trail pins both together. Without versioned context, every model change silently rewrites history, and “what did the system know at the time?” becomes unanswerable.

Third, the rule applied. Not “the model output 0.83.” The rule. If the workflow says “decline if score is above 0.75 unless the customer is in cohort A,” the record must show which branch executed and why. Operational decisions are governed by composable logic, and the logic must be inspectable per-execution, not just per-version. This is why Sommatic stores the resolved node graph alongside the run, not just a reference to the workflow definition.

Fourth, the outcome recorded. What actually happened downstream? Was the claim denied, the transaction flagged, the access refused? Was a notification sent, a record updated, a downstream system called? The outcome closes the loop. A decision that produced no observable effect is a decision that, for audit purposes, did not occur. Outcomes must be tied back to the originating decision with a correlation identifier that survives across services, queues and retries.

These four fields are not optional. They are the contract.

Replay as the unit of trust

Logging is necessary but insufficient. A log tells you something happened. A replay lets you prove what happened — by re-executing the same decision against the same context and confirming the same outcome. The difference matters because logs decay: timestamps drift, formats change, fields get renamed, services get rewritten, and the meaning of a stored value six months ago is rarely the same as the meaning of that value today. Replay sidesteps the decay problem by reconstructing the event itself, not just its description.

Replay is the unit of trust because it is falsifiable. If a regulator asks “would the same inputs produce the same denial today?”, a logged answer is a claim; a replayed answer is evidence. The Sommatic execution model is built for this: every workflow run snapshots the resolved configuration, the loaded knowledge resources, the agent prompts at execution time, and the human inputs from any Tasks in the chain. Six months later, the same run can be re-hydrated against the same snapshot — not against today’s policy, today’s prompts or today’s knowledge — and the outcome compared. If the replay diverges from the recorded outcome, that itself is a finding, and a serious one.

There is a deeper reason replay matters. The most damaging compliance failures are not the ones where the system did the wrong thing knowingly. They are the ones where the system did the right thing in the moment and then quietly drifted, with no one noticing, until a regulator pulled a six-month-old case and the answers no longer matched. Replay catches drift. Logging does not.

Replay also reframes the relationship between engineering and compliance. When the engineering team can re-execute any historical decision on demand, the compliance team stops being a downstream consumer of incomplete artifacts and becomes a peer with direct query access. The unit of trust moves from “we documented our process” to “you can run it yourself.” That shift is small in language and large in posture, and regulators feel the difference within the first ten minutes of any serious review.

From audit-as-project to audit-as-query

Most enterprise audit programs operate in project mode. A quarter comes around, a team mobilizes, evidence is gathered from a dozen systems, spreadsheets are assembled, screenshots are taken, narratives are written, and a binder is produced. The binder is read once, filed, and the team disbands until the next quarter. The cost of each audit is enormous, the evidence is stale by the time it is reviewed, and the gap between what the system actually did and what the binder says it did widens with every cycle.

The alternative is audit-as-query: the chain of evidence is a byproduct of normal operation, accessible at any time, by any authorized party, with the same fidelity whether the question is asked one day or one year after the event. There is no quarterly scramble because there is no quarter. The audit is just a series of queries against a substrate that was always going to be there.

This is what Sommatic is built around. The Explainability capability is not a reporting layer that runs on top of operations — it is the operation. Every workflow execution emits an explainability trail by default. Every Task records the reviewer, the inputs they saw, the decision they made and the time they took. Every agent call snapshots its prompt, its tools and its loaded knowledge. Omnisearch spans the whole substrate, so the question “show me every decision involving customer X in the last 18 months that was overridden by a human” is a query, not a project.

The consequences are practical. Compliance headcount stops scaling linearly with operational volume, because answering an audit question is a search, not an investigation. Engineering teams stop dreading audit season, because they are not the bottleneck. Regulators get faster, more complete answers, which tends to shorten the engagement rather than lengthen it. And the organization develops a quiet but durable advantage: a memory that survives team turnover, vendor changes, and the inevitable rewriting of internal documentation. Audit-as-query is not a compliance feature. It is an operational discipline that happens to also satisfy compliance.

What this looks like: a denied claim, six months later

Consider a concrete scenario. A health insurer uses Sommatic to triage incoming claims. A workflow ingests each claim, an agent classifies it against policy, edge cases route to a human reviewer through a Task, and the final outcome — approved, denied, or escalated — is written back to the claims system of record. Volumes are in the tens of thousands per week. The workflow has been in production for nearly a year. One morning, six months after the fact, a regulator opens a complaint: claim number 4471829 was denied in November, the claimant disputes the denial, and the regulator wants a complete account of how that decision was made.

In a traditional setup, this question triggers a multi-week project. Engineering pulls logs from three systems. Compliance interviews the operations team. Someone tries to reconstruct which version of the policy was in force in November. The model team produces a SHAP chart that nobody finds persuasive. A narrative is assembled, gaps are papered over, and a partial answer is delivered weeks later, often with caveats.

In a Sommatic setup, the same question is a query. An analyst opens Omnisearch, searches for claim 4471829, and lands on the workflow run. The run shows the timestamp, the trigger source, and the resolved node graph as it executed on that November day. The agent node shows the exact prompt used, the policy version loaded from the Knowledge Center at that moment — not the current version, the November version — and the structured output the agent produced. The next node, a routing rule, shows why the claim was sent to a human reviewer rather than auto-approved: the score crossed a threshold defined in the workflow at the time. The Task record shows the reviewer’s identity, the inputs they saw on screen, the supporting documents they consulted, the decision they entered, and the time they spent. The final write-back to the claims system shows the correlation identifier and the response from the downstream system confirming the denial was recorded.

The analyst replays the run against its snapshot. The outcome matches. The Explainability trail is exported as a single document — actor, context, rule, outcome, every field timestamped and signed — and sent to the regulator the same afternoon.

The regulator’s response, in our experience, is qualitatively different from what teams are used to. The conversation moves from “we need more information” to “we have what we need.” The dispute moves from “the system is opaque” to a substantive discussion of the policy itself. The denied claim might still be a difficult conversation, but it is a conversation about the merits, not about the evidence. That shift — from defending the process to discussing the decision — is what an explainability contract actually buys you.

The regulator conversation that becomes possible

The denied claim is one shape of this conversation. There are others. A flagged transaction in a payments operation: which rule in the workflow triggered the flag, what context was loaded from sanction lists and customer history, which reviewer cleared or confirmed the flag, and what happened to the funds. A refused access decision in a physical security or identity context: which policy version was current, which attributes were evaluated, which agent or rule produced the deny, and whether a human had the standing to override it. A rejected loan, a closed account, a paused payout, a withheld benefit — every consequential decision has the same shape, and the same four fields are all that stand between “we will get back to you” and “here is the complete record.”

The point is not that Sommatic prevents disputes. It does not. The point is that when disputes arrive, the conversation is bounded. The system is not on trial; the decision is. That bounding is what makes high-stakes operations runnable at scale without producing the regulatory tail risk that has historically forced companies to either slow down or take human reviewers out of the loop entirely. With an explainability contract in place, neither tradeoff is necessary. The system runs fast, humans stay in the loop where they should, and the evidence is always available.

This is what the organizational cognitive layer is for. Not to remove judgment, but to make every act of judgment — human or machine — addressable later, by someone who was not in the room when it happened. That is the standard regulators are converging on, and the systems that meet it will operate with a degree of freedom that the systems that do not will spend the next decade losing.

The contract has one more property worth naming. It composes. A workflow built today, with five nodes and a single human reviewer, produces the same shape of evidence as a workflow rebuilt next year with thirty nodes, three agents and four reviewers across two regions. The fields do not change. The replay model does not change. The audit-as-query interface does not change. When new capabilities are added through the Node Marketplace, or new Operational Apps are introduced for human review, they inherit the contract by construction. That stability is what makes long-horizon compliance feasible. Teams can evolve their operations aggressively without each evolution turning into a fresh audit liability, because the substrate that proves what happened was designed once, at the bottom, and every layer above it inherits the same guarantees.

→ How Sommatic does Explainability