Designing for HITL: the contract between automation and human judgment

Most workflows do not fail at the automated steps. They fail at the seams — the moment a rule cannot decide, the evidence is ambiguous, the case is rare, or the policy was written for a world that no longer exists. Teams instrument the happy path with care and leave the unhappy path as an afterthought: a Slack ping, an email to a shared inbox, a ticket nobody owns. The interesting question is not whether to automate. The interesting question is where the human earns their seat in the run, what they see when they sit down, and how the workflow knows they finished.

TL;DR

• Pure automation breaks at three predictable seams: high-impact actions, ambiguous evidence, and novel situations. Design for those before you design the happy path.
• A task in Sommatic is a contract — assignment, queue, priority, SLA, evidence, validated output — not a notification and not a ticket.
• Done well, HITL lowers cost per case and raises trust at the same time, because humans only see the cases that need a human.

Where pure automation breaks

Every team that builds workflows eventually hits the same wall. The first ninety percent of cases run clean. The last ten percent consume most of the operational budget, generate most of the customer complaints, and produce most of the postmortems. Those cases are not random. They cluster into three classes of decisions that should never auto-resolve, and recognizing the class is the first step in deciding where to place a human.

First, high-impact decisions. A refund of fifteen dollars and a refund of fifteen thousand dollars are not the same operation, even if the workflow looks identical. The blast radius of getting it wrong is what makes the second one a different problem. High-impact decisions are where the cost of a false positive (auto-approving fraud) and a false negative (denying a legitimate case and losing the customer) are both meaningful, and where the organization wants a named person attached to the outcome. The rule of thumb is simple: if a wrong answer would require a postmortem, the workflow should pause and ask.

Second, ambiguous evidence. Some cases are clear on the inputs. A digit was transposed; the address matches; the device fingerprint is familiar. Other cases sit in the middle of the distribution — the document is partially legible, the agent extracted three of four fields with high confidence and one with low, the customer’s story does not match the system of record but is internally consistent. Ambiguous evidence is not a model failure; it is a model doing its job and telling you the answer is genuinely uncertain. Auto-resolving an uncertain case is how trust gets burned.

Third, novel situations. Every workflow is written against a known distribution of cases. The moment a case shows up that the policy did not anticipate — a new regulator, a new fraud pattern, a new product line, a partner that broke their contract — the workflow has nothing to say. A good system recognizes the novelty (a confidence threshold not met, a path not taken before, a policy reference not found in the Knowledge Center) and routes the case to a human who can decide and, more importantly, leave a trace that the policy needs updating.

The work of designing for HITL is the work of cataloguing these seams before the workflow ships, naming each one, and deciding what happens when the workflow reaches it.

The task as a contract

A task is not a notification. A notification is something a system sends; a task is something an operator owes the workflow. A task is not a ticket either. A ticket lives in a queue that may or may not be worked, with state that may or may not be reflected in the originating system, and outcomes that may or may not be machine-readable. A task is a contractual gate, not an inbox item.

In Sommatic, the contract has six clauses, and a task does not exist without all of them.

Assignment. The task knows who owns it. Not a group inbox, not a rotation that nobody monitors — a named actor with the role, the scope and the permission to act. If the assigned operator cannot act (out of office, out of scope, conflict of interest), the task knows how to reassign without losing its history.

Queue. The task lives in a queue with rules about ordering, capacity and prioritization. Queues are not just lists; they are the unit at which work-in-progress is measured, throughput is reported, and bottlenecks are detected.

Priority and SLA. The task carries a priority and a deadline, both derived from the case, not from operator preference. A high-value refund and a low-value refund are not the same task at the same priority, even if the workflow that raised them is the same.

Evidence trail. The task arrives with the artifacts the operator needs to decide: the originating event, the agent’s proposal, the relevant policy passages from the Knowledge Center, the prior decisions on related cases, the customer record. The operator does not go hunting; the case comes to them.

Validated output. The task does not close on a free-text comment. It closes on a structured outcome — approve, deny, escalate, request more information — that the workflow can consume. The validation is part of the task definition, not a downstream check.

Audit. Every transition on the task — who picked it up, what they saw, what they decided, how long it took, what evidence they referenced — is captured and replayable. Explainability is not a feature of the workflow only; it is a feature of the task.

A task that is missing any of these clauses is not a task. It is a notification with ambition.

Where tasks come from

Tasks do not appear out of nowhere. They are produced by three sources, and each one has a different design posture.

Workflows pause for them. The most common source is a Tasks node sitting inside a workflow. The workflow runs, hits the node, and yields control. The instance state is persisted; the engine knows it is waiting on a human. When the task closes with a validated output, the workflow resumes from where it paused, with the operator’s decision bound to the run as evidence. This is the cleanest pattern: the workflow author makes the decision to gate at design time, and the operator never sees a case that is not theirs to decide.

Agents escalate to them. An agent working a case can decide, mid-run, that the case is outside its envelope. The agent does not improvise; it raises a task with the context it gathered, the proposal it was about to make, and the reason it stopped. This is the pattern that keeps agents inside their lane. The agent’s job is not to be right about everything; the agent’s job is to be right about what it knows and to escalate cleanly about what it does not.

Operators dispatch them from Command Center. Sometimes the case starts with a human. A supervisor sees something in Omnisearch, opens it in Command Center, and decides this needs to go to a specific operator with a specific deadline. The dispatch produces a task with the same contract as the other two sources — assignment, queue, priority, SLA, evidence, validated output — and the operator on the receiving end cannot tell, and should not need to tell, that the task started with a human rather than a workflow.

Three sources, one contract. That uniformity is what makes the work measurable.

Where humans work them

A task that arrives in a generic inbox is a task that has lost half its value before the operator opens it. The operator now has to gather the context, find the policy, look up the prior decisions and remember the validation rules. The work of the workflow is undone by the work the operator has to do to start working.

In Sommatic, tasks are worked inside Operational Apps — purpose-built React surfaces produced by the App Engine, each one designed for a specific kind of decision. A refund review app is not a generic queue with a comment box; it is a screen that shows the transaction, the customer’s history, the policy passage the workflow matched, the agent’s proposal, and the structured approve/deny/escalate controls that produce the validated output. A document review app shows the source image side by side with the extracted fields, lets the operator correct what the model got wrong, and submits the corrected payload back to the workflow.

The Knowledge Center is loaded into the app, not linked from it. The operator does not click out to a wiki; the policy passages relevant to this case are present at the moment of decision. The prior decisions are loaded too — the last five refunds for this customer, the last twenty for this merchant, the comparable cases that closed in the last quarter. The operator works with the case, the policy and the precedent in one surface.

This is where Operational Apps differ from ticket systems. A ticket system is a generic container for any kind of work. An Operational App is a specific container for a specific kind of decision, with the contract of the task and the context of the case baked into the screen. The result is shorter decision times, more consistent outcomes, and audit trails that capture not just what the operator decided but what they had in front of them when they decided it.

Design patterns

There is no universal rule for where to gate. There is a small library of patterns that show up across operations, and most workflows are an assembly of a few of them. Three patterns appear often enough to name.

Refund triage gate. The workflow receives a refund request. A rule check classifies the case by amount and risk. Small refunds under a threshold, from low-risk customers, with clean evidence, auto-approve and write the outcome back. Ambiguous refunds — middling amount, partial evidence, no recent history — go to a task queue worked by frontline operators inside a refund review app. Large refunds, refunds for flagged accounts, or refunds above a regulatory threshold escalate to a task in a supervisor queue with a tighter SLA and a second-approval requirement. One workflow, three paths, one validated outcome shape.

Document review queue. The workflow ingests a document — an invoice, a contract addendum, a customer-submitted form. An OCR node extracts text. An Agent proposes structured fields with a confidence score per field. Fields above a confidence threshold pass through; fields below the threshold raise a task in a document review app where the operator sees the source image and the extracted fields side by side. The operator confirms or corrects, and the workflow resumes with the validated structured payload. The agent never publishes uncertain extractions, and the operator never re-keys data that was already extracted correctly.

Exception triage. The workflow runs and something breaks — a downstream system returns an error, a contract validation fails, a step times out. Instead of dropping the run, the workflow raises a task carrying the full execution context: which node failed, what the last successful step returned, what the engine tried, what the error said. The task lands in an exception queue worked by operators who understand the workflow well enough to decide whether to retry, skip, repair the data and resume, or cancel. The exception is not a log line in a monitoring dashboard; it is a piece of work with an owner and a deadline.

The common thread across the three: the workflow author decides at design time where the seams are, and the runtime makes the seams operable.

What this looks like: the refund triage gate end-to-end

Walk one refund through the system to see how the contract holds together.

A customer requests a refund for an order that arrived damaged. The request lands through the support channel as a structured event and triggers a refund workflow. The first node is an Agent that reads the customer message, retrieves the order from the system of record, pulls the customer’s prior refund history, and proposes a classification: refund eligible, amount $48, evidence supports the claim (photo attached, order delivered five days ago, no prior fraud flags). The Agent’s proposal is written into the workflow state as a structured payload.

The next node is a rule check. The rule says: amounts under $25 from customers with at least three clean transactions auto-approve; amounts between $25 and $200 with supporting evidence go to a frontline task; amounts over $200 or any case with risk flags escalate to a supervisor task. This refund hits the middle band. The workflow advances to a Tasks node.

The Tasks node creates a task with the contract intact. Assignment: routed by queue rules to a frontline refund operator on shift, with the right scope for the customer’s region. Queue: refund-triage-frontline. Priority: standard. SLA: four business hours. Evidence: the customer message, the order record, the photo, the Agent’s proposal, the relevant policy passages from the Knowledge Center on damaged-goods refunds, and the customer’s three most recent prior interactions. Validated output: approve at the proposed amount, approve at a different amount with reason, deny with reason, escalate to supervisor with note. The workflow instance state is persisted; the engine yields.

The operator opens the refund review app from their queue. The screen shows the customer message on the left, the photo and order detail in the center, the Agent’s proposal and policy reference on the right. The operator reads the message, glances at the photo, confirms the proposal matches the policy, and approves at the proposed amount. The validated output writes back to the task. The task closes. The workflow engine resumes the instance.

The next node executes the refund through the payment integration. A subsequent node sends the customer a confirmation. A final node writes the closed case back to the support channel. The run is complete. Total wall-clock time: under five minutes. Total operator time: under thirty seconds. The audit trail captures the Agent’s proposal, the rule decision, the task assignment, the operator’s view of the evidence, the validated outcome, the payment confirmation and the customer notification — all bound to one run identifier and replayable six months from now.

That is what the contract buys: an operator decision that costs thirty seconds and produces a record that survives an audit.

The economics of HITL done well

A common worry about human-in-the-loop is cost. If a person has to look at every case, you have not automated anything; you have built an expensive notification system. That worry is correct, and it is also why the design matters. The economics of HITL change when humans only see cases that need a human.

When the workflow gates correctly, three numbers move. Cases per operator-hour goes up, because the operator is not chasing context — the case arrives with the evidence loaded. Decision quality goes up, because the operator sees the policy and the precedent at the moment of decision, not after. Cost per case goes down, because the share of cases that need a human shrinks toward the share that genuinely needs one.

A fourth number moves that is harder to put on a dashboard: trust. Operators trust workflows that escalate honestly. Customers trust outcomes that come with a name attached. Auditors trust systems where every closed case has a structured outcome and a traceable path. Trust is what lets the organization push the auto-resolve threshold up next quarter, because the cases that gate are gating for the right reason.

The trap is the inverse. A system that auto-resolves cases it should have gated burns trust quickly, and the response is usually to roll back automation — to put humans back on cases that did not need them. The system that gated too eagerly is recoverable; the system that auto-resolved too eagerly often is not.

Designing for HITL is not designing for the human as a fallback. It is designing for the human as a first-class participant in the run, with a contract, a surface and a measurable contribution. Done that way, automation and human judgment stop competing for the same cases and start operating as one system.

Sommatic ships the pieces that make this concrete: Workflows that pause cleanly, Tasks with a contract intact, Operational Apps that load the case and the context together, a Knowledge Center the apps read from, Agents that know when to escalate, a Command Center that lets supervisors dispatch and observe, and Explainability that captures every step. The work of deciding where to gate is yours. The work of making the gate operable is ours.

→ How Sommatic handles Tasks