Agentic SOC Part II: The Agent's Word Is Not Evidence

Agentic SOC Part II: The Agent's Word Is Not Evidence

In July we wrote this about the investigation agent we were building.

It has one gate to pass before it touches production, which is a blind grade against our senior analysts on real historical cases.

It ships when it survives that, and not before.

Since then the pipeline has run about thirty investigations against real offenses from our production SIEM, across roughly a dozen tenants, delivering into an internal queue that sits alongside StrongHold MCSS.

The blind grade has not happened.

Before we could grade how well the agent investigates, we found we could not trust its account of what it had done.

Every defect we could actually detect was a failure of the record rather than of the reasoning.

Searches that were never issued, recorded as complete.

Evidence citations renumbered between the worker and the report.

Coverage claims the execution log contradicted.

You cannot grade an investigation whose record you cannot trust, because you would be grading the record.

This post is that finding, in detail, including the parts that do not reflect well on us.

The short version

What runs now.

A manager and worker pipeline on real offenses since July.

The manager reasons, dispatches specialist workers and writes the report.

Each worker returns exactly one structured payload.

A full investigation takes five to eighteen minutes across about eight steps.

What we found.

We built a review agent to read completed runs and file findings about the pipeline.

It has raised more than two hundred.

Almost none are about the quality of the investigation.

Nearly all are about the gap between what the system recorded and what actually happened.

Why that is the whole game.

A search that was never issued, recorded as complete with zero rows, is indistinguishable from a search that ran and found nothing.

One is evidence of absence.

The other is absence of evidence.

In an incident that is the difference between saying we checked and it did not spread, and saying we did not check.

Two statuses you should hold us to.

The automated per run quality grader is switched off by a configuration flag, and every run currently completes with grading skipped.

The blind grade against our senior analysts, the one Part I promised, has not started.

What has not changed.

Humans own every customer facing word.

No customer has received anything this pipeline wrote.

What runs now

Part I described this as an internal chat with capability behind it.

That is not what we built.

Chat turned out to be the wrong shape for work that has to happen whether or not somebody is sitting there, so it became a pipeline that runs on an offense and produces a written investigation.

The constraint that makes it work is narrower than it sounds.

A worker can send exactly one message back.

Not a conversation.

One structured payload, validated against a schema, or the turn is refused and the worker retried.

Everything a worker learned has to survive that bottleneck or it does not exist downstream.

The manager runs on our largest model, because deciding what to investigate next is where being wrong is most expensive.

It reads what came back, decides what to dispatch, and writes the final report.

The workers are smaller and faster.

One establishes what the offense is.

Others run the historical sweep, check the ticket history for whether this pattern has been seen before, and hunt for related activity the detection missed.

They run concurrently where they do not depend on each other.

Figure 1. One investigation, eight steps, twelve and a half minutes. Steps overlap deliberately, which is why time inside agent calls exceeds the wall clock. These durations include waiting on SIEM queries, so they are not a measure of model speed.

Model tiering is real, and it is not free.

We put a cheap fast model on one of the synthesis steps.

It dropped ten required fields from its payload and had to be re invoked, which cost more in total than running the better model once.

Model choice should follow schema complexity, not apparent task difficulty.

Summarising sounds easy.

Summarising into a large structured contract is not.

Briefed re invocation turned out to matter more than anything else we built.

When the manager judges an answer insufficient, it does not accept it and it does not fail the run.

It sends that worker back with a specific brief about what is missing.

This fires on roughly a third of runs and accounts for about seven percent of total agent time.

It is the cheapest quality mechanism in the system by a wide margin.

What the review found

The review agent reads completed runs and files findings about the pipeline itself.

Two hundred and six raised.

Eighty eight were things we already knew and had seeded so they would not be re reported.

Of the two hundred currently carrying a severity, forty one are blocking, seventy two major and eighty seven minor.

We opened them expecting analytical mistakes.

Wrong verdicts, missed pivots, bad calls on severity.

That is not what they say.

• Sixteen sweep searches that were never started were recorded as complete with zero rows, and shipped into the report as if they had run.
• A reconciliation function treated any status word it did not recognise as success, which manufactured twelve more of the same.
• A worker numbered its query rows one through seven while the plan numbered them differently, so the runner read all seven as unplanned and every evidence citation in the delivered report silently shifted position.
• A coverage note written by the model claimed all events were retrieved without truncation, while the execution log recorded otherwise.
• A worker returned that the activity was not wider than the offense and had touched zero other hosts, and the delivered report said the opposite.
• The report writer certified that it had covered a blocking gap by naming that gap in its own sources list.

Here is the part worth being precise about, because it would be convenient to blame the model.

Only two of those six are the model reporting on itself.

The other four are our own code: a permissive parser, a runner that recorded intent as outcome, a numbering scheme that did not survive a component boundary, a report that contradicted the payload it was built from.

The common failure is taking a word for an action. Sometimes the word came from a model. Just as often it came from our own lenient handling of a status field. Anyone who has watched a SIEM integration map every unrecognised severity to "informational" has seen the same bug in a different costume.

Figure 2. Every invocation is recorded with the exact prompt that reached the agent and the single payload it returned. This is the record the findings above were checked against. Without it none of them would have been findable.

What "checkable by machine" actually cost

Part I ended the playbook argument with a claim: unlike an analyst's judgement, the contract is checkable by machine.

That was right.

What we underestimated is that the contract needs its own contract.

Every field where the model describes its own completeness is a claim, and claims have to be checked against the execution record rather than against the model's summary of it.

Four things, in order of how much they helped.

Deterministic linting before delivery. Not a model checking a model. Code that refuses specific shapes: a verdict resting on ticket history rather than evidence, a verdict that contradicts its own stated coverage, a citation that resolves to nothing.

Every number traces to a recorded query. If a fact reaches the report writer as prose rather than as a citation, it gets no id, and no id means the linter rejects it.

Reconciliation fails closed. Anything unrecognised is a failure, loudly, where it happens.

The runner wins. Where the model's account of coverage, tool availability or completeness disagrees with what the runner observed, the runner is authoritative and the disagreement is recorded.

If you are building something similar, that list is the post.

Budget for provenance rather than intelligence.

Never let a model self report completeness.

Fail closed on anything unrecognised.

Make re invocation cheap and normal.

And build the reviewer before you think you need it, because nearly everything here came from it rather than from us noticing.

When it breaks

Runs do fail, and we would rather show it.

The console records four end states: finished, failed, crashed and refused.

A refused run is the system working, turning away an offense it should not touch before spending anything.

Crashes in our current store are mostly infrastructure and malformed input, and a meaningful share are test harness runs with synthetic offense identifiers.

Failures are usually a worker returning a payload the schema will not accept, which is irritating rather than dangerous, because a refused payload never reaches the report.

We are not going to quote a success rate. The denominator is polluted by harness runs and we have not separated it cleanly enough for the number to mean anything. When we can, we will publish it.

One genuine embarrassment: our test suite was inserting rows into the live run queue, so test runs and real runs shared a store.

The review agent found that.

We did not.

When a run fails, nothing is lost.

The offense sits in the queue and is worked the way it always was.

What is not true yet

Two separate gates, and both deserve a straight answer.

The automated per run grader is off. It is disabled by a configuration flag, and every run completes with grading skipped. We turned it off while the pipeline was changing too fast for the grade to be meaningful and have not turned it back on. Worse, for a period the delivered report still carried a quality checks appendix, populated only with the deterministic checks, under a heading that implied a grade that never ran. Our own review agent caught that.

The blind grade against our senior analysts has not started. That is the gate Part I named, and we have not passed it because we have not attempted it. The reason is this post. Until the agent's record of its own work was trustworthy, grading the work would only have graded the record.

Read strictly, we have crossed half of the line Part I drew.

The pipeline reads real offenses from the production SIEM today.

It writes nothing a customer sees, and no customer has received a word it produced.

That is a defensible position for an internal analyst aid and it would not be defensible for anything customer facing, which is exactly why the scope is what it is.

Where the data goes, since running live invites the question. The pipeline reads offense data from our SIEM and sends it for model inference. It writes nothing into any customer environment. If you are a customer and want the processing and retention terms restated in writing, ask and we will send them.

The other Part I threads, briefly.

The portal where customers watch the work is still a blueprint, unchanged.

Onboarding and gap analysis is progressing separately and is not part of this pipeline.

What this means if you are our customer

Nothing has reached you from this pipeline, and nothing will until it earns it.

What you can take from it now is a question worth asking any vendor selling you agentic security.

When the report says coverage complete, who wrote that word, the model or the runner? If nobody can answer, the report is an opinion with formatting.

That is what **Full Visibility.

Full Transparency.** has to mean when the thing being made visible is our own tooling failing an audit.

Where this goes

The direction has not changed.

Agents do the work.

People make the calls that carry consequences.

What has changed is that we now know where that seam actually sits, and it is not where we drew it in July.

We assumed the human's job would be judgement about the security question.

Today a good deal of it is judgement about whether the machine's account of its own work can be trusted, and the engineering job is to make that second question unnecessary.

On the review queue, one last honest note.

Two hundred and six findings raised, eighty eight already known, one fixed and marked applied, and the rest not yet triaged.

We are publishing that count before we have worked through it rather than after we have tidied it.

The reviewer is itself an agent, which means its findings are testimony too, and testimony gets reviewed by a person before it changes anything.

Part III is the blind grade result, whichever way it goes.

Verify us.

Come and see it run

If you want to see the console in this post with live data rather than screenshots, including the failures, talk to a security expert and we will walk you through a real run end to end.

Not a slide deck.

The actual pipeline, including the parts we are still arguing about.

---

Author · Gregori Nazarovsky, CTO, QMasters

Last updated · 2026-09-12

Reading time · 9 min