AI and Detection Engineering: The Instrument, Not the Author

AI and Detection Engineering: The Instrument, Not the Author

We went looking for our noisiest detection rules and found something else first. In our own QRadar deployment, 89% of the searchable stored events were health telemetry from QRadar monitoring itself — zero security value, sitting in the middle of every hunt. Then our own event-rate numbers disagreed with the console. That disagreement turned out to be our bug, not the platform's. The correction is the whole point of this post.

AI-assisted detection engineering is the use of large language models to measure, interrogate, and maintain a SIEM's detection layer — not to author rules unsupervised. In practice, AI is strongest before authorship: quantifying noise, verifying event-rate math, ranking rules by real fire data, and flagging telemetry gaps. Detection logic itself remains code: version-controlled, peer-reviewed, schema-validated, and proven against emulated attacks before deployment. A rule that parses is not a rule that detects — and no AI-generated rule should reach production without a human merge and empirical validation.

The market is busy selling AI-as-author. Our evidence says the value is AI-as-instrument. Detection engineering is an empirical discipline before it is a writing discipline, and the industry is automating the writing while skipping the empiricism.

This is what AI detection engineering looks like in practice, and what follows spans IBM QRadar, Splunk, and CrowdStrike Falcon Next-Gen SIEM. The measured numbers are from QRadar, because that's where we did this particular work — but the discipline is platform-agnostic, and so are the traps.

The Four Verbs, Resolved

Part 1 framed AI's work as four verbs: ask, investigate, summarize, spot trends. Part 2 put an architecture under them — the AI lives on a read plane; execution belongs to humans and reviewed code.

Detection engineering is a write-heavy discipline. It merges, deploys, tunes, and retires. So here's the resolution that keeps the series honest: those write verbs are not the AI's. Inside a discipline full of them, the AI's verbs stay exactly what they were — read verbs. It asks, investigates, summarizes, and trends. Everything that changes production is a human decision running through a pipeline.

Part 2's thesis was that policy lives in the OAuth scope, not in the prompt. Part 3's is its sibling: policy lives in the pipeline, not in the prompt.

The Loop

Detection engineering is a loop, and the AI's role — and its hard boundary — is different at every stage.

Stage 0 — Collect: You Cannot Detect What You Stopped Collecting

The loop starts before the first rule is written. On one console we measured, 1,800 of 3,310 log sources were silent — including an event collector that had emitted nothing since September 2024. Meanwhile a single category, firewall traffic, accounted for roughly two-thirds of all events. Every detection mapped to those silent sources was, in effect, decoration.

If you've never audited this, it's the first place to look — our QRadar SIEM audit checklist covers the log-source-quality pass, and Windows event collection is where estates most often go quiet without anyone noticing.

This is why we built telemetry integrity as its own surface rather than a dashboard afterthought — and why it has a freeze rule: when our own collector is degraded or stalled, it must not accuse its sources of going silent. A monitoring system that cries wolf about its own outage teaches people to ignore it.

One design decision here is worth stating because it cuts against this series' own thesis: we deliberately did not put the telemetry sampler behind MCP. A sampler has to enumerate every source, and MCP's token caps truncate. Truncation in an investigation costs you a paragraph; truncation in an inventory silently invents silence. So that collector talks to the API directly. Knowing where not to point the AI is part of the engineering.

Stage 1 — Measure: The Strongest Stage

This is where AI earns its place, and where our headline findings came from. Natural-language questions become AQL: which rules fire most? what is actually consuming our ingest? which sources went quiet?

Then it got interesting. Our fleet event rate came back at roughly 8K EPS. The console said something closer to 18K. One of those was wrong.

The resolution: *EPS is SUM(eventcount), not COUNT(). QRadar coalesces similar events into stored records, so counting records undercounts raw ingest. Measured over a five-minute window in our environment, SUM(eventcount) matched the console at roughly 14–18K EPS*, while COUNT() gave about 8K — and a two-hour average diluted it down to ~5K. Worse, coalescing is per log-source type, and the spread was enormous: on our estate, on that date, Check Point ~2.4×, FortiGate ~2.7×, Windows Security ~6×, DNS Debug ~17.6×, CloudTrail ~18.7×, and health telemetry ~1.0×. There is no valid global multiplier. Anyone selling you one is guessing. This isn't only an engineering concern, incidentally — EPS is often the unit your SIEM licence is priced in, which is its own argument.

Which brings back the 89%. Health telemetry was 89% of searchable stored events but only about 17% of raw EPS — precisely because it doesn't coalesce while real sources do. Two true numbers that look contradictory until you know the mechanism.

And the part that matters most: we got that wrong first. An early query selected devicetype without including it in the GROUP BY, which produced confidently wrong output — health telemetry logged as roughly 0% of EPS when it was really ~17%. Nothing errored. The query ran, returned numbers, and lied. It was caught by a human cross-checking the result against the console, not by the AI noticing. AI-assisted measurement helped us get it wrong before it helped us get it right. Verification culture is what saved it.

Two smaller honesty notes from the same work. QRadar's own top-rules health endpoint timed out in our deployment, and its analytics API rejects server-side sorting — so our noise ranking uses average_capacity as a proxy, sorted client-side. It's a proxy, and we say so. For empirical truth we use a different path: RULENAME(creEventList) returns the rules and building blocks that actually fired on each event, so a GROUP BY over event data gives measured noise instead of a metadata guess. Metadata proxy and measured truth are not the same thing, and knowing which one you're holding is the job.

The trap is platform-specific; the discipline isn't.

We measured the QRadar row. The other two are where to look, not what we found — which is the point.

Natural language flattens those dialects, and that is genuinely useful and quietly dangerous: an AI will produce a confident, fluent query in a language you can't spot-check. Whatever the platform, the rule is identical — reconcile the number against that platform's own console before you believe it. We didn't, at first.

Finally, a Part 1 callback: our QRadar connector filters that health telemetry by default, bypassable per call. The noise policy lives in the server, not in the prompt.

The read plane: a read-only MCP server is the instrument, never the author. Diagram via Wikimedia Commons (CC BY-SA).

Stage 2 — Draft: The AI Opens the PR

AI drafts well. First-pass rule logic, explaining what an inherited rule from 2019 actually does, and translating detections between query languages — Sigma to SPL, SPL to AQL, AQL to CQL — all genuinely useful, all genuinely time-saving. Cross-platform translation is where drafting shines and where it bites hardest: a rule that is perfectly valid in Splunk can convert into flawless AQL that references a QRadar field which doesn't exist in your deployment.

The boundary is absolute: the output is a pull request, never a deployment. And the failure mode to fear isn't a syntax error — those get caught. It's the plausible failure. A model that half-remembers a schema will produce a rule referencing a field that doesn't exist in your environment. It parses. It passes review if review is careless. It deploys cleanly. And it never fires, forever, silently. Noisy failures get fixed. Silent failures get budgeted for.

Field names come from the live environment. Never from the model's memory. There's a worked example of exactly this below.

Stage 3 — Validate: Emulation Is the Only Truth

A rule tagged T1059 is a hypothesis, not coverage. The only thing that converts a claim into a measurement is running the technique and seeing whether anything fires.

A MITRE technique tag is a hypothesis. Only executing the technique and observing whether anything fires converts it into measured coverage. Source: MITRE ATT&CK.

That's why we built a macOS ATT&CK validation harness — 31 techniques across 11 tactics, executed on real endpoints through EDR remote response, each self-cleaning, then scored into a coverage matrix and an explicit gap list. What it taught us in its first runs was mostly about the harness itself, and every lesson generalizes:

• The correlator over-credited. Lenient parent-technique matching (treating T1059 as a match for T1059.00x) credited three Execution techniques for what was actually a credential-access alert — and marked the credential techniques "missed." Our coverage number was wrong in both directions simultaneously.
• A payload silently never ran. A quote-escaping bug in one technique's shell payload produced a syntax error that killed the whole thing. On a coverage matrix, a technique that never executed looks identical to a technique that executed and wasn't detected. We now syntax-check every payload offline before any run.
• A wrong field name reads as "not detected." If the alert schema field you're correlating on is misnamed, the harness reports a clean miss rather than an error. That is the single most dangerous property in this entire discipline, and it's exactly the shape of an AI hallucination.
• Some gaps are choices. We deliberately excluded destructive techniques that aren't safely reversible. Those are documented as intentional exclusions, not as missed coverage. Honest gap accounting means distinguishing "we chose not to test this" from "we don't detect this."

And the disclosure that belongs in public: our live-tenant alert-schema validation is still pending. Until that's done, we don't publish coverage conclusions from this harness — because we don't yet trust the instrument enough to indict the detections.

We built our own harness because macOS was our gap. You may not need to. On the Splunk side, Attack Range (Apache 2.0, maintained by Splunk) builds an instrumented cloud lab, runs Atomic Red Team techniques against it, and forwards the telemetry into Splunk for exactly this purpose — developing and testing detections against real attack data rather than imagined data. The tooling matters far less than the principle: the technique runs, or your coverage is a rumour.

For teams who need production-grade gap analysis without building a custom harness, CardinalOps provides a detection-coverage platform that maps your existing SIEM rules to MITRE ATT&CK, quantifies coverage gaps, and flags rule-quality issues across QRadar, Splunk, Microsoft Sentinel, and more. It's a commercial complement to the measurement discipline described here — the same questions apply; the instrumentation is pre-built.

The same disease shows up one layer down, in the pipeline. Our connector linter's first real catch was a defect in our own previously published QRadar connector samples: a GET request carrying a form-encoded body, which means the API never sees the filter parameters at all. The connector validates. It runs. It silently ignores its own filters. In the same family of samples, a hand-written workflow only ever read page 1 of a paginated API — quietly collecting a fraction of the data it claimed to. A connector that runs is not a connector that filters. Same disease as the rule that parses but doesn't detect.

Note what made the generated connectors trustworthy: not the model, but the gates. Every output is validated against IBM's schema and our own lint rules before anyone looks at it. AI output is acceptable because it passes deterministic gates — not because the model is trusted.

Stage 4 — Deploy: The AI's Role Is None

There is no AI role in this stage. That's the point. Human merge, CI, and the pipeline.

If you want a reference implementation rather than our opinion, Splunk publishes one: its security_content repository defines detections as YAML with tests attached, validated and packaged by contentctl, and shipped to customers as the Enterprise Security Content Update. Detection-as-code isn't a QMasters preference — it's how the vendors themselves ship detections. The CrowdStrike equivalent surface is NG-SIEM correlation rules, which turn log events into Falcon detections in the same console the SOC already works in. Different platforms, same requirement: a diff, a reviewer, and a test.

Stage 5 — Tune and Retire: Propose, Never Apply

Tuning is where the security argument gets sharp. A threshold edit, a whitelist addition, a rule quietly disabled — an assistant that can silently tune your SIEM is functionally indistinguishable from an attacker suppressing your detections. So the AI proposes, with fire data attached, and a human applies. Every time.

Retirement leaves evidence: fire history and overlap analysis in the pull request. Retired is not deleted. On why generic correlation rules fail in the first place — and what tuning them at scale actually costs — see detecting APTs with the QRadar suite.

Worked Example: Drafting a CloudTrail Detection

Theory is cheap. Here is a real drafting loop, on a detection worth having: someone turned off CloudTrail logging. It maps to MITRE T1562.008 (Impair Defenses: Disable or Modify Cloud Logs), it's low-volume, and it's about as close to unambiguous evil as cloud telemetry gets.

The ask, in English: "Tell me when someone stops or deletes a CloudTrail trail."

The AI's first draft, for QRadar:

SELECT eventName, sourceIPAddress, username
FROM events
WHERE eventName IN ('StopLogging', 'DeleteTrail')
LAST 24 HOURS
LIMIT 100

It reads beautifully. It passes a tired reviewer. It would never have worked.

What review caught — four things:

1. eventName is not a QRadar column. It's a field in AWS's CloudTrail JSON payload. QRadar surfaces payload fields through custom event properties, whose names are deployment-specific — and if nobody created and enabled that property, it isn't queryable at all. The model didn't lie; it pattern-matched AWS's documentation into QRadar's grammar. That is precisely the plausible failure this post keeps warning about.
2. The clause order is wrong. In AQL, LIMIT comes before LAST. As written it's a parse error — which makes this the lucky bug, because it fails loudly. The field-name bug fails silently.
3. No log-source filter. Nothing scopes this to CloudTrail, so it scans the estate to find a handful of events.
4. No exception path, no test evidence. Trails legitimately get stopped during infrastructure work. A detection with no tuning story becomes a muted detection within a month.

What shipped, after somebody opened the deployment and looked up the actual property names:

SELECT
  QIDNAME(qid) AS event,
  LOGSOURCETYPENAME(devicetype) AS log_source,
  "AWS CloudTrail Event Name" AS event_name,
  "AWS CloudTrail Source IP" AS source_ip,
  username
FROM events
WHERE devicetype = <your_cloudtrail_type_id>
  AND "AWS CloudTrail Event Name" IN ('StopLogging', 'DeleteTrail')
LIMIT 100
LAST 24 HOURS

Note the discipline in the WHERE: we filter the raw indexed column (devicetype) and decode only in the SELECT. Wrapping a column in a decode function forces a full scan — the difference between a rule that returns and a rule that times out.

The same detection, as a CrowdStrike NG-SIEM correlation rule in CQL:

eventSource = "cloudtrail.amazonaws.com"
| in(eventName, values=["StopLogging", "DeleteTrail"])
| groupBy([userIdentity.arn, sourceIPAddress, eventName])

Saved under Next-Gen SIEM → Monitor and investigate → Rules, it raises a Falcon detection in the same console the SOC already works in. Note how much shorter it is — and resist the conclusion that shorter means safer. It's shorter because the parser already did the normalization work. Whether it did it the way you assume is a question you answer by looking, not by hoping.

Two more, briefly

Root console login — maps to T1078.004 (Valid Accounts: Cloud Accounts). In a healthy account this should fire approximately never, which makes it a superb canary and a terrible thing to discover you never had:

SELECT
  "AWS CloudTrail Event Name" AS event_name,
  "AWS CloudTrail User Identity Type" AS identity_type,
  "AWS CloudTrail Source IP" AS source_ip
FROM events
WHERE devicetype = <your_cloudtrail_type_id>
  AND "AWS CloudTrail Event Name" = 'ConsoleLogin'
  AND "AWS CloudTrail User Identity Type" = 'Root'
LIMIT 100
LAST 7 DAYS

Enumeration burst — a principal collecting AccessDenied errors across services, mapping to T1580 (Cloud Infrastructure Discovery). This is the aggregation shape, in CQL:

eventSource = "cloudtrail.amazonaws.com"
| errorCode = /AccessDenied|UnauthorizedOperation/
| groupBy([userIdentity.arn], function=count(as=denials))
| denials > 25

And here is where this post eats its own tail. Where does 25 come from? If you picked it because it felt right, you've built a coin flip. The only honest way to choose it is to measure your own baseline first — Stage 1, again.

Worse, on QRadar the unit is a trap. Remember that CloudTrail coalesced at roughly 18.7× in our estate: a threshold counted in stored records is not a threshold counted in API calls. Set 25 against coalesced records and you may be waiting for something closer to hundreds of real denials before it ever fires. The rule would be live, green, and quietly useless. A threshold in the wrong unit is a rule that parses but doesn't detect.

On the field names above: eventName, sourceIPAddress, userIdentity, and errorCode are AWS's own CloudTrail schema and are safe to rely on. How they surface in your SIEM is not portable — in QRadar they're custom event properties named however your deployment named them; in NG-SIEM they depend on the connector's normalization. The property names in these examples are illustrative. Yours will differ, and confirming them is step one, not step nine.

What We Commit To

1. Every detection is code — versioned, diffed, peer-reviewed. The AI may open the PR. Only a human merges.
2. No AI-authored rule reaches production without schema validation against the live environment, lint and syntax gates, and test evidence attached.
3. A rule must prove it detects, not merely parse. Three silent-failure layers, three gates: the field doesn't exist (schema validation), the field exists but the value space differs (fixture replay), the source isn't collecting at all (telemetry integrity).
4. Zero-fire is an incident. A rule that has never fired means the logic never matched or the telemetry died. Both are findings. Neither is peace.
5. MITRE honesty. Coverage has three states: claimed, validated by emulation, and validated recently. Validation decays as the environment changes.
6. Tuning is a change and gets the same review as a new rule. The AI proposes; it never applies.
7. Retirement leaves evidence. Fire history and overlap analysis, in the PR.
8. Measure the measurement. Every AI measurement claim is reproducible — the query is logged and re-runnable. "Show me the query" is the review standard.

"Read-Only" Is a Scope, Not a Feeling

One concrete example of why this needs actual engineering attention rather than good intentions. CrowdStrike's own Falcon MCP exposes an NG-SIEM search tool, and its documented scope requirement is both NGSIEM: READ and NGSIEM: WRITE — because the Next-Gen SIEM search API is asynchronous and job-based. Running a query creates a search job, and creating a job is a write.

Nothing is wrong with that design. But notice what it does to a lazy assumption: searching feels harmless, therefore a search tool must be read-only. It isn't. Read-only is a property of the scopes you granted, not of how the verb sounds. Hand out a "read" tool without reading its scope table and you've granted write. Part 2 said policy lives in the OAuth scope. This is what that costs in practice: somebody has to actually read the scopes.

What We Won't Claim

• We won't quote a false-positive reduction percentage. We have no controlled comparison, and neither does anyone else quoting one.
• We won't quote an ATT&CK coverage percentage. The denominator is contested, and our own validation harness isn't fully verified yet.
• We won't say "self-tuning SIEM" or "autonomous detection engineering." Our governance forbids the capability, so our marketing can't claim it.
• We won't tell you to use our numbers. The 89%, the 14–18K EPS, the coalescing multipliers — those are our estate, our window, our date. The takeaway is measure yours.
• We won't say the AI found the bug. A human cross-checking AI-assisted output against the console found it. Human-plus-AI loop, not AI magic.

Where to Start

1. Measure before you author. Find out what your ingest actually is — and verify the math against your console before you trust it.
2. Check who's still talking. Silent sources make silent detections. Count them.
3. Put detections in git. If a rule can reach production without a diff and a reviewer, nothing else here matters.
4. Validate by emulation. Run the technique. A tag is a hypothesis.
5. Point the AI at questions, not at the merge button.

This is the discipline behind how QMasters runs detection engineering, MDR, and 24×7 SOC and incident response — across the platforms our teams live in every day: IBM QRadar, Splunk, and CrowdStrike. The guardrails live in the system, not in the prompt.

If you're working out where AI belongs in your own detection practice, that's a conversation we're glad to have as peers.

Give the AI the instrument. Keep the pipeline. Because the AI never holds the merge.

Frequently Asked Questions

Can AI write detection rules?

It can draft them, and that's genuinely useful for first-pass logic, translating between query languages, and explaining inherited rules. It should not author them unsupervised. An AI-drafted rule is a pull request: it needs schema validation against the live environment, lint and syntax gates, peer review, and test evidence before a human merges it.

What is the biggest risk of using AI in detection engineering?

Silent failure. A model that half-remembers your schema produces a rule referencing a field that doesn't exist. It parses, deploys cleanly, and never fires — forever, with no error. Noisy failures get fixed; silent ones get budgeted for. This is why field names must come from the live environment, never from the model's memory.

Is a rule mapped to a MITRE ATT&CK technique the same as coverage?

No. A technique tag is a hypothesis. Coverage has three states: claimed, validated by emulation, and validated recently — because validation decays as your environment changes. Only executing the technique and observing whether anything fires converts a claim into a measurement.

How do you measure SIEM events per second correctly?

Sum the event count, don't count the records. SIEMs coalesce similar events into stored records, so COUNT() undercounts raw ingest. In our QRadar environment, SUM(eventcount) over a five-minute window matched the console at roughly 14–18K EPS while COUNT() returned about 8K. Coalescing ratios vary per log-source type — there is no valid global multiplier, so measure your own.

Is detection-as-code different across QRadar, Splunk, and CrowdStrike?

The discipline applies to all three — only the traps differ. Our measured examples come from IBM QRadar (AQL), where event coalescing skews naive event-rate math. Splunk (SPL) and CrowdStrike Falcon Next-Gen SIEM (CQL, running on the index-free LogScale engine) have quirks of their own. The constants across every platform: verify each number against that platform's own console, keep detections in version control with tests attached, and prove coverage by executing techniques rather than by trusting a tag.

Should an AI be allowed to tune or disable SIEM rules automatically?

No. An assistant that can silently change thresholds, add whitelists, or disable rules is functionally indistinguishable from an attacker suppressing your detections. Tuning is a change and deserves the same review as a new rule: the AI proposes with empirical fire data attached, and a human applies it.

Does AI reduce false positives?

We don't publish a number, because we have no controlled comparison — and neither does anyone quoting one. What AI demonstrably does is make the estate measurable: ranking rules by real fire data, quantifying noise, and surfacing telemetry gaps faster than manual analysis. Humans still decide what to tune.

---

Gregori Nazarovsky is CTO of QMasters, where he leads the engineering behind the company's SOC, MDR, and detection platforms. The measurements and tooling described here were built and run in-house. This is Part 3 of a series on AI in security operations — Part 1: MCP and the SIEM; Part 2: CrowdStrike RTR and incident response at scale.

References

• MITRE ATT&CK
• IBM QRadar Ariel Query Language (AQL)
• IBM QRadar Universal Cloud REST API samples
• Splunk Security Content — detections as code (Apache 2.0)
• Splunk Attack Range — detection validation lab (Apache 2.0)
• Atomic Red Team — Red Canary
• CardinalOps — Detection Coverage Platform
• CrowdStrike Falcon Next-Gen SIEM
• Falcon MCP — NG-SIEM module (scope requirements)
• Model Context Protocol — modelcontextprotocol.io