I recently attended a talk at BSides Pittsburgh called “I Built an Agentic SOC in a Weekend. That’s a Problem.” by John Dwyer, the Deputy CTO and Head of ARC Labs at Binary Defense. In it, he discussed how not all agentic SOCs are built equally. Although an agentic SOC can spit out answers and appear extremely confident in the investigation, the result is not always accurate.
Dwyer built an agentic SOC in a weekend with ChatGPT. It had a beautiful UI, confident score, explanation of the triage, and more. At face value, the outcome can look like it’s well thought out and accurate. But, as he explained, a confidence score is easy to generate. However, a confidence score without evidence is just unpolished uncertainty. Under the model and platform, the evidence boundary can be too thin.
As Dwyer said, “the trap is that incomplete work can look like complete work, but that doesn’t mean the work is good. That just means the writing is good.”
Let’s dig further into the different ways agentic SOCs are built in today’s market.
Two architectures, two very different answers
There are two dominant ways to architect an agentic SOC investigation and they produce very different results, even on the same model.
LLM-led investigation. This approach gathers whatever context it can reach (such as past tickets, documentation, prior case history) to reach a determination. It can produce excellent output, but its design rewards a coherent-sounding summary over confirmed evidence for a real disposition. Every LLM call is a probabilistic dice roll; your job as a system designer is to shorten the odds of a bad roll, and an LLM-led architecture doesn’t do enough of that work for you. Feed it context before evidence and it will happily reach a conclusion that fits the context, even when the context is misleading.
Evidence-led investigation. The evidence-led approach classifies the alert into a genus first, process execution, OAuth/identity, and so on, each with a known schema of attributes. That classification runs deterministically. From there, the system builds an evidence graph. Only after the graph is built does the LLM run hypothesis generation over it, check whether it has enough information to make a disposition, and loop back for more evidence if not. Context comes last, weighted appropriately against what the graph actually shows.
And if you’ve read that and you’re thinking “wait… doesn’t that produce the same outcome still?”
WRONG.
Same model, same alert, different verdict
Dwyer built both versions on ChatGPT-4.5 and fed them the same process execution alert: a web-hosting process spawning a command interpreter.
The LLM-led version pulled prior cases first. It found that the same admin account had triggered false positives during routine replication activity, and it weighed that history heavily. It softened its assessment. A real intrusion, waved through on the strength of a pattern that no longer applied.
The evidence-led version built the process chain first. It found a WMI break and a suspicious binary spawning at the same moment. It weighed the malicious signal above the history and escalated.
One model. One alert. One missed it.
Why the rails matter
Every investigation starts with something deterministic. A process creation alert has a process execution genus. An OAuth alert has an identity/SaaS genus. That classification never requires judgment, so it should never be left to a model.
Encoding that first 20% of an investigation into rails (the steps a human analyst does automatically every time) means the LLM never has to rediscover it. The probability introduced during evidence gathering drops sharply before the model makes a single judgment call.
That is the difference between the two runs above. The LLM-led version had to decide what mattered while it was still deciding what happened. The evidence-led version knew what happened first, so history became one input, not the whole story. Your edge cases live in exactly that gap.
What questions do you need to ask vendors to pick the right agentic SOC?
Here’s what Dwyer suggests:
- Does it build the investigation narrative with a trusted foundation of evidence?
- Is the context factored into the investigation before the evidence?
- Am I able to replay the same investigation? As in, if I take the same payload and run it through the system, do I get the same result?
- What tools does the agent have access to? How are those tools scoped?
- What does the agent do if it doesn’t have enough data to make an assumption?
The best agentic SOC shows you its evidence
SOCs face a lot of pressure to get the triage and investigation right. A confidence score is not the same as a track record. An agentic SOC platform can report 99% confidence and still be wrong far more often than that number implies, especially in an LLM-led architecture with no evidence graph behind it. When Morpheus is uncertain, it defers to a human.
SOCs need an agentic SOC platform that uses an evidence-led approach where you have observability into the decision-making process, with an evidence state that you can trust. A well-architected agentic SOC can handle your edge cases too.
Pick the right one and alert fatigue stops being a problem.
John Dwyer gave such an excellent talk at BSides Pittsburgh. The event was such a blast and the Pittsburgh community was so lovely. Thanks for having me and D3 at BSides Pittsburgh!
And P.S., if you want to learn more about how D3 Security is taking an evidence-first approach with its accountable agentic SOC platform, book a demo.
