Evaluation rubric
How to Evaluate an Agentic SOC
Updated October 2026
Five scored tests measure the AI layer of an agentic SOC in a proof of concept. They run on the buyer’s own alert stream and are scored by the buyer’s own analysts, with the platform in assist mode and every state-changing action behind an approval gate: agreement with Tier 1 on the same alerts; nothing dropped, meaning every alert reaches a decision and none is left open; alert-to-case assembly, meaning related alerts land in one case with nothing lost; verdict to containment in assist mode, meaning a drafted response that waits for an approver; and replay, meaning the same alert run again returns the same verdict and the same reasoning. The tests sit on top of the POC’s success criteria, with the buyer’s existing SOAR checklist as the floor, and they can be run on any vendor, including D3.
Why can’t a SOAR checklist evaluate an agentic SOC?
It checks that the workflow engine works. It cannot tell you whether the verdicts are right, whether anything was missed, or how long a verdict takes. Those are the three things an agentic SOC is for.
What are the five scored tests?
| Test | What it measures | How it is scored | Pass looks like |
|---|---|---|---|
| 1 · Agreement with your Tier 1 | Whether the AI’s verdicts match your analysts’ verdicts on your alerts | Your analysts compare each AI verdict with their own, one alert at a time, on your stream; disagreements reviewed with the evidence | An agreement sheet your analysts signed, with every disagreement explained |
| 2 · Nothing dropped | Whether every alert reaches a decision | Count of alerts left open or undecided at the end of the window, and the stated reason for each escalation | None left open; every escalation names the check that could not be run |
| 3 · Alert-to-case assembly | Whether related alerts are assembled into one case without loss | Duplicates and linked alerts traced back to the source tool; every linked alert visible in the case | Nothing lost; one case per incident, with the links shown |
| 4 · Verdict to containment, assist mode | Whether a confirmed finding produces a drafted, approval-gated response | For each confirmed finding: a containment plan in front of a named approver; the approval record (action, approver, timestamp, result) | Plan drafted; nothing fires without approval; the record is complete |
| 5 · Replay | Whether the same alert gets the same answer | The same alert run ten times; verdict and reasoning diffed | Same verdict, same cited evidence, same reasoning path |
How do you score agreement with your own analysts?
Your analysts score your alerts, one by one. The vendor does not pick the sample. Disagreements are reviewed with the evidence on the table, and the analysts sign the sheet. A vendor’s published number is a starting point, not a score: ask for the method and the stream it was measured on.
What counts as “nothing dropped”?
Every alert reaches a decision inside the window. An alert that cannot be decided is escalated with the missing check stated, never left open and never closed on missing evidence.
How do you test replay?
Run the same alert ten times and diff the verdict and the reasoning. This is the 10-Run Test on the Morpheus Challenge page, and it works on any platform.
What should the POC charter contain?
- The buyer’s own checklist, as the floor
- The five tests, with pass conditions
- A decision rule agreed before the build
- A named same-day contact on each side
- A readiness review before the tenant goes live
- A fix calendar for every open item
- The two-page readout for the decision maker
FAQ
Frequently asked questions
Which tests measure the AI layer in an agentic SOC POC?
Five scored tests run on the buyer’s own alert stream and scored by the buyer’s analysts: agreement with Tier 1, nothing dropped, alert-to-case assembly, verdict to containment in assist mode, and replay. They sit on top of the POC’s success criteria and the buyer’s own checklist, agreed and signed before the vendor builds anything.
Should we use a vendor’s published accuracy number to evaluate an agentic SOC?
Treat it as a starting point, not a score. Score agreement with your own analysts on your own stream, and ask every vendor, including D3, for the method and the stream behind any number it publishes.
What does human in the loop mean during an agentic SOC POC?
The platform runs in assist mode: it investigates and drafts, and a named approver decides every state-changing action. The autonomy mode is configuration the buyer owns; the POC does not move it.
Can the same rubric be run on more than one agentic SOC vendor?
Yes. Same alerts, same analysts, same window. The rubric and the score sheet are vendor-neutral.
How many alerts do we need?
Enough for your analysts to score one by one on your own stream.
Does the rubric cover an MSSP running several clients?
Yes. Run it per tenant, with the tenant’s severity model and the rules written in the partner’s words.
Score Morpheus on your own alert stream
Bring the five tests to a 30-minute demo and we will walk through how each one runs on your stack.