It’s easy for the people on the front lines of security operations to see how adding security automation makes their lives easier and their organizations safer. However, SOC managers still need to be able to quantify the value of automation to senior executives and other stakeholders through SOAR reporting. This makes reporting and metrics critical features of any security automation platform, because they give SOC managers the tools to advocate for continued investment in automation.
In this article, we’ll cover four ways that SOC managers can assess the performance of their automation strategies and demonstrate that value to others.
Compare Task Performance
Automation helps analysts do their jobs faster, more accurately, and at a level that often transcends their experience or skill level. The trick is turning these outcomes into metrics that can be easily understood by stakeholders. Here are some ways to highlight the performance improvements caused by automation:
- Track mean time to respond (MTTR) and compare automated response times to manual response times.
- Zoom in further to track manual versus automated playbook steps, e.g. how long it takes to gather contextual data, how long it takes to identify a genuine incident, and how long it takes to analyze a suspicious file in a sandbox.
- Capture how automation helps analysts perform tasks that are usually handled by higher tier analysts. This can be difficult to quantify, but it can be done, such as by comparing the following scenarios:
-
- A Tier 1 analyst manually gathers intelligence and escalates it to a Tier 2 analyst for assessment and action.
- Intelligence is automatically gathered and risk-scored, so that a Tier 1 analyst can take action themselves.
Measure Integration Value
A lot of the value of automation relies on integrations with other security systems, so it’s important to have a way to show which integrations are supporting progress in the SOC.
For each ticket or incident report, you can track which systems were involved in flagging and escalating the alert. Was it the firewall? Was it the firewall + a threat intelligence platform? Was it data-loss prevention? You can further zero in on the events that are confirmed to be genuine incidents to track which systems are responsible for generating false positives, and which systems are more accurate.
When you’re logging the sources of events and contextual data in a platform like D3 SOAR, you can create reports that track incident detection and conviction rates broken down by integration. This helps the SOC manager identify underperforming systems that can be tuned or replaced, and also demonstrate to stakeholders the ROI from integrations that are contributing to lots of incidents being resolved.
Establish Benchmarks to Track Progress Over Time
While new types of attacks emerge all the time, the broad categories of incidents are mostly static, such as malware, phishing, DDoS, etc. You can use your automation platform’s trend analysis tools to create benchmarks for response and detection times across incident categories.
With robust analytics, such as those included in D3 SOAR, you can create monthly or quarterly scorecards to track MTTR and MTTD (mean time to detection) for each category, with the ability to drill down on elements in the report to see the individual incidents being measured. A few clicks will quickly reveal the outliers, showing what went wrong in incidents that were not resolved efficiently.
Benchmark reporting can help SOC managers demonstrate progress made by automating the response to each incident category over time, and how the results compare to program goals.
Create Case Studies
When reporting to security executives, you can create case studies from security data accumulated over the past month, year, or other time period. Instead of broadly stating, “we’ve stopped a lot of phishing attacks since adding automation,” you can create a full picture of that incident type, for example:
- We saw 200 phishing attacks this month.
- 167 were false positives, 33 were true positives.
- Through automation, we responded to each false positive in three minutes and each true positive in six minutes on average.
- Compared to the same month last year, in which we spent on average 20 and 30 minutes respectively, we were able to save 60.5 hours.
- At an analyst wage of $90/hour, that translates to savings of $5445, just for this single incident type.
D3 SOAR is an enterprise-grade security automation and orchestration platform that strengthens organization’s security operations, saves security teams valuable time, and has the reporting features to easily demonstrate the value of these improvements to stakeholders. Join D3’s weekly SOAR demo to learn more about how security automation can help you meet your security operations goals.
