Should I build a security metrics platform? Is the juice worth the squeeze?

All organizations are using security metrics and measurement in at least some capacity. Once you get to the point where a quick fix or yearly external audit no longer satisfies – you need to think about a new solution. So a question as old as technology rears its ugly head: buy or build? 

I watch A LOT of TV and movies. It’s kind of my thing. I watch so much stuff that sometimes I think I may run out of content. On the bright side, unlike Arnold Schwarzenegger, I have terrible recall, so I can always re-watch a show and enjoy it as if I have never seen it before.  

Because of all this watching, I frequently find myself using movie quotes as a soundtrack for my life. A personal favorite, and what I consider to be words to live by, is the advice Timothy Olyphant gave Emile Hirsch in The Girl Next Door: “Always know if the juice is worth the squeeze.” I have made many decisions by asking myself that simple question. Is the juice really worth the squeeze?  

How’s that for an intro to a blog about cybersecurity? Well, in the words of Steve Martin in Planes, Trains, and Automobiles, “Have a point, it makes it so much more enjoyable for the listener.” 

So, here’s the point: 

If you want to build your own security metrics platform, the juice might not be worth the squeeze. Why spend ages growing your orange tree, only to get half a glass of OJ in 18 months? Is it easier to just go to the store and buy a gallon? 

A measurement and security metrics program is important

When it comes to maintaining a good cybersecurity posture, facts matter more than opinions. When your actor-turned-CISO Matt LeBlanc says to the security team, “How you doin’?”, you want to respond with facts and figures. It’s a lot more meaningful than quoting The Lego Movie and saying, “Everything is awesome!”

Enter security metrics. 

Security metrics allow us to make strategic decisions based on fact-based concepts. Like mean time to remediation of vulnerabilities; percentage of devices missing a security control; average time to disable an account for a person who leaves the company; what percentage of mobile devices have not been patched; how many users have elevated privileges; or even measuring how you are performing against internal SLAs. Metrics and cybersecurity go together. 

The question is: how do you turn your real-world environment into verifiable and repeatable metrics? 

This is a data problem.

If you think about it, the data that you need to drive metrics relevant to your world exists somewhere in the systems you have deployed across your organization.  

Do you want to generate metrics around vulnerability remediation? You need to pull data from your vulnerability scanners.  

Looking to measure gaps in your control deployment against your complete asset inventory? Pull data from those controls as well as systems that are device-aware.  

Are you trying to generate user-centric metrics like those associated with Identity & Access Management or Privileged Access Management? Then you need to pull from traditional security controls and systems like Active Directory, but you might also need HR data. 

So how do you combine data sources that really were never meant to work together? For many organizations, the answer is a complicated formula that involves spreadsheets, APIs, CSV exports, python, and a whole lot of time.  

Think about this: you want to proactively report on how many days it takes to disable an account for a newly departed employee and then layer business unit data into that to identify which part of the business is failing to meet the established internal SLA for account deactivation. How do you do that? 

This straightforward task could require combining data from at least three sources. But it is highly unlikely your HR data is in a format that will line up with the data in the access control lists for the systems that maintain the user accounts. And that is just one, relatively simple, use case. 

Homemade or store-bought?

So how long does it take to build a system that can generate security metrics you want to measure? 

That really depends on several factors. What are you trying to measure? How extensive do you want to get with your metrics? How often do they need refreshing? Do you have data expertise in-house? Do you have people with extra bandwidth who like to work on new and exciting projects? Do they have the bandwidth to maintain the system?

Clearly, this isn’t an exact science so let’s borrow from some real-world examples.

Having done some research and discussing this topic with several enterprises that have attempted a build-it-themselves approach to security metrics, a few trends have emerged:

• Over 30% of a security team’s time is spent doing manual reporting of security metrics.
• Attempts to automate metrics usually don’t start showing results for well over a year (over 2 years is the most common answer).
• Scalability becomes a concern as data volume increases and maturity starts to max out. Think about building your own CRM versus buying Salesforce. 
• Triangulation across data sources is very error-prone (e.g. Active Directory, antivirus, your vulnerability scanner and your firewall may all track the same device using a different identifier).
• Companies must prioritize which metrics they will start with and then “road-map” the more complicated ones for later.
• There are some other interesting statistics from security pros who have tried building and maintaining their own application in our recent Security Metrics Report.

In most cases, building your own security metrics platform is time-consuming, challenging, and requires far more care-and-feeding than was expected. 

And that is really the point. If you are considering building your own metrics platform, you’ve got to ask yourself, is the juice worth the squeeze? 

We take a look at the debate in more detail in Buy vs Build: Continuous Controls Monitoring for Security Measurement.