What are the key challenges of security metrics?
January 15, 2020
(Updated September 2021) In this post, we’ll take a look at a few of the challenges of security metrics we’ve observed when helping our customers to build their security measurement programmes. We have found the following consistent challenges for almost every one of our customers:
- Frameworks are important, but every metrics programme is unique.
- Security metrics are rarely shared.
- Security metrics are hard to change.
- There are practical limitations, such as resources, access to data and tooling.
Why do security teams struggle to implement and mature their metrics programmes?
The case for security metrics is clear – you can’t improve what you can’t measure. Metrics can provide visibility on where you are exposed to risk and help you to make data-driven decisions about what action to take, rather than relying on limited information or repeating what’s worked in the past.
But security metrics are also a source of frustration for some security leaders and their teams. Implementing and maturing a metrics programme isn’t always easy. It’s a complex and nuanced topic without a one-size-fits-all solution.
Frameworks are a common thread, but every metrics programme is unique
Organisations often align their security metrics programmes with frameworks like the National Institute of Standards and Technology (NIST) and Center for Internet Security (CIS).
There are numerous possible approaches to managing and measuring security, so using an industry-accepted framework can help reduce the ‘paralysis of choice’ and give confidence that an organisation is on the right track.
However, frameworks can present one of the challenges of security metrics. It’s rare for an organisation to implement a metrics programme that maps perfectly to one of these ‘off-the-shelf’ frameworks. We typically see them being used more as a basis for a customised version.
These frameworks cover a broad range of security areas, so an organisation would need to implement a wide range of data collections and measurement to assess all of them. Because organisations can’t realistically implement them all, we tend to see customers focusing on combinations of metrics that align with their priorities.
These combinations can be highly organisation-specific. Different industries have different threats and regulatory needs. Your organisation may have audit points from a report that you must address within a fixed time. This uniqueness makes direct comparisons challenging. One CISO can’t really compare notes with another if they’re trying to measure and improve different things.
While CIS is making strides in becoming more achievable, there are still many cumbersome and unachievable security frameworks out there.
Metrics are rarely shared
Another key challenge of security metrics is that organisations go to great lengths to keep their security programme under wraps.
Talking openly about your security controls or tooling just makes it easier for a threat actor to sidestep those things and this information could damage an organisation’s reputation if their approach to security is seen to be sub-par. This extends to metrics. What you choose to measure says a lot about your priorities, risk exposure and security posture.
For this reason, most organisations will only discuss their metrics anonymously, in a closed forum or in a working group. But even then, there’s some hesitancy. As mentioned earlier, people are often unsure about their metrics programme because there’s no simple solution.
Sharing what they measure with others might expose them to criticism. If nobody knows what you’re measuring, then no one can tell you (rightly or wrongly!) that you’re doing it wrong.
Metrics are hard to change
By the time that a metrics programme gets implemented, a lot of work will have gone into educating stakeholders, iterating the approach and getting those measures approved. At a senior level, any decisions relating to security require a lot of negotiation.
As you get to know your data better and as your priorities and problems evolve, you will probably notice metrics that could be improved upon. Of course, you wouldn’t want to change your key metrics all the time – you need to establish a baseline in order to measure improvement (or decline), but ideally you would be able to mature your measurements over time. Unfortunately, there are various challenges which can make it hard to do this.
A few years ago, I worked with a team who had invested a lot of time getting agreement from both Security and IT on the appropriate metrics to use to measure their vulnerability management programme . When an alternative approach was suggested, they conceded that the new approach may be more effective, but it wasn’t worth rolling back on all of that hard work.
This was quite an eye-opener for me – it’s not simply a case of ‘picking the best maths’. In fact, I quickly learned that that is the easy part!
When you decide on a set of metrics, you establish what aspects of your security programme you’re going to open up to scrutiny. If this changes, stakeholders will want to know why. There’s also the challenge of conveying a lot of information to senior stakeholders in a short time. If the way you measure changes they have to invest more time in order to understand the now unfamiliar presentation of data. As a result, maturing a metrics programme can be a slow process.
There are practical limitations – resources, data, tools
The final challenge of security metrics lies in the practical limitations. Organisations need to weigh up the cost of collecting the data for their ‘ideal’ metrics programme versus the insight it will provide them. Often what would need to be done to collect the data to assess a control in a particular way could be considerably more onerous than the implementation of the control itself.
Some tooling or logging systems may simply not be set up to collect/store the required information, at the required frequency. For instance, if you wanted to measure whether unauthorised devices were quarantined in a timely manner. If we take ‘timely’ to mean ‘within 24 hours’, then the entire network would need to be scanned at least once a day to prove or disprove this. This is something that many organisations don’t have in place.
Similarly, there are often compromises between what you want to measure and what you have time to measure. Security metrics can be time-consuming, especially when the data is being gathered and analysed manually, often in a spreadsheet.
The perfect, as they say, is the enemy of the good. Metrics programmes have to be rooted in what you can realistically manage and also what you’re trying to achieve. Frameworks are useful to provide a recognisable way to organise and communicate your metrics, but you also need to consider what works for your team and your tooling. Measuring something is almost always better than measuring nothing. If nothing else, it gives you the opportunity to learn about your data and see what’s possible and what’s not.
If you’re struggling with any of the challenges listed above, you’ll be pleased to hear we have a dedicated section of our website to this topic – the Security Metrics Hub. We’ll be looking at why the demand for metrics is growing, how teams can meet that demand and how teams can develop and deepen their own metrics programme. We’re also going to be reaching out to the infosec community to learn more about your challenges and how you’re tackling them.