The key challenges of security metrics [Updated September 2021]

In this post, we’ll take a look at a few of the challenges of security metrics we’ve observed when helping our customers to build their security measurement programmes. We have found the following consistent challenges for almost every one of our customers:

• Frameworks are important, but every metrics programme is unique.
• Security metrics are rarely shared.
• Security metrics are hard to change.
• There are practical limitations, such as resources, access to data and tooling.

Why do security teams struggle to implement and mature their metrics programmes?

The case for security metrics is clear – you can’t improve what you can’t measure. Metrics can provide visibility on where you are exposed to risk and help you to make data-driven decisions about what action to take, rather than relying on limited information or repeating what’s worked in the past.

But security metrics are also a source of frustration for some security leaders and their teams. Implementing and maturing a metrics programme isn’t always easy. It’s a complex and nuanced topic without a one-size-fits-all solution.

Frameworks are a common thread, but every metrics programme is unique

Organisations often align their security metrics programmes with frameworks like the National Institute of Standards and Technology (NIST) and Center for Internet Security (CIS).

There are numerous possible approaches to managing and measuring security, so using an industry-accepted framework can help reduce the ‘paralysis of choice’ and give confidence that an organisation is on the right track.

However, frameworks can present one of the challenges of security metrics. It’s rare for an organisation to implement a metrics programme that maps perfectly to one of these ‘off-the-shelf’ frameworks. We typically see them being used more as a basis for a customised version.

These frameworks cover a broad range of security areas, so an organisation would need to implement a wide range of data collections and measurement to assess all of them. Because organisations can’t realistically implement them all, we tend to see customers focusing on combinations of metrics that align with their priorities.

These combinations can be highly organisation-specific. Different industries have different threats and regulatory needs. Your organisation may have audit points from a report that you must address within a fixed time. This uniqueness makes direct comparisons challenging. One CISO can’t really compare notes with another if they’re trying to measure and improve different things.

While CIS is making strides in becoming more achievable, there are still many cumbersome and unachievable security frameworks out there.

Metrics are rarely shared

Another key challenge of security metrics is that organisations go to great lengths to keep their security programme under wraps.

Talking openly about your security controls or tooling just makes it easier for a threat actor to sidestep those things and this information could damage an organisation’s reputation if their approach to security is seen to be sub-par. This extends to metrics. What you choose to measure says a lot about your priorities, risk exposure and security posture.

For this reason, most organisations will only discuss their metrics anonymously, in a closed forum or in a working group. But even then, there’s some hesitancy. As mentioned earlier, people are often unsure about their metrics programme because there’s no simple solution.

Sharing what they measure with others might expose them to criticism. If nobody knows what you’re measuring, then no one can tell you (rightly or wrongly!) that you’re doing it wrong.

Metrics are hard to change

By the time that a metrics programme gets implemented, a lot of work will have gone into educating stakeholders, iterating the approach and getting those measures approved. At a senior level, any decisions relating to security require a lot of negotiation.

As you get to know your data better and as your priorities and problems evolve, you will probably notice metrics that could be improved upon. Of course, you wouldn’t want to change your key metrics all the time – you need to establish a baseline in order to measure improvement (or decline), but ideally, you would be able to mature your measurements over time. Unfortunately, there are various challenges that can make it hard to do this.

A few years ago, I worked with a team that had invested a lot of time getting agreement from both Security and IT on the appropriate metrics to use to measure their vulnerability management programme. When an alternative approach was suggested, they conceded that the new approach may be more effective, but it wasn’t worth rolling back on all of that hard work.

This was quite an eye-opener for me – it’s not simply a case of ‘picking the best maths’. In fact, I quickly learned that that is the easy part!

When you decide on a set of metrics, you establish what aspects of your security programme you’re going to open up to scrutiny. If this changes, stakeholders will want to know why. There’s also the challenge of conveying a lot of information to senior stakeholders in a short time. If the way you measure changes they have to invest more time in order to understand the now unfamiliar presentation of data. As a result, maturing a metrics programme can be a slow process.

There are practical limitations – resources, data, tools

The final challenge of security metrics lies in the practical limitations. Organisations need to weigh up the cost of collecting the data for their ‘ideal’ metrics programme versus the insight it will provide them. Often what would need to be done to collect the data to assess a control in a particular way could be considerably more onerous than the implementation of the control itself.

Some tooling or logging systems may simply not be set up to collect/store the required information, at the required frequency. For instance, if you wanted to measure whether unauthorised devices were quarantined in a timely manner. If we take ‘timely’ to mean ‘within 24 hours’, then the entire network would need to be scanned at least once a day to prove or disprove this. This is something that many organisations don’t have in place.

Similarly, there are often compromises between what you want to measure and what you have time to measure. Security metrics can be time-consuming, especially when the data is being gathered and analysed manually, often in a spreadsheet.

The perfect, as they say, is the enemy of the good. Metrics programmes have to be rooted in what you can realistically manage and also what you’re trying to achieve. Frameworks are useful to provide a recognisable way to organise and communicate your metrics, but you also need to consider what works for your team and your tooling. Measuring something is almost always better than measuring nothing. If nothing else, it gives you the opportunity to learn about your data and see what’s possible and what’s not.

Sound familiar?

If you’re struggling with any of the challenges listed above, request a demo to find out more about how Panaseer’s Continuous Controls Monitoring can help you identify vulnerabilities and automate cyber security risk assessments.

What to learn more about cybersecurity metrics?

Take a look at our Metric of the Month series. We discuss the what, why, and how of the most valuable security metrics and measures. We’re collaborating with thought leaders in security, risk and IT, with years of executive experience to share valuable examples and insight into security and risk measurement best practices.

If you’ve got questions about anything in this post, feel free to get in touch with me on Twitter or Linkedin to carry on the conversation.