The importance of automating security controls coverage

Welcome to our new ‘Metric of the Month’ series! In this series, we want to spotlight a range of valuable security metrics and measures, and discuss them with thought leaders in the security industry.

At Panaseer we are pioneering the Continuous Controls Monitoring category of security and risk. Part of that is engaging in conversations about the importance of automating security measurement by sharing ideas about quality metrics with the wider community. As the first instalment in the series, we are going to talk about one of the essential groundwork metrics of any effective security programme: security controls coverage.  

We caught up with David Fairman to talk about security controls coverage. David is an experienced thought leader when it comes to security measurement, having run several successful programmes in CISO roles at major financial services institutions across the US, UK, EU and APAC. He is a member of our advisory board, knowledgeable security pro, adjunct professor and Continuous Controls Monitoring advocate.

What is security controls coverage?

In short, security controls coverage metrics provide the organisation with a view of the level of completeness of the deployment of controls across their environment. Controls coverage metrics are also the basis of a Continuous Controls Monitoring programme. Unfortunately, organisations typically struggle with understanding if their controls are deployed everywhere they expect them to be, especially if their inventories are not complete. Taking your inventory to the next level requires continuous triangulation and reconciliation between various data sources to give an organisation the assurance it needs with respect to accuracy and completeness.

‘The only way you can have true confidence in your overall security programme is to measure not only controls operating effectiveness, but also by measuring your controls coverage. I want to know where I have gaps. As security professionals, the things that get you in trouble are the things you don’t know about.’

Why is it important to measure?

‘The only way you can have true confidence in your overall security programme is to measure not only controls operating effectiveness, but also by measuring your controls coverage’, said David.

‘I want to know where I have gaps. As security professionals, the things that get you in trouble are the things you don’t know about. Compromises occur where there is a lack of control, or a control has failed. We know that change in the environment can lead to controls not being deployed, controls being removed or causing controls to fail. You need to be able to capture that as soon as possible.’ 

In order to provide a high level of confidence in your overall security posture, you need to know:  

1. that your controls are working effectively, and  
2. that you have 100% coverage where you expect them (that is to say, where your policy defines where they need to be). 

You need to understand where the control gaps are in your environment in order to address and remediate those gaps. If you don’t know where the gaps are, that’s where the compromises are most likely going to happen.

David spoke about the importance of controls coverage in a Metric of the Month panel:

How do security controls coverage metrics provide value?

As noted above, the first clear point of value is giving you confidence in your security posture. 

The other key use case from David’s perspective is that ‘controls coverage is an essential data point in risk quantification’. Methodologies like FAIR (Factual Analysis of Information Risk) and Cyber VaR (Cyber Value at Risk) allow organisations to quantify risk. Cyber VaR, particularly, is very data-driven. It looks at a wide variety of aspects of security, risk and controls, including external threat landscape, internal events, threat scenarios, security capability, security controls coverage, and your overall security posture. It brings all this together to give you a view of overall residual risk which can be quantified into a value that’s meaningful to the business. 

As part of a wide-ranging Cyber VaR initiative at NAB, David’s team used controls coverage and overall security posture as essential data-points to feed into the overall capability assessment and measurement. Without those key measurements, it’s extremely difficult to quantify risk to any degree. 

What is the role of automation?

‘Automation drives consistency. When a process is automated, we know that we get accurate results time and time again. That means we have a high level of confidence in the validity of those results and the data is not in question.’ 

Part of the challenge of supporting quality controls coverage metrics comes in the manual-intensive process of creating them over and over. You have to go to each tool individually, compile all the data together, then clean, aggregate, normalise, deduplicate and correlate all that disparate data. And then by the time you’ve done all that and you’re ready to use the data, it might out of date. Questions arise around data integrity, and discussions about reducing risk devolve. 

When we spoke about the importance of automation, David had a few things he wanted to call out: 

The first: ‘Automation drives consistency. When a process is automated, we know that we get accurate results time and time again. That means we have a high level of confidence in the validity of those results and the data is not in question.’

The second: ‘Reducing operational cost. There is also a key thing that any security function always needs to be thinking about: how are we reducing operational costs and maximising productivity benefit?’ Automating the processes around not just controls coverage metrics, but all security measurement, allows you to scale and reduce the cost of operations. You can take all the manual effort out, reducing hands in the process, and reducing required resource. A report showed that security teams spend 36% of their time on reporting. Automating this process allows your security people to focus more on doing security rather than reporting it. 

What’s on your ideal controls coverage dashboard?

We’ve looked at why it’s so essential to measure security controls coverage and ideally automate it. So, once you’re there, or there abouts, what controls would you want to see on your dashboard?

I asked David this question hoping he would say something easy so I could flash up a nice dashboard like this showing % of devices missing from certain controls:

I could say: ‘Hey, you’ve got your Active Directory, device manager, vuln scanner, EDR, and you’re ready to go and smash out some security.’ 

Unfortunately, that’s not what happened.  

While ‘% of devices missing an EDR’, and ‘% of asset inventory completeness’ were things that David’s teams looked at in their measurement programmes, that doesn’t tell the full story. 

Rather, he said: ‘It’s all highly dependent on the organisation. It’s not about the controls themselves, but the objectives and outcomes the organisation is trying to achieve by implementing those controls.’ 

He noted high-level objectives for any security team: 

• Reduce the mean time to detect an incident. 
• Reduce the mean time to contain an incident. 
• Reduce the number of data leakage events. 

Easier said than done, but that’s the job. 

When security teams are striving towards these high-level objectives, it is essential to have controls coverage metrics like those in the image to get a basic understanding. Ensuring your controls are deployed and operating as designed will result in the delivery of those objectives. 

‘If there are control gaps or failures, you have exposure and that’s where the incidents are most likely going to occur.’

When you have gaps in your controls, especially detective controls, ‘mean time to detect’ may not just blow out but might not even appear because that control isn’t operating where it should be. That means you can’t realistically drive that metric down and be confident in the metric without ensuring controls coverage.  

‘If there are control gaps or failures, you have exposure and that’s where the incidents are most likely going to occur.’

Having a high level of controls coverage is also about becoming as proactive and preventative as possible. ‘The more you can be preventative, the less you need to rely on detect and respond.’ 

Who makes use of controls coverage metrics?

There are numerous stakeholders within an organisation that need to see security metrics – all the way up to board level. When it comes to controls coverage in particular though, David noted a few key audiences. 

The primary audience is the control owners, whether they are within the security function, infrastructure team, application development, or front-line staff. ‘It’s important for the control owner to understand how those controls are performing in terms of coverage so that they understand if there’s any exposure. This is especially so for the front-line team, because they’re the ones who are responsible for managing the risk and they need to take action to address it.’ 

Some other audiences are people in compliance, audit and regulatory functions. ‘These stakeholders will be able to rely on that data in order to make informed decisions, measure compliance to policy and identify any gaps or risks within that environment.’ It is key for them to consume that data and to be able to rely on it to perform their role. That way they can drive risk-based conversations and take actions as needed. 

David once again highlighted the importance of data trust: ‘When we’re all using the same set of data, we have high confidence in that data because it’s been automated. We understand how it was derived and we trust it.’ 

Because they’re using the one set of consistent data, all these audiences are working from the same reference point. That means discussions are about risk, the right trade-offs and prioritisations. It’s not around the accuracy of the reporting. It allows discussions to stay on-topic and really get to the important issues without challenging the data. 

How does controls coverage relate to Continuous Controls Monitoring?

Controls coverage works as the underlying basis of any Continuous Controls Monitoring programme and needs to be embedded as a foundational element of any high-quality security risk and controls programme. 

‘It’s imperative to start continuous Controls Monitoring if you really want to drive consistent operational and effective 24/7 security operations.’

I asked David for a few final words about CCM in the wider sphere of security: 

‘We are in an environment now where adversaries are more sophisticated. The velocity and complexity of attacks are continually rising. The days of point-in-time, once-a-year assessments and audits are long gone. They are no longer sufficient to maintain an effective defensive posture against that threat. It’s imperative to start continuous Controls Monitoring if you really want to drive consistent operational and effective 24/7 security operations.’