How to measure security vulnerabilities that evade patching

In the complexity of an enterprise vulnerability management programme, vulnerabilities can sometimes evade patching. There are times when you may think you have fixed a vulnerability, but the patch failed for some reason. This is a risk that mature security programmes should be measuring.

Why should you measure vulnerabilities that have evaded a patch?

Patch management is a central part of any cybersecurity programme, and is crucial to adhering to every security controls framework, such as NIST or CIS. And it is far from a perfect process. There is so much vulnerability data coming in from your tools, even the most mature patch management programmes can’t remediate everything. Even when implementing effective vulnerability prioritisation, patches aren’t always deployed on time or with complete effectiveness. The “vulnerabilities evading patch” metric is an important way of double-checking that your patch management programme is achieving its goals. If you think you’ve patched a vulnerability but actually haven’t, that poses a risk to the organisation. You really don’t want to see an incident occur and discover it’s because you didn’t properly deploy a patch for a vulnerability you thought was addressed. This is a typical example of a control failure that can be avoided by using this kind of metric. Our 2022 Security Leaders Peer Report found that an astonishing 82% of security leaders have been surprised by a security event, incident, or breach that evaded a control they thought was in place.

What does good look like when it comes to vulnerabilities evading patching?

When measuring this metric, you need to view it within the context of your organisational standards and the maturity of your vulnerability and patch management programme. For example, your target patching window might be 7, 14 or 30 days, depending on criticality – this metric ensures you’re meeting your own thresholds and SLAs (service level agreements). The hypothetical Panaseer dashboard above looks at the total vulnerabilities in the organisation. Then how many of those vulnerabilities that have been patched within SLA. Then the “vulnerabilities evading patch” metric measures how many of those expected patches are not deployed correctly. This information is drawn from a variety of data sources such as vulnerability scanners, patch management tools, and inventories. If a patch is expected on a specific device, but the version is incorrect on the device itself, that would be identified as a vulnerability that has evaded patching. With progression and improving maturity of your cybersecurity measurement programme, for example by using a Continuous Controls Monitoring platform, you can develop these metrics considerably. There are endless ways to slice and dice the data to get greater context around metrics, for example by software family and exploitability, business unit, or worst affected devices. Using these, you can see what risks you’re living with and if your overall risk posture is improving, which is great to report at an executive level. Part of what makes a good metric is that it prompts more questions and piques curiosity. You need to be more technical and answer questions like:

• What portion of new vulnerabilities from the last month were fixed?
• What got past the process?
• What type of software are they on?
• Is patching more effective dependent on region, business line, or type of hardware?

As the maturity of your measurement programme increases, the ability to answer such questions on the spot is incredibly valuable. With a less mature programme, if you want to ask questions like these, you may get a response two weeks later, by which time the data is out of date. That way, you can’t be fully confident in your reporting and security posture.

Who looks at vulnerabilities that evade patching?

Assuming your patch management programme is prioritising your most critical vulnerabilities, this metric is extremely important for ensuring you’re reducing cyber risk. As a result, a lot of stakeholders will be interested. For example, IT operations teams are generally responsible for patching. So, they will want to know if their efforts are effective. A security team is interested because they want to know what gaps exist that they must defend. Then you have your oversight groups – risk and audit functions. These people are interested because they want to understand whether you’re working within the firm’s risk appetite, that is to say, whether you’re patching things within SLA. And whether those patches are working as intended. Senior technology management are also interested. They’ve spent a lot of money on tools and want to know if they’re effective. It’s much easier for them to take a brief look at measures like these and see they are in the green, than to commission studies into the effectiveness of the technology in question.

How can Continuous Controls Monitoring help?

When it comes to vulnerabilities evading patch, CCM (Continuous Controls Monitoring) provides two key things: trust and context. It’s dangerous to trust a patching or vulnerability tool by itself. Or any tool by itself, for that matter. CISO David Fairman noted in our recent ransomware report: “Just because you’re getting a report from one tool, doesn’t necessarily mean you should have a high level of confidence. What have you done to validate the completeness and accuracy of the report and data?” If your patching tool says “everything is alright” but your vulnerability tool is raising a red flag, which do you trust? The truth is probably somewhere in between. It can often lead to mistrust between teams and finger pointing. CCM uses advanced data science and automation to aggregate data from multiple sources across security, IT, and HR tooling, providing a much higher level of data trust. Similarly, a vulnerability scanner or patch management tool doesn’t have the context around your environment. It doesn’t know if a particular device is part of a business-critical process or key location. For the vulnerabilities evading patch metric, CCM enables layering of metrics, applying that context, so you know what to prioritise for remediation. If you’re in a situation where a patch hasn’t been deployed correctly on a business-critical application or process, that’s important to know so you can take action to reduce that risk. If you want to see how Continuous Controls Monitoring can provide a new level of data trust in measuring the effectiveness of your security programme, book a demo with our team, or if you’re enjoying the Metric of the Month series, subscribe below.