What makes a quality security measurement programme?

Or: Will Santa bring me a new security measurement programme? (Hint: No)

If only Santa Claus handed out security measurement programmes. That’d be cool. You’d think with all the PS5s and Xbox whatever number they’re on nows coming this holiday season, it wouldn’t be that much of a stretch for Santa and his elves to sidestep into security tech. 

Then we could get some classic holiday films based on security tools. I’m picturing Arnie in Jingle All the Way running around the city looking to find the latest Crowdstrike action figure for his son (who looks suspiciously like a certain eight-year-old Darth Vader, by the way). Or even the greatest holiday movie of all time, Die Hard and all those control failures that led to the breach at Nakatomi Plaza. I mean, is it that easy to scare a utility employee into shutting down a multi-block section of the grid? Probably. 

Anyway, if you’re secretly hoping for security tooling under the tree or in the bottom of a stocking, unfortunately, I don’t think it’s going to happen. So, if you want to play with some new security toys in 2021, there are a few things you need to keep in mind. 

The hot holiday toy we are talking about today is a nice shiny security measurement programme. 

Here at Panaseer we talk about security measurement A LOT. That makes sense though, because security measurement is what we do. 

In order to be successful and have just the right shine, a quality security measurement programme invariably needs to be tailored to an individual organisation. Each org has different priorities, policies, standards, geographies, business structures, infrastructure and security tooling. While there is no ‘one-size-fits-all’ solution, there are a few types of metrics and measures that every programme needs. 

At Panaseer, we support security metrics and measurement programmes for some of the biggest and most respected organisations in the world. That means we know a thing or two about it. So, let’s have a look at some of the type of metrics and measures that make for a quality programme: 

 Informational measures 

Informational measures are straightforward counts and sums. For example, total number of vulnerabilities, or total number of Windows 7 machines. They are the building blocks for many more complex measurements. 

Here’s an example from an asset management use case. Measures like total number of devices, with a breakdown by operating system and device type. 

Coverage metrics 

Coverage metrics provide essential context for any performance measures. Part of making an effective programme is understanding not just what you can measure, but also you can’t measure. 

There is no information on the state of vulnerabilities on devices that have not been scanned. That means that for every security area you assess, you need to track the coverage and completeness of the data sources.  

Let’s take an example dashboard about source and controls coverage. This dashboard gives an idea of where your controls are deployed and the coverage of your data sources, compared with where your policy dictates they should be deployed. 

 As part of our recent Metric of the Month series, we chatted with experienced CISO David Fairman about security controls coverage. He spoke about how essential it is to understand your controls coverage to build an effective security measurement programme. 

Here’s what he had to say: ‘The only way you can have true confidence in your overall security programme is to measure not only controls operating effectiveness, but also by measuring your controls coverage. I want to know where I have gaps. As security professionals, the things that get you in trouble are the things you don’t know about.’ 

Policy metrics 

Policy metrics allow you to track adherence to standards across your organisation. That could be internal standards, or it could be regulatory standards. Or both. It’s Christmas, go nuts. While you’re at it, have that third mince pie. 

Here’s an example of the kind of policy metrics you might look at in an end-point management use case. Then you can garner a naughty list of all the devices failing policy. 

Diagnostic metrics 

If you have identified areas of subpar performance using policy metrics, diagnostic metrics provide more in-depth insight that helps you to narrow down the root cause and quickly identify actions that help reduce risk. Then from there you can take remedial action. 

This example is a little more complex. This dashboard shows the ‘percentage of employees who failed at least one phishing test by department and title’: 

This diagnostic metric can be used to great effect in a user awareness analysis use case. It looks a little more complex than the other examples in this blog, but that’s because it is. Here we are looking at a user awareness use case from another angle. Looking at diagnostics like this in your organisation can help you explore both risk exposure and the effectiveness of departmental security training.  

Does one department consistently perform better than others? Yes, in this case its HR. What security training approaches are HR using that we can incorporate into other departments? Do certain roles perform badly across the board? Maybe they need more targeted training? 

These kinds of diagnostic metrics are really good to get targeted information and so take specific action in the organisation to improve security.  

So, what does a quality security metrics programme need? 

Informational measures to see the total number of all the stuff you need to see. Coverage metrics to see the coverage and completeness of your controls and their deployment. Policy metrics to see how your status compares to your policies and regulations. Diagnostic metrics for that extra dash of insight.

If you are looking to kick 2021 off by playing with some new security measurement kit, check out Santa’s naughty and nice list (aka our Guide to security metrics and measures) or book a demo.