Applying the ‘Goal Question Metric’ method to cybersecurity measurement

The ‘Goal Question Metric’ (GQM) approach is commonly used to improve and measure software quality, but it can also be applied to cybersecurity.

James Blake is the co-founder and director of Security B-Sides London, former-CISO at Mimecast and currently Field CISO at Rubrik, and not to mention an expert in cybersecurity measurement.

In this article, James talks us through the Goal Question Metric approach to cybersecurity measurement: what it is, why he uses it, and how.

What is the Goal Question Metric method?

It’s split into three clear sections: Goals, Questions, and Metrics.

You need to start with the goals. “Goals are always applied to objects,” says James. He mentions Douglas Hubbard’s How to Measure Anything in Cybersecurity Risk, which says one of the biggest challenges with cybersecurity measurement is that we don’t understand what we are measuring, that is to say, the object of measurement.

“Defining very discreetly the object of measurement can help to refine and constrain our metrics.” Examples of objects might be product, process, service, resource, or even a person.

Typically, goals will include improvement. You need to define what you want to improve, and why. For example:

Improving security documentation to on-board new joiners more effectively.

The object here is the security documentation and the goal is improvements in effectiveness of on-boarding new joiners.

The next stage is questions. Come up with questions that would show you are achieving that goal. For the documentation goal, some typical questions might be:

• Is our documentation good enough?
• How many changes are made to the documentation? When? By who?
• How clear is the documentation to those who need to consume it?
• When was it last reviewed?

The last section is metrics. These are the things you are actually measuring. You need to see what data sources you have available to create quantitative metrics to answer these questions. Even potentially subjective metrics can be quantified: “On a scale of 1 to 10, how good is our documentation?” You can convert responses in the ordinal scale to a quantifiable percentile and track them over time.

A single goal can break down into a number of metrics – it could be three or four, but can add up to 50 or more. Crucially though, combined they give you a comprehensive view on whether you’re achieving your goal.  You can start with a small number to get a yard stick measurement, then add and refine over time.  It’s a very pragmatic approach that allows you to start now and start simple, but then continually improve.

How do you use Goal Question Metric for cyber asset management?

To put the Goal Question Metric approach into context, James posed an example in one of the most crucial parts of any security programme: cyber asset management.

“Over the years, I’ve built 91 SOCs for companies ranging from the world’s largest retailers, telcos, manufacturers, financial services organisations and more. One of my bugbears is that they all struggle with knowledge of assets, even though it’s number one on every cybersecurity controls framework. This lack of knowledge means every other aspect of cybersecurity is built on a bed of sand.”

James notes that many of the challenges around asset management are not technical, but cultural or political.

With that in mind, he highlights a high-level operational goal for the CISO: “Understand our environment accurately, so we can determine the appropriate levels of risk, apply the appropriate controls, and prioritise incident response.” This is largely aligned to what security frameworks say is the reasoning behind understanding assets.

Then he asks some straightforward questions:

• Do we know our assets?
• Do we know what’s running on those assets?
• Do we know which assets are critical?
• Do we have an accurate map of how our assets are connected?
• Do we know which assets are internet-facing?

You need to break each question into metrics. Ideally you would have a few, maybe five, but even one quantitative metric aligned to your goals and questions can be beneficial.

“If you can show meaningful alignment to goals with one metric, you have a better chance of convincing the business to make that effort and align with you to get the other metrics.”

Example security metrics for cyber asset management

Here are a set of metrics that James suggests for the first question: “Do we know our assets?”

1. What percentage of my devices that are connected to the network are described in the inventory?

To find this, James would scan the network, see what IP addresses are there, pick up the MAC address through the switch tables, and then see which of those are recorded in the inventory.

To do this on an ongoing basis, James suggests listening passively for all ARP (Address Resolution Protocol) requests on your network, and looking for IP addresses. “You can very quickly build a picture of what’s actually connected to your network, then look at your CMDB and realise it’s about 10% complete,” says James.

“You can also consolidate the different views of assets that are spread across multiple different platforms:

• Which systems are in the IT operations management database?
• Which systems are scanned for vulnerabilities?
• Which systems are backed up?
• Which systems have performance and availability monitoring?
• Which systems are logging to log management platforms?
• Which systems are currently being orchestrated by our virtual and container orchestration systems?

Bringing all this data into a single normalised form provides a better overall picture and can serve as a starting basis for a CMDB.”

This can be a big effort if done manually, but there are tools out there that can automate this process, such as Continuous Controls Monitoring.

“Don’t get locked into perfection paralysis aiming for a perfect CMDB,” says James. “Instead aim for a better one than you had yesterday. Rinse. Repeat. Any improvement you deploy quickly adds value to every risk assessment and incident response, rather than aiming for some mythical future state of perfection.”

2. What percentage of servers are connected to the network as described in the inventory?

Servers are a high priority to protect because they house data, which are susceptible to the most common attack techniques – this is what the adversary is going after to encrypt, to steal, or wipe. Using a vulnerability or active scanning tool, you can fingerprint the operating systems deployed on the network and see which are servers.

James highlights a crucial point here. “The value is in the data. It’s not the software and hardware we want to protect, we’re actually protecting the confidentiality, integrity and availability of data.” So, one of James’ key metrics is:

3. What percentage of data stores are classified and described in the inventory?

Again, there are scanning tools out there that can help you with this data discovery.

Here’s a mock dashboard from the Panaseer platform to put these metrics into perspective:

There’s a reasonable amount of work going into this approach, James notes, especially regarding data gathering, but much of it is repeatable. You’ll likely find that you’re using the same metrics to answer multiple questions.

The benefits of the GQM approach to security

The reason this approach is so effective is because it helps to tell a story. By relating the metrics to key questions, and key questions to business goals, CISOs will get much greater buy-in from their stakeholders. James makes a suggestion: “If there’s a particularly unsupportive stakeholder in the business, use their goals as the basis for a GQM approach. Then reframe the measurement of what your programme delivers to show value to their outcomes to get them on-side.”

“Goal Question Metric is a framework for metrics that can help you continually deliver business goals and prove it to the board. You can look at it in the same way you would a framework like MITRE ATT&CK or NIST CSF, in that you have a conceptual layer in goals; an objective layer in questions; and a quantitative layer in metrics.”

“In my current role, I have 27 goals, about 270 questions, and thousands of metrics. So, yes, it is a lot of work to build out the programme, but once it’s in place, it demonstrates ongoing value to the business and shows that you as a CISO are running a tight ship of operations.”

The final word

Goal Question Metric is an extremely effective way to build a cybersecurity measurement programme. It helps align that measurement to organisational goals and build bridges between cybersecurity and the wider business.

As James noted, though, it can be a lot of effort to build out such a programme – many goals, more questions, and even more metrics. That’s where Continuous Controls Monitoring becomes invaluable. It automates your cybersecurity measurement programme, using automation, and advanced data science to create accurate and trustworthy metrics. It’s not just for asset management, but multiple security domains – vulnerability and patch, user awareness, PAM and IDAM, and more.

To see how Panaseer can automate your security measurement, get in touch or book a demo.