11 metrics to empower your threat modelling programme

Threat modelling is a crucial part of cybersecurity. Not just in terms of tool deployment, but also to help enable a cultural shift towards proactive cybersecurity. This month we’re joined by a leading expert in threat modelling, Adam Shostack. He spends most of his time coaching threat modelling, and has authored several books, such as Threat Modeling: Designing for Security, co-author of The New School of Information Security, as well as future release Threats: What Every Engineer Should Learn From Star Wars. Adam shared his view on the importance of threat modelling (or “modeling” depending on your inclinations) and how to measure its effectiveness. Jump to:

• What is threat modelling?
• Activity metrics for threat modelling
• Outcome metrics for threat modelling
• The final word

What is threat modelling?

“Threat modelling is a collection of techniques that help us anticipate threats,” Adam explains. “But it’s not just tool deployment, it’s culture change. We are trying to get better at designing security into the things we're building. It behooves us to measure and observe not just at the technical layer, but also the human layer.” Threat modelling metrics are a way to help with that effort. To give a non-cybersecurity example, Adam says traffic signs are a type of threat modelling. In London, you see “LOOK LEFT” or "LOOK RIGHT" written in big letters where pedestrians cross the road. This reduces the threat of traffic collisions caused by people, such as tourists, looking the wrong way. The threat modelling methodology is clear. Tourists coming from a country that drive on the right side of the road, will naturally look one way, so you tell them where to look. “It’s that simple,” says Adam. “Ideally, you would use these techniques before you have written any code or made any design decisions.” Adam has created a simple structure for some powerful results. He calls it the Four Question Framework for threat modelling, and the questions are:

1. What are we working on?

2. What can go wrong?

3. What are we doing about it?

4. Once the project is finished: Did we do a good job?

But before you begin with these questions, it’s important to consider the bigger picture. “Know why you’re doing it. That applies to both threat modelling and measuring your threat modelling,” explains Adam. “Know what matters, and why it matters. And use that as a basis for measurement.” Beyond the four questions above, Adam outlines two sets of threat modelling metrics – activity and outcome.

Activity metrics for threat modelling

Activity metrics measure what you are doing. Adam provides several examples:

5. What percentage of your total systems have you threat modelled?

This should align to your organisation’s specific goals – your total system may not be in scope for a threat modelling project. The spirit of this metric is coverage – are you threat modelling everything you should?

6. Did you ask the four questions?

It seems obvious, but this metric helps to drive the basics and create a culture of threat modelling.

7. What percentage of new features are you threat modelling as you go?

You don’t have to focus your threat modelling on big systems. You can threat model more easily for individual projects as you go, for example as part of a sprint. As mentioned previously, it’s more effective to threat model proactively, which allows you to stop accumulating security debt.

8. How many people have you trained?

As a threat modelling coach, this is an important metric for Adam as it’s an indicator of the maturity of a threat modelling programme.

Outcome metrics for threat modelling

Often in security, no outcome is the best result. No breaches? Great. No events? Great. Adam explains it: “Outcome metrics are a little weird because if you are threat modelling well, then you’re changing the outcomes.” This is especially true in the case of proactive threat modelling, where you don’t have a baseline measurement. If you’re threat modelling a legacy system, you can measure before and after. There’s a tendency in security to measure things that provide a straightforward number or percentage. How many phishing tests were clicked? How many pen tests have we run? “We can crisply measure them,” says Adam, “but many of the metrics I like get into the squishy human stuff.” Adam provides some example outcome metrics:

9. How many controls did you deploy as a result of your threat modelling?

“We believe that the controls we implement as a result of our threat modelling are having an effect, but that effect is difficult to measure, especially if there aren’t obvious outcomes,” Adam notes. A simple solution is to measure the number of controls you deployed. It may not be as accurate as individual outcome metrics for each control, but it can give you a better idea of where you are. We can also measure how often detective or response controls are firing to help show threat modelling outcomes, for example:

10. How many security escalations are happening?

Good threat modelling should reduce the number of security escalations over time, but also reduce their urgency. There should be fewer last minute “we need to ship this” sorts of escalations. And finally, a qualitative outcome metric that Adam espouses:

11. Would you recommend threat modelling to a colleague?

It’s a bit of a tongue-in-cheek satisfaction question, but it’s powerful: “It captures so much, because if engineers say ’yes’, then you know they’re really getting value from threat modelling. Based on the answer to this one question, I have insight into the maturity and quality of threat modelling work being done in an organisation.” As an aside, Adam notes that this metric is only effective in an established threat modelling programme. As an expert threat modelling coach, Adam’s clients often start from a point of dysfunction. “But it’s always in the back of my mind,” he notes.

The final word

As Adam said at the beginning, threat modelling isn’t just about tool deployment, it’s about cultural change. “Culture eats strategy for breakfast. If we can change engineering and development so that those four questions - “What are we working on?”, “What can go wrong?”, “What are we doing about it?”, “Did we do a good job?” - become innate and automatic, you infuse safety and security into the culture and therefore the technology you’re building.” In terms of security measurement, many of the metrics Adam suggests come from a human perspective. To get answers, you have to ask people rather than go to a data source or tool. But he also recommends a holistic approach to security measurement and a metrics programme: “Security measurement reflects a reality of complexity.” The Panaseer platform was created to help with these top-level challenges. It simplifies cybersecurity measurement using automation and advanced data-science. This enables users to take a more holistic and proactive approach to cybersecurity measurement. It’s important to have an element of both human expertise and data-driven insight for a robust security measurement programme. Gaining a continuous understanding of the coverage of the security controls in place your environment, whether put in place due to threat modelling or otherwise, is crucial to creating a proactive approach to cybersecurity. To find out how we do this, get in touch or book a demo.