Blocking user non-compliance: the security metric to stop risky behaviour

Every day, every enterprise organisation must block user non-compliant actions. There are plenty of link clicks, file downloads, and emails sent containing sensitive information. Crucially, though, some of these clicks, downloads, and emails could pose a security threat. The riskiest need to be blocked with defensive tools like firewalls.

We’re joined by Shawn Bowen to discuss the importance of measuring how effectively security teams block user non-compliance. Shawn has been in CISO or equivalent roles for over a decade, currently leading the security function at World Fuel Services.

What is blocking user non-compliance?

“As a CISO, my job is to block all our employees’ risky behaviour or activities,” says Shawn. “Most security teams will look at the number of things they block, see there are 1,000 or 10,000, and give themselves a pat on the back for blocking so many actions. Normally you would want these numbers to be high because it helps to justify your security programme. You can report back to stakeholders, and say: ‘Look how much we blocked, give us more money please.'”

“But, is that a measure of how good my security programme is? Or is it a measure of how good that specific tooling is? The real reason I want to measure this metric is because we can use it to measure how good our security education is.”

“It sounds backwards,” says Shawn, “but it is better to have a lower number of blocks (while still retaining the 100% block rate, of course) because that means I’ve successfully educated users. Better cyber-awareness means less non-compliant behaviours and therefore fewer blocks to make.”

The education piece is potentially more important than focusing on blocking risky behaviours. Because if no-one clicks on anything they shouldn’t (as unrealistic as that may be), the overall cyber-risk to the business is hugely reduced.

“I use a lot of analogies,” says Shawn. “If you have a two-year-old who wants to touch the shiny red barbecue, you want to teach them once that it’s hot, so you shouldn’t touch it. You don’t want to block their hand away 100 times a day. A lot of security programmes are security versus users. But that’s not what we’re supposed to be, in my opinion, it should be security versus the bad guys.”

The “number of blocks” metric isn’t just a measurement of the blocking you are doing, but also of the effectiveness of your security education.

Measuring blocks as proof of improvement

With all metrics, we should ask the question: “What does good look like?” Shawn really highlighted the goal: “Better than yesterday.”

For this specific blocking metric, 100% of blocks is clearly good. But where you really see the improvement, as Shawn says, is in a lower number of blocks than your previous measurement and a trend that shows positive progress.

The key is to show you are improving your security programme over time.

“In my 11 years as a CISO, the one thing I say the most about security measurement is that you need to look at it over a period of time.” While that in itself isn’t a metric per se, Shawn sees it as hugely important. Not just for the number of blocks, but for all metrics.

“Too often we look at X number of phishes, viruses, or incidents. If it’s just a number, it’s not really valuable to me. To get that value, you need context. We need to take that number and show it relative to yesterday, last week, last month, last quarter. That’s something a lot of security people don’t do too well.”

Here’s a mock dashboard of what these metrics might look like, aggregated across your tools, so here’s one that focuses on this trend over time:

The amber and green on this dashboard indicate where the metric is in line with pre-established tolerance ranges or service level agreements (SLAs). For example, the hypothetical owner of this dashboard sets a block rate SLA at 99%. It’s green because it remains at a steady 100%. Continuous Controls Monitoring allows users to set their SLAs and continuously measure metrics against them, so that you always know where you stand compared to your goals.

Continuous Controls Monitoring (CCM) can help develop metrics to get business context on otherwise standalone measures. The first part of the dashboard shows total blocks from the enterprise, but the second shows where in the business the blocks are being made. This kind of view can help you focus on the specific areas where security training programme improvement is required.

Catching phish

Shawn dug a little further into the user awareness and phishing side of this metric, because clicking on risky or phishing links is one of the non-compliant user actions that requires blocking.

“Most security teams measure phishing clicks in binary sets,” says Shawn. Click = bad. No click = good.

But he disagrees with that approach. “In my opinion, clicking the link is bad and not clicking the link is neutral. Not good. Good is reporting the link. Any day someone asks me about security, whether it’s a report of a phish or not, because it shows they’re curious, educated and want to learn more. That’s what good looks like.”

In fact, Shawn has brought the phishes reported metric to board level in an organisation that previously used the binary set. “I explained that I care about this metric because these are the people who are actively securing our organisation.”

Here’s a mock dashboard that looks at other metrics that show the story of improving a security programme via coverage of employee phishing tests, phishing click rate, and phishing report rate.

It provides the added business context, breaking down employee user awareness performance by department and job seniority. The +/- indicator to the right of each percentage shows the difference versus the previous measurement.

With dashboards like this, you can see how the changes you implement to your security programme are having a real-time effect on its efficacy.

Measure the value of your security programme

As mentioned above, tracking metrics over time enables you to quantify the value of your security programme.

If you take an action, you want to see that it’s giving you improvement over time. Say you get a new email phishing tool, you want to know that it has improved your overall security, and you can tell that by tracking a positive trend over time in your phishing metrics. As we said above, this metric can prove a downward trend in the number of blocks you have made, thus showing the improvements you’ve made in security education over that period.

Measuring the value of your security programme is hugely important for its future success. And this goes well beyond the singular metric we are talking about today.

“The human attack surface is not an easily measurable environment,” says Shawn. It can be difficult to prove return on investment for ideas or tools. By measuring in the way Shawn suggests, you can demonstrate return on investment, which as a security leader, can get you buy-in for future ideas.

“Security is rarely looked at as a money-making opportunity. Stakeholders rarely see security in action, except when it gets in their way. So, showing your evolution in any area will help you justify another portion of the security programme.”

How does Continuous Controls Monitoring help?

Continuous Controls Monitoring (CCM) can help in various ways. First of all, it provides a “single pane of glass”, aggregating the view of multiple tools into a single, trustworthy view of metrics. Then there’s monitoring daily activities to see a continuous trend over time, something that Shawn notes is essential to the success of any security programme.

Crucially, though, Shawn notes: “It also allows us to pair blocked risky behaviour with tweaking our system. We can use Continuous Controls Monitoring to see the efficacy of enhancements. We should see a spike in one measure when we tighten something up. If we start blocking a new thing, we should see a spike in blocks. Then we should hopefully see those blocks go down as we’re starting to educate on that topic.”

As you make changes to your security programme, such as improvements to user awareness or closing controls gaps, you can see the impact it has on the efficacy of the overall security programme in the measurement over time. That’s extremely valuable to a CISO, not just to know you are improving, but to demonstrate that fact to stakeholders.

We also highlighted the value of understanding business context of measurement. This allows you to understand specific areas of the business where you can focus improvements, such as in the case of focusing on specific job types in particular departments (as per the above example). Not to mention measuring metrics versus SLAs, so you always know how you compare to your goals.

If you’re looking to find out more about Continuous Controls Monitoring, or you’re interested in getting more interviews and discussions with security leaders and measurement experts, sign up for our Metric of the Month newsletter using the form below.