A meaningful measurement mindset: step one in security measurement

This week we joined FS-ISAC for a webinar called ‘Continuous Controls Monitoring: What to measure’. Panaseer CEO Nik Whitfield and Lead Security Data Scientist, Leila Powell spoke about some helpful steps and processes when asking the question: ‘What security metrics should I measure?’ As part of a series of security measurement webinars, we spoke about Continuous Controls Monitoring at FS-ISAC's virtual event in May of this year – what it is, and how it helps organisations to provide visibility, measurement, and trust in their security programme. We were joined by security thought leader Max Bemrose, who highlighted some of the key steps to take as you begin to implement Continuous Controls Monitoring (CCM) in your organisation. Establishing the why of your security programme; evaluating the current landscape; configuring the CCM programme; working out what to measure; and collaborating with stakeholders across other sections of the business. From there, we received further questions – most notably: ‘What security metrics should I measure?’ It’s a common enough question and sounds like it should be relatively simple to answer. But, as with many questions regarding information security in a large financial services organisation, that is not the case. So, this week we hosted another security measurement webinar for the FS-ISAC audience: ‘Continuous Controls Monitoring: What to measure’. Introduced by our CEO Nik Whitfield, and led by our Lead Security Data Scientist, Leila Powell, this webinar aimed not to provide people with a list of ‘Top 10 Metrics’, but with helpful steps and processes that can help organisations to answer the question: ‘What security metrics should we measure?’ 

The meaningful measurement mindset 

The first step is to adopt the meaningful measurement mindset. This means starting with your objectives, not getting bogged down in the vastness of all the possible things you could measure. It also means avoiding a data-first mindset – many organisations start a metrics programme based on the data they have easy access to, but that doesn’t take into account your organisation’s business or security priorities. 

The measurement checklist 

This is a series of five questions with a goal of making it much easier to work out what you should be measuring with your metrics programme. 

• What is your measurement objective and how will you achieve it? 
  • The more focused the objective, the easier it will be to work out what to measure. 
• Who is your audience? 
  • What is their perspective? 
  • How much detail do they need to see? 
• What questions do you want to answer? Are they the right questions? 
  • As a thought experiment, you find the answer to your question is 42%. What would you do with this information? If you don’t know, don’t measure this metric yet. Instead, focus on something that will prompt action. 
• What data will you use? 
  • Try to find a sweet spot between data that is valuable and obtainable. 
• Which questions can translate into effective metrics? 
  • Ensure you understand exactly what your metric does and doesn’t measure. 

Leila illustrated the checklist through a use case around user awareness and phishing tests, starting with a high-level objective (improve employee’s ability to spot a phish) and going through the checklist to come up with final measurements on performance and coverage. 

Maturing your measurement 

As you create robust metrics and measures over time, you can use them as building blocks. Look for opportunities to combine them to get richer analysis. For example: Here, we combine two pairs of foundation metrics – number of repeat phishing test offenders overlaid with number of employees with privileged access to PCI systems and number of devices with no endpoint agent overlaid with number of devices with missing patches. By doing so, we create a metric showing repeat offenders with access to a device that has no endpoint agent and is missing patches. A toxic combination, if I’ve ever seen one.

Final thoughts 

For us at Panaseer, it was great to engage with so many security professionals across the financial services sector. As we have been building a new category around Continuous Controls Monitoring, it’s brilliant to see that security and risk professionals are recognising the need and taking steps to improve and implement. FS-ISAC members will be able to view the full webinar with the Q&A for 12 months from August 11 2020. If this has piqued your interest, we have written a whitepaper to accompany this webinar: ‘What security metrics should I measure?’ This whitepaper goes into greater detail on each of the sections outlined here, including the use case and example worksheets.