How to automate security metrics without upsetting your colleagues
March 31, 2022
Security automation brings many benefits, but it will also ruffle some feathers.
The need for greater automation in security metrics and measurement is clear to most people in our industry. Security teams have the luxury of access to an enormous amount of security data, giving insight into every aspect of their environments. Yet the data volume and complexity has become overwhelming, which in turn leads to commonplace control failures.
With greater use of automation, your metrics and measures become more accurate and effective at preventing control failures. It also makes organisations more efficient — our own research shows that security teams spend 54% of their time manually producing reports.
But while the need for automation has become obvious, we’d be wrong to assume that everyone will immediately be on board with it.
Getting over the data hurdles
Automating security metrics and measures is typically seen as a data and technology challenge. And there are definitely big hurdles to overcome in this area.
The fundamental problem is with data quality. How do we get a clean, consistent view of our assets and controls from the silos of raw security data? Bringing together disparate datasets from different tools and teams is a headache in itself, before we consider the complexity of cleaning, joining and analysing it so we can confidently report to stakeholders, all of whom have different requirements.
There are several ways you can attempt to crack this nut, ranging from spending hours in spreadsheets to implementing a Continuous Controls Monitoring platform that uses entity resolution and data science to automate the process. The former option has shown itself unable to scale nor produce the level of data quality required to satisfy oversight bodies and internal stakeholders. The emerging truth is that a ‘built for purpose’ platform is required to meet the level of professionalism required of security metrics today, in the same way ERP replaced manual process for the finance function.
Assuming we’ve managed to solve this sizeable part of the problem, we shouldn’t underestimate the potential fallout when we start sharing new, higher quality measurement data with colleagues. Not everyone will welcome automation and transparency with open arms.
When security automation becomes a people problem
When we increase data quality through automation it brings accuracy and precision, and therefore confidence in our metrics and measures. Security teams become more efficient and can focus on activities that have the most business impact.
But it also challenges existing processes and brings skeletons out of closets. In general, human beings don’t like change that wasn’t their choosing, and they don’t like to be wrong. Our colleagues will be expected to give up metrics and measures they may have spent years building up locally, and this will take some persuasion if they’re to be supportive.
The truth is, marking our own homework is reassuring and comforting — moving from localised, bottoms-up, siloed metric generation to a centralised, consistent, automated process will create disagreements. Inevitably, some people will feel threatened and emotions can rise. Feeling like someone else is judging our performance in a new way, in front of our colleagues, can be extremely stressful, especially when that performance measure could be job threatening.
In my experience, the reaction can be compared to the seven stages of grief. The conversation typically follows this pattern:
- Shock: “What do you mean the numbers say xyz?!”
- Denial: “Your numbers are so different to mine, they can’t possibly be right. We’ve been measuring it like this for years.”
- Anger: “I can’t believe you would try to undermine me like this!”
- Bargaining: “Okay, you might be partly right. How about we keep the bulk of what we do and add some of your new metrics.”
- Sense of loss: “I preferred the old way of doing things. I’m not comfortable in this new world.”
- Depression: “This is too hard. I don’t feel in control. I can’t deal with it.”
- Acceptance: “The new data quality is so much better and I’m making headway in areas where we’ve been stuck for years. Why didn’t we do this sooner?!”
It takes preparation, perseverance and empathy to bring people on this journey and ensure everyone is on the same page at the end. Over the years we’ve found there are ways to make this process easier.
Techniques for harmonious automation
A smooth transition from distributed, siloed measurement to automated, centralised measurement relies on two factors: communication and strong stakeholder management.
Everyone involved in metrics and reporting needs to understand the benefits of moving to a centralised philosophy for data and measurement. When metrics and measures are created by individuals or departments, they are restricted by the skills and data they have available. It also produces outcomes that align to their own objectives and agenda, without necessarily taking into account the organisation’s strategic goals, consistent approaches to measurement or context available in other parts of the organisation.
From personal experience, it’s normal to discover unconscious assumptions in how the data is prepared which makes it look better than reality, and in some cases you might discover deliberate attempts to manipulate data so it looks better to stakeholders. This includes things like changing parameters to mask the length of time it’s taking to fix vulnerabilities, or even descoping shadow IT from measurements.
Ultimately, it needs to be constantly reiterated that automation improves data quality, which is vital for solving the biggest problem in security: control failure. The aim is to create a win-win-win. A win for the area being measured, a win for the measurement program and a win for the company in reducing risk.
One particularly useful tactic is to offer a period of data amnesty. Agree that no metrics will be shared with the wider business for three months, so teams have an opportunity to understand the new data and put a plan in place to improve their numbers. This helps to overcome the initial shock and anger that people feel when presented with metrics that might portray their team or function in a bad light.
Once they’re on board, they have a period of time to get data trending in the right direction, so when the metrics are shared, there’s a success story to report.
The only way forward
By understanding the challenges we’re likely to face when automating metrics and measures, we can plan ahead and make the process as smooth as possible. In complex environments there will always be bumps in the road, but these can be minimised if we give equal focus to the impact on our colleagues alongside the technology and data challenges.
The rewards on offer make automation a necessity. Organisations struggling with poor quality metrics and reporting must realise the urgency of centralising their data and making it available to all lines of defence, or a control failure is inevitable. As adversaries improve their level of automation and data analysis, in particular to find and exploit control gaps, it becomes more likely that a control failure will result in a breach.
Panaseer’s Continuous Controls Monitoring platform automates security metrics and measures, bringing accuracy and efficiency to security reporting. Book a demo to find out more.
