Build vs buy framework for Continuous Controls Monitoring

To build, or not to build, that is the question.

“Build vs buy” is an increasingly important decision for companies looking to implement a Continuous Controls Monitoring (CCM) solution. Organizations large and small find themselves with a growing number of standards and compliance regulations that require disparate systems and processes in place to support them.

The daunting task of managing large cyber risk inventories manually, via spreadsheets and homegrown databases, is a decidedly futile challenge for already overwhelmed security managers. Leading analyst Jie Zhang of Gartner highlights CCM as a solution: “To address this issue, leaders in security and risk management must adopt technology such as Continuous Controls Monitoring. It automates the monitoring of cybersecurity controls’ effectiveness and aids in gathering relevant information in almost real-time.”

Inevitably, companies find themselves faced with a decision. Is it more beneficial to build a custom CCM management system to meet specific needs? Or purchase an existing commercially available tool with comprehensive functionality already built in?

If a cybersecurity team wants to build their own CCM function, there are a lot of difficult challenges to address. Let’s explore the benefits and challenges of build vs buy in this article.

Recommended decision framework

A generally accepted framework to evaluate build vs buy across costs and strategic value is broken down with core decision factors:

• Risk
• Maintenance
• Upgrades
• Integration
• Time-to-value.

These are some of the challenges that any business faces trying to build their own platform, with answers that are relevant to CCM based on these dimensions. Additional detailed CCM-based program challenges are covered in the next section.

Risk

The Build option introduces risk around developing from scratch without the benefit of numerous deployment learnings, unknown timeline extenders, and the lack of fully staffed QA teams continually keeping solutions current.

Maintenance

Considerations include: Continuous improvement, quality management, knowledge transfer, and problem resolution.

The Buy option explicitly includes maintenance as business-as-usual, where the Build option relies on ongoing maintenance outside of core competencies. But I’d also question whether the Build core competency maintaining solutions is connected to multiple sources and continually adding new technologies.

Upgrade

Considerations include; long term infrastructure required to keep pace with continuous change.

The Build option adds costs for every ongoing upgrade, known and unknown. The Buy option includes this as business-as-usual.

Integration

Considerations include; Total Cost of Ownership, Flexibility, Benefits, and Risk.

External and internal integration remains a constant focus of any CCM project. The Build option carries the burden of building integration with ongoing risk and cost. The Buy option includes economies of scale of learnings across a global installed base, reducing risks and costs.

Time to value

Considerations include; Implementation Timeline, and Total Cost of Ownership.

The Build option carries the risks and costs associated with custom work, ultimately and consistently delaying time to value. While custom work can seem appealing in the planning phase, the ultimate time to value is generally a more appealing driver of a program.

The Buy option replaces custom work with configuration of settings, attributes, reports, and layouts in an efficient manner, reducing time to value. Finally, the Buy option includes a Customer Success Program dedicated to time-to-value throughout the life of the program.

The speed of change calls for expert knowledge

In this section, we will address the challenges more specific to CCM that companies will need to overcome if they want to build effectively. To quote our customers who have attempted the Build option:

“We built it ourselves. It was a monstrous beast. Never again. We couldn’t keep up with stakeholder requirements.”

“Panaseer proposed a solution in 2021 and we decided to build our own. It probably cost twice as much as the Panaseer quote. And we now see, three years on, that we have built an inferior version of what Panaseer had in 2021. Looking at what you have now, the cost and value gap is huge.”

Here are some of the many challenges:

First, the speed of change calls for expert knowledge. It’s also worth noting that the cybersecurity landscape is constantly changing, both in terms of the specific estates of organizations and the wider industry. That’s particularly relevant to the CCM space, because a CCM tool must adapt to those changes. For example, CCM should be able to help an organization with changes such as:

• addressing previously unknown zero-day risks;
• addressing audit findings and weaknesses;
• and changes in infrastructure due to M&A activities.

These changes require CCM best practice knowledge to manage effectively. As such, Panaseer recommends partnering with a firm that not only has out-of-the-box capabilities but has managed services staff ready to assist with the rapid adoption of changes brought forward by new challenges. The Build option is challenging to staff for future scenarios requiring talented subject matter expertise to continually evolve the program.

The second is data ingest. According to ISACA, the most considerable block to making data-driven decisions is “poor quality information”. One of the main goals of CCM is to address this issue. To be effective, CCM needs to take in data from many sources across security and the wider business. We’ve found when we replace Build option solutions, APIs have often been misunderstood or there are issues with permissions and configurations. This leads to the wrong data being ingested and therefore less trustworthy results.

The solutions

And here’s what you do with that data. Most security functions lack dedicated data scientists. But good data science is essential to good CCM. Normalization of data is critical. Specifically, the entity resolution process, which is extremely complex (but made to seem less so in our blog about it).

Reliable, trustworthy data is essential to CCM, otherwise, what’s the point? You need to be able to prove data lineage, you want transparency and inspectability. From the highest-level scorecard to the most granular detail about a single record. You want the same data being used, and trusted, by all stakeholders.

Another part of what makes CCM so powerful across the organization is the way it supports collaboration. Features like the scorecard allow non-technical users to engage with the platform. A Build option solution often struggles to enable non-technical users as it struggles to translate complex cybersecurity concepts into the language of the business.

Adding Panaseer flavor to the decision-making framework

Large organizations will often consider or start to build their own tools when existing vendors’ solutions are perceived to lack specificity, maturity and functionality. In a maturing space such as CCM, we encourage executives to evaluate a build vs buy decision. But, in our experience, at least for Continuous Controls Monitoring, “build” has often been an unsuccessful strategy. Time and again, we have heard companies return a year or two after initial discussions because their in-house project became too expensive or didn’t function as expected. There are many advantages for a buy decision, but even still there are remaining questions.

Our platform

Panaseer’s CCM platform has evolved through the experience of many large global implementations. Through this vast experience, we’ve built a range of features and nuances that wouldn’t necessarily have been identified during the development of in-house tools. This lowers risk and maintenance, offers upgrades and integration, and delivers time-to-value.

Specific to Panaseer, we have further broken down criteria to help guide the decision-making process for this strategic long-term decision.

To build, or not to build, what is the answer?

Panaseer has invested over ten years in infrastructure and capabilities with many iterative improvements. Enterprise scale provides ongoing improvement and the power of a large customer community.

For long-term continuous and automated processes, the build vs buy decision requires careful consideration, and we hope this analysis will assist the decision-making process!

But buy, obviously.