Cyber Risk Doesn’t Have a Data Problem. It Has an Uncertainty Problem

Spend enough time in cyber risk forums, either online or in person, and a particular pattern becomes familiar. Once a discussion turns to quantifying cyber risk and what that actually requires, its not uncommon to hear someone say: we don’t have enough historical data to assess this properly.

I’ve said it myself in the past and I understand the appeal, but while cyber risk is undeniably adversarial, dynamic, and asymmetric, the “not enough data” claim is often little more than a convenient excuse used to stall an analysis. This isn’t always as a result of bad intent though, but more often than not is because it sounds rigorous and like a reasonable scientific position.

My first concern is the potential abandonment of the quantification effort, which given the nature of cyber risk is precisely the reason we need to use models, scenarios, and probabilities instead of over indexing on historical data.

There is a more egregious issue though, and that is when rather than grappling with the scenario, a risk matrix gets pressed into service and despite their known deficiencies¹ a score is somehow selected.

For many, a number or label on a risk matrix, regardless of how it was arrived at, is sufficient grounds to move the conversation along. It satisfies those more interested in the performance of risk management than the actual management of potentially risky outcomes.

But despite the many ways the not enough data claim is usually delivered, over time I’ve come to realize that it isn’t wrong because we always have more data than we need and need less than we think we do². It’s wrong because it’s usually the entirely wrong diagnosis. Most of the time, this isn’t a data problem. It’s an uncertainty problem.

To be more specific, most risk programs don’t have a working understanding of what kind of uncertainty they’re actually dealing with, and that knowledge gap is a significant issue.

Not all uncertainty is the same

ISO 31000’s definition of risk as the effect of uncertainty of objectives is deceptively simple. Most practitioners accept it without much thought and most risk programs quote it, but it is ultimately ignored in practice.

This shows in the way we commonly talk about uncertainty in a risk context, where we tend to treat it as a single thing – a fog that more data, more controls, or more process will somehow eventually lift. The reality though is more complicated and nuance matters significantly.

Uncertainty broadly falls into three categories.

Aleatory uncertainty is inherent randomness. It is the irreducible variability in a system that cannot be eliminated with more data. While additional data allows us to estimate the distribution of outcomes more accurately, it does not remove the uncertainty itself.

For example, even in a perfectly understood environment, we cannot predict exactly when an attacker will strike, which employee will click a phishing email, or which specific server will fail first. These outcomes are inherently variable. We can estimate their likelihood over time, but we cannot eliminate the variability itself.

Epistemic uncertainty is a knowledge gap. It speaks to the uncertainty that exists because we do not fully understand a system, lack sufficient data, or have imperfect models. Unlike aleatory uncertainty, epistemic uncertainty is reducible in principle through better data, improved models, experimentation, or structured expert judgment.

For example, many organizations cannot answer basic questions such as how long it would take to detect an attacker with internal access, how effective their monitoring controls actually are, or how long recovery would take after a major ransomware event. The uncertainty in these areas is not inherent randomness; it exists because the organization has not measured, tested, or modeled these things.

Ontological uncertainty can often be the most difficult to detect because it doesn’t operate as a gap within the model. It operates as a flaw in the model itself. Unlike epistemic uncertainty which means we lack sufficient information to answer the right question, ontological uncertainty means we aren’t aware the right question exists. We don’t know what we don’t know, and we’re using the model to check the model.

In cyber risk this appears when threat models are built around assumed attacker behavior rather than documented techniques, when control effectiveness reflects design intent rather than operational reality, or when risk boundaries no longer match the actual system being operated. It leads to risk analyses that can be internally consistent but still be wrong and this is not because of bad data, but because the model is answering the wrong question with confidence.

To sum it up:

Aleatory uncertainty relates to variability in the world.
Epistemic uncertainty relates to gaps in our knowledge.
Ontological uncertainty relates to gaps in our model of the world.

It should be clear that addressing these three categories require fundamentally different responses and when it comes to cyber risk analysis, that’s a critical point that needs to be kept in mind. If we apply an epistemic solution to an aleatory problem, we waste analytical effort. If we apply a data-gathering strategy to an ontological problem, we won’t even know what we’re measuring.

And ultimately if our program has no framework for distinguishing between the differences in uncertainty, it is structurally unprepared to respond appropriately to any of them.

The question isn’t whether there is fog. The question is whether you can still navigate.

The data problem is often a misdiagnosis

Going back to the stalled risk analysis shows us why the data problem is often a misdiagnosis.

When an analyst says we need more historical data, that claim carries an implicit assumption that the constraint on the analysis is aleatory uncertainty. What they may be trying to indicate is that they simply have not observed the phenomenon enough times to characterize it’s distribution. While that is sometimes true, in cyber risk it often isn’t.

We can consider the question of threat actor capability here and ask: is our uncertainty about whether a sophisticated attacker can bypass our detection controls aleatory or epistemic?

The answer is primarily epistemic. We often lack intelligence on a threat actor’s specific tooling, their familiarity with our environment, their patience and persistence. Obtaining more breach data from other organizations doesn’t resolve this.

What could resolve it, or at minimum reduce it, is threat intelligence, adversary emulation, and an honest assessment of our detection coverage against documented attacker techniques. This is precisely why the MITRE ATT&CK framework exists, to give analysts a structured way to reason about adversary behavior without needing a sample size.

For any cyber risk analysis effort, clearly understanding the specific type of uncertainty at play and how it can be resolved or reduced matters because the required analytical investment is completely different depending on the type of uncertainty we are dealing with.

Treating an epistemic problem as if it requires aleatory data is a category error and it’s one that routinely justifies inaction. We don’t have enough data quite often tends to become a permanent deferral, because there will never be enough historical cyber incidents in your specific sector, against your specific infrastructure, involving your specific controls configuration.

So rather than asking about the extent of available data, the discerning analyst first asks a different question: what kind of uncertainty is driving this particular gap, and what kind of evidence addresses that specific type?

Sensitivity analysis as a forcing function

Questions like these lead to sensitivity analysis, applied not just to model inputs, but also to the investment of our own time. Before requesting more data, we must be able to demonstrate that new information would materially change the output; the goal is to move past ‘nice to have’ data and prove that more of it would actually change the decision.

As an example, if we vary the historical breach frequency assumption by a factor of two in either direction and the loss estimate barely moves, we’ve proven that frequency isn’t the binding constraint. This demonstrates that the uncertainty we’re noticing isn’t caused by an aleatory gap or by a lack of historical data. Instead, it signals that something else is driving the risk and that ‘something else’ is likely where our focus should actually go.

Conversely, if we vary the capability assumption about a specific threat actor relative to the resistive strength of our controls and the output shifts dramatically, we’ve now located the actual analytical leverage point. This shows where an investment can actually pay off — in threat intelligence, in red team exercises, in structured expert elicitation of people who understand attacker behavior. Not in waiting for a larger incident database.

It’s important to note that this reframing doesn’t dismiss data needs. Instead, it justifies them as it pivots the standard from “we need more data” to “this specific type of uncertainty is the binding constraint, and this specific evidence addresses it.” The latter is a claim that can be interrogated and challenged. It forces analytical accountability in a way that vague appeals for more data do not.

The governance dimension

The easy misread here is that this is a critique of individual analysts when it definitely isn’t.

Most persons working in risk today were not trained to think through uncertainty types as the dominant frameworks they inherited and adhere to don’t address these distinctions. While ISO 31000 gestures at uncertainty, it doesn’t decompose it and most risk management curricula move quickly past or completely ignore epistemology and into process.

Heat maps, likelihood x impact matrices, red-amber-green reporting and other similar tools that ultimately filled the gap were not designed with any explicit epistemological grounding and as a result they actively obscure the distinctions that matter.

This is in essence a framework design problem as much as it is a practitioner competency problem. A well-structured risk program, one that takes seriously its own governance and oversight function, should ensure this literacy is built in. That means ensuring participants are working from shared definitional frameworks, that analytical methods are matched to the type of uncertainty being characterized, and that claims about data needs are subject to the same rigor as claims about risk levels.

Key to this is appreciating that ontological uncertainty, the kind that often arises from persons ascribing different meanings to the same terms, or from inconsistent interpretive methodology, doesn’t fix itself through better data collection. The fix lies in better governance based around clear definitions, structured communication, and deliberate alignment on what we’re actually measuring and why.

What a better program looks like

The practical implication of all of this is that a risk program that takes uncertainty seriously looks and behaves differently from one that doesn’t.

The first question it asks will always be differentiates instead of opening with how much data is available, it opens with what kind of uncertainty is driving the gap.

This single reframe has the potential to change the entire analytical trajectory in that it determines whether the right investment is more data, better intelligence, adversary emulation, or a harder look at whether the model itself is sound.

It applies sensitivity analysis before requesting resources. Before commissioning threat intelligence, running a red team exercise, or expanding data collection, it demonstrates that resolving that specific uncertainty would materially change the output.

This shifts the standard from “we need more information” to “this specific gap is the binding constraint and this specific evidence addresses it” which is a claim that can now be interrogated and held accountable.

A better program also treats the three uncertainty types as requiring fundamentally different responses rather than variations of the same one. Probabilistic modeling where the problem is inherent variability. Structured elicitation and targeted intelligence where the problem is a knowledge gap. Deliberate assumption challenge and model interrogation where the problem is the framing itself.

Absolutely none of this requires a program to be mature in the conventional sense. It requires it to be honest though, about what it knows, what it doesn’t, and whether it’s even asking the right questions.

That honesty is harder than it sounds, but it’s the precondition for analysis that actually supports decisions rather than just documenting that analysis occurred.

References

1. Michael Krisper, 2021. “Problems with Risk Matrices Using Ordinal Scales” ↩︎
2. Douglas Hubbard, 2023. “How to Make Better Decisions in Cyber Risk Management” ↩︎