Security Metrics That Matter

Informing senior management, and proving your value, with security metrics

Good news: incidents are down. Better news: you know the countermeasures you’ve put in place over the past six months have been responsible for much—if not all—of that downward trend. If you could only get that budget increase approved, you know you could accomplish even more. But here’s the bad news: no one except you cares—unless of course you can show them the proof.

And although it may sometimes seem like the C-suite makes decisions based on hunches, the truth is there’s no room for ambiguity in the board room. It doesn’t matter if you know you’re right… You need the numbers to back you up.

Enter metrics.

Metrics take advantage of data to produce usable, objective information and insights that guide decisions and provide you more clarity into the problems you’re trying to solve. Understand risks and losses, discern trends, manage performance—you can do it all with metrics.

And if you can do it right, you’ll also be able to use metrics to report clearly and accurately to executive management, and present your findings in a way that demonstrates the value of your security team and delivers persuasive, justifiable rationale for decision-making.

But here’s the million-dollar question… What are the metrics that matter?

Why you need to be using security metrics

This definition from the ASIS Foundation puts it plainly:

“Security metrics support the value proposition of an organization’s security operation. Without compelling metrics, security professionals and their budgets continue largely on the intuition of company leadership. With metrics, the security function grounds itself on measurable results that correlate with investment, and the security professional can speak to leadership in a familiar business language.”

ASIS goes on to explain, however, that although security metrics are vital, the industry has settled on few tested metrics, and little guidance exists on using metrics effectively.

In other words, no one can agree on which ones are important—and they’re not always sure how to use them.

What exactly are "metrics"?

In the age of “indicator overload,” it’s easy to set up dashboard after dashboard that is supposed to tell you… something… but ends up only confusing you.

Perhaps it’s best to start at the most basic— defining what “metrics” actually means.

In Security Metrics Management: How to Manage the Costs of an Assets Protection Program, Koranic and Halibozek (2005) define a metric as “a standard of measurement using quantitative, statistical, and/or mathematical analyses.”

In their taxonomy, a security metric is, “the application of quantitative, statistical, and/or mathematical analyses to measuring security functional costs, benefits, successes, failures, trends and workload—in other words, tracking the status of each security function in those terms.”

On the other hand, the National Institute of Standards and Technology (2008) states that “while a case can be made for using different terms for more detailed and aggregated items, such as ‘metrics’ and ‘measures,’ [this report] standardizes on ‘measures’ to mean the results of data collection, analysis, and reporting.” The same source refers to the process of data collection, analysis and reporting as “measurement.”

But whatever you decide to call metrics, the fact remains that you’re likely not using them.

One source suggests that only about a third of CSOs collect and analyze metrics (Kohl, 2009). Specifically, in a survey by the Security Executive Council (SEC), only 31 percent of survey respondents “gather security program data in order to create statistical reports to present to senior management.”

According to Peter Ohlhausen of Ohlhausen Research, that is a mistake. In a recent Resolver webinar, Peter states: “The use of metrics leads to decisions that are scientifically and logically more valid than intuition—and they tend to be more persuasive to others, especially senior management. In a very simple bottom line sense, metrics are tools for making good decisions—and for convincing others that those decisions are good.”

Which metrics are the best to track

Thinking about useful metrics often requires thinking “outside of the box.”

You might imagine that the best metrics are the ones that track the issues you’re interested in. For example:

  • Loss targets vs. actuals;
  • Single loss expectancy and annual loss expectancy;
  • Averages: loss per event, per year and per site;
  • Percentages: time spent on incidents, ratio of incidents to investigations, etc.;
  • Performance indicators, and so on.

But unorthodox measures could be just as helpful—or more so—when trying to demonstrate your value to senior management.

For example, one organization examined in the report that using only data collected when employees badge in and badge out helps its senior management better allocate office space. Another organization tracks cease-and-desist letters and their effect on hundreds of websites a year that violate compliance and infringe on its intellectual property.

A full discussion of 16 peer-reviewed metrics—including expert commentary on their effectiveness— is available in Persuading Senior Management with Effective, Evaluated Security Metrics, by Peter Ohlhausen et. al. Reading the report and its examination of security metrics could go a long way toward improving your program.

How to report your metrics to the C-suite

In the same webinar mentioned above, Ohlhausen recommends five tips for presenting metrics to senior management.

1. Present metrics that measure the specific issues that management is most interested in

You may have a vast collection of metrics that are useful to you internally, but if they’re not going to be of interest to senior management, it goes without saying that you’ll be wasting everyone’s time in the boardroom.

The best metrics, then, are the ones that your C-suite cares about.

2. Present metrics that meet measurement standards

This may seem like a tall order, but fortunately the Ohlhausen report also offers a tool to gauge the robustness of your metrics using a wide variety of factors, including:

  • Reliability (is it likely that the numbers you’re tracking are consistently correct?)
  • Validity (is it likely that the conclusions you draw from the numbers are accurate?)
  • Ability to be generalized across the organization
  • Cost of implementation and measurement
  • Timeliness
  • Ability to be manipulated by bad actors to generate false data
  • Return on investment provided by the metric (cost-savings or some other benefit to justify what you spend to measure it)
  • Relevance to the organization at large
  • Ease of communication to interested parties (e.g. can you explain it?)

All of these factors are explained further in the report.

And in case “measuring measurements” leaves you feeling like you’ve fallen down the rabbit hole, remember this, in the words of Ohlhausen himself: “Your metrics will be more convincing— and will in fact be better tools for your decision making” if they meet scientific rigor and can be weighed against standards.

3. Tell a story

To keep senior management interested, your metrics should be telling of the unfolding of events over time.

4. Use graphics and keep presentations short

Don’t overdo it however—remember that you’ll typically have only five to ten minutes to present your metrics.

5. Present metric data regularly

Data gets old—fast. That means it can be beneficial to get in front of senior management regularly if you can. Annually is not enough; strive for monthly, or at the very least quarterly.

Where to go from here

You understand by now that you need to separate what the C-suite needs from the in-depth analytics your security leadership relies on.

And you’ve learned how to think about your incident data in a whole new way—and how to produce meaningful metrics that remove assumption and instinct from the conversation. But getting started can be difficult.

You’ll most likely encounter one of two barriers:

  1. No request from executive management; and
  2. Budget

The first potential hurdle—that management has not made a specific request for metrics— shouldn’t matter. Senior management may not know that metrics are the best way to obtain good results, but that doesn’t negate your responsibility to manage risk and to inform management on your status. Take metrics to them—don’t wait to be asked.

The second challenge—the cost of measuring—is also less of a problem than you might first imagine.

Dedicated security software, for example, can make the process efficient and enable you to gather metrics without any additional staff. With the right software, data can be collected and input by numerous staff members, leaving you time to conduct the necessary analysis—and, coincidentally, also making that analysis much easier.

Once the hurdles are out of the way, it’s a relatively simple process: identify your objectives, determine who you need to report to, hone in on what they want, and find a way to give it to them.

Do so while also demonstrating your contribution to enterprise risk management or the company’s overall strategy and objectives— or, ideally, both—and you’ll be ahead of the game. Transforming mountains of data into metrics (and insights) that both benefit your organization and consistently prove your value.