How Do You Measure Performance You Can’t See?

By John Kamensky

October 30, 2012

Law enforcement has it tough when it comes time to measure their performance. Their effectiveness can’t be just measured on what is reported. After all, the goal of most law violators is to not be caught! So how can we measure what we can’t observe?

Cracking the challenge of measuring unobserved behaviors – drug smuggling, tax fraud, counterfeiting, and illegal immigration – is key to developing better strategies and targeting resources in the right places to catch and deter illegal actions.

Interestingly, observes Dr. John Whitley in a new IBM Center report, “the local law enforcement community has been a pioneer in measuring and reporting performance and in using these data to drive strategy development and manage execution.” He points to successes in places like New York City.

But he also observes that “many areas of federal law enforcement do not systematically collect, use, or report basic data on crime rates within their jurisdictions.” This is in part because federal crimes – like tax evasion or drug smuggling – are less likely to be reported than local crimes – like muggings or break-ins.

Dr. Whitley offers a useful tutorial about measurement and statistics. He says that “when outcomes are hard to measure, proxy variables are often used,” such as the number of arrests for various crimes. He notes that “arrests are an important element in the projection of law enforcement that should be measured, but they are not the outcome.” In support, he quotes Washington, DC police chief Cathy Lanier: “Arresting people is not a measure of success. Less crime is a measure of success.”

Five Methods for Estimating Unobserved Events

Dr. Whitley says that, in order to “effectively manage federal law enforcement activities, officials and policy-makers in charge must have an idea of what is happening.” He describes five statistical methods for estimating data on unobserved events:


Dr. Whitley cautions that getting the measures right is critical. If the wrong measures are selected, then it could result in decision-makers making bad decisions or wasting money. But he says the law enforcement community has some inherent advantages because their performance management systems are “relatively straightforward,” even though technical challenges exist because “outcomes are often unobserved and cannot be directly measured.” But if approached thoughtfully, “it is possible to bring the radical reforms that have been seen in state and local law enforcement to the federal level.”

(Image via Novelo/

By John Kamensky

October 30, 2012