Impact of 'noise' on assessing school shooting threats

Part 1 of my unpublished academic journal article based on Kahneman's theory of noise and the undesirable variability in expert decision making.

Apr 15, 2024

Nobel prize winning economist Daniel Kahneman died two weeks ago on March 27, 2024. I started the research for this paper when I read his book, Noise: A Flaw in Human Judgement, in 2021. His most famous book, Thinking Fast and Slow, had a profound impact on how I think about problems. His last long form interview was on Freakonomics Radio about Noise.

25 years after Columbine is the time to think critically about school security. Using the lessons from one of the brightest minds in behavioral economics is a place to start.

Noise is the unwanted variability in decisions made by experts who are looking at the same information. Translation: Two people see the exact same thing differently.

Common examples are:

Insurance underwriters look at the same client and decide on different premiums.
Judges hear the same details of a fictional criminal case and give different length prison sentences.
Doctors look at the same test results and give different diagnoses.

When Kahneman asked CEOs to estimate the amount of noise (variability) in decisions made at their companies, they estimate about 10%.

Exclusive and Personal, Daniel Kahneman Talks to Us About Noise | Alliance for Decision Education — More about Noise

When noise was measured by formal audits, the variability in premiums quoted by different underwriters at a major insurance company was higher than 50% which was costing the company billions of dollars. Think about how profoundly different the outcome to a criminal trial is if one judge grants probation while another sends a person to prison for a decade.

Locking down a school, dispatching dozens of officers, and parents leaving work early after they get texts from their kids about a school shooter all have a huge economic (and emotional) costs every time a school shooting threat is made.

Kahneman found that noise is present during any situation where people use judgement. This includes school officials and police officers who are evaluating the severity of a school shooting threat.

Nobel-winning behavioral economist Daniel Kahneman, who upended his field, dies at 90 | The Times of Israel

Because of variability, noise turns important decisions into lotteries. It’s important to understand that noise is not about individual biases, it is a measure of the aggregate differences within a group. Because of this, a single person or individual decision can’t be evaluated. Noise can only be measured by looking at how multiple people make the same decisions and then evaluating the results with statistics.

Ironically there is noise in the academic peer-review process too. Depending on which reviewers are assigned to a paper, they might publish or reject the same manuscript. This series of Substack articles is based on a survey that I conducted after reading Kahneman’s book in 2021. I cowrote a subsequent academic journal article on the survey results with Dr. Jillian Peterson, Dr. James Densley, and Gina Erickson from The Violence Project.

The article was rejected by peer-reviewers because the survey methodology wasn’t a random sample. I used a listserv of school police officers’ email addresses and they had to opt in to participating in the study which can cause a selection bias. Maybe that matters, maybe it doesn’t. In Part 2, I’ll post the fictional school shooting threat scenarios and survey questions so that you can judge the validity of the methods and findings.

Regardless of the representativeness of the participants, the results point to a significant amount of noise that exists when police officers evaluate school shooting threats. To evaluate noise at an insurance company, you don’t need to randomly sample employees. You just need to gather matching sets of data from employees who are doing the same tasks and expected to produce similar results.

To honor Kahneman’s legacy, I’ve decided to publish the findings here with permission from my co-authors.

Noise Audit of School Threat Assessors

Recognizing and evaluating noise is important for any decisions that you would expect to be made consistently. You would expect that school police officers in Idaho and Florida both evaluate the severity of a school shooting threat in generally the same way. If both officers look at exactly the same information and then one officer decides to immediately lockdown the school (10/10 severity) while the other dismisses it as a hoax (1/10 severity), this inconsistency shows a problem.

A black and white drawing depicting a group of police officers engaged in a decision-making process. The scene shows four police officers, two male and two female, standing around a table looking at a map. They are in uniform, with badges visible, and showing expressions of concentration and discussion. The drawing is detailed, capturing the intensity of their discussion and the environment around them, which includes radios and other police equipment.

To conduct a noise audit, a group of experts are given exactly the same information to independently analyze and the differences in how they decide the outcomes are measured. In this study, school police officers were each given the same set of fictional school shooting threat scenarios to evaluate.

Short summary of the study

School shooting threats come in many forms—direct communication, social media posts, written plans and drawings, ominous messages scrawled on the bathroom wall, hearsay, and anonymous 911 calls. Some are real, while others are hoaxes. There is no clear playbook for how to evaluate and respond to these threats.

This study reports findings from an anonymous online survey of 243 law enforcement practitioners—including 136 School Resource Officers—directly responsible for assessing threats in US schools. On a Likert scale from 1 (no threat) to 10 (high threat), they rated the severity of six fictitious vignettes constructed from common themes identified in real school shooting threats documented by the K-12 School Shooting Database. The survey documented:

1-10 severity score for each scenario
Menu of response options for each scenario including lockdown, school closure, arrest, and referral to mental health or social services
Narratives comments on how they perceived the threat
Explanation of why those chose the response options
Participant demographics including gender, age, years of experience, rank, experience responding to actual school shooting incident, children/grandchildren in k-12 schools, school district size, city, and state

Findings showed that parenthood, geography, and prior experience are correlated with how severely threats were perceived. Most importantly, there was wide variation in responses—known as “stable pattern noise”—which suggests individual experts with similar training consistently assess identical criteria differently.

For five of the six scenarios, scores ranged from 1 to 10 with averages between 3.5 and 7.5. This means that some individuals assessed a scenario as “10” (high threat) while the entire group of 243 officers collectively rated it as 3.5. Inversely, some officers rated a scenario as “1” (no threat) while the aggregate group score was 7.5. This is a huge range of wildly inconsistent assessments!

These findings indicate a high level of system noise in assessing school shooting threats.

Details of the study

This research paper has too much information for one Substack article so I’ve broken it down into four parts that will be published over the next three weeks (unless there is a major incident that requires timely and focused analysis).

To give you a hint about how this story ends, the wisdom of crowds—or aggregating decisions—is the most effective way to reduce noise. The good news is doing this is a less complicated solution than it sounds.

I hope you enjoy learning about my study and the concept of Noise. Kahneman's Nobel Prize winning work had significant impacts on how economists think about behavior. I hope that measuring noise can help make schools safer places where police can consistently assess the situation and make the best decisions when threats are made.

David Riedman is the creator of the K-12 School Shooting Database and a national expert on school shootings. Listen to my recent interviews on Freakonomics Radio, New England Journal of Medicine, and Iowa Public Radio the day after the Perry High shooting.

School Shooting Data Analysis and Reports

Discussion about this post