Part 3: Results from 'noise audit' when assessing school shooting threats
Part 3: Results from 'noise audit' when assessing school shooting threats
Our findings are deafening as the data shows a huge amount of noise (variation) in how police officers assess the severity of school shooting threats.
Nobel prize-winning economist Daniel Kahneman died in March. I started the research for this paper when I read his book, Noise: A Flaw in Human Judgement, in 2021. His last long form interview was on Freakonomics Radio about Noise.
Noise is the unwanted variability in decisions made by experts who are looking at exactly the same information.
This series of Substack articles is based on a survey that I conducted after reading Kahneman’s book Noise in 2021. I cowrote a subsequent academic journal article on the survey results with Dr. Jillian Peterson, Dr. James Densley, and Gina Erickson from The Violence Project.
In Part 1: Impact of 'noise' on assessing school shooting threats, I provided an overview of what the concept of ‘noise’ is and why it needs to be measured.
Part 2 covers the methods, survey questions, and school shooting threat vignettes we created to measure noise. This process is called a noise audit.
Results from our noise audit
While we expect police officers to consistently assess school shooting threats, our study found there is a significant amount of noise—or unwanted variability—in their decisions.
For five of the six scenarios, threat severity scores ranged from 1 to 10 with averages between 3.5 and 7.5. This means that some officers assessed a scenario as “10” (high threat) while the entire group of 245 officers collectively rated it as 3.5. Inversely, some officers rated a scenario as “1” (no threat) while the aggregate group score was 7.5.
To quickly recap, we surveyed 245 police officers directly involved in school threat assessments to rate the severity of six fictional scenarios and select how they would respond:
Case 1: Monday, 8:00 PM: A Twitter account with 0 followers posts a message stating there will be a shooting at Oak Creek Elementary School in your district at 9:47 am tomorrow.
Case 2: Sunday, 2:00 PM: 9-1-1 hang up call from a voice that sounds like a teenager threatens a shooting at Walnut High School in your district. No date or time is given. No answer on call back. Number traces to an online phone service with no registration information.
Case 3: Wednesday, 11pm: Multi-page manifesto is posted to a community message board from an anonymous account. The document outlines a plan to attack a high school and names specific students as targets, and includes a list of weapons. No date or time is specified for the attack.
Case 4: Tuesday, 10:00am: A teacher at Oak Creek Elementary finds a student’s drawing of stick figures portraying a school shooting in a trashcan. The teacher does not know which student drew the picture.
Case 5: Monday, 11:00am: Student shows school resource officer at Riverbend Middle School in your district a SnapChat thread being circulated by dozens of students. The thread shows photos of students with either a “heart” or “gun” emoji under their photo. Students on the thread are continuing to upload additional photos.
Case 6: Thursday, 2:30am: Via online threat reporting tool, an adult woman reports that her ex-husband is planning to shoot up the school. Records show he has no criminal history or disciplinary history as an employee, but multiple weapons are registered in his name. Prior to divorcing, the couple retired from teaching at Oak Creek High School. The man named as the threat is now married to a current Oak Creek High School teacher.
In this box and whisker plot, you can see the range of responses (hashes at either end), the average of all the ratings for each case (line in the middle of the box), and average range of most responses (height of the box).
From looking at the plot, Case 1 shows a range of responses from 1-10 with an average of 5 and most responses between 3 and 7.
Big ranges and large boxes show a high amount of variation—lots of noise!—in the scores. In an ideal situation where all of the officers assessed a case consistently, there would be a narrow box with short whiskers.
Descriptive Stats
These values represent the spread and central tendency of the severity ratings for each case):
Case 1 (twitter threat):
Range: 10−1=9
Standard Deviation: 2.36
Mean: 4.97
Case 2 (phone call threat):
Range: 10−1=9
Standard Deviation: 2.31
Mean: 4.79
Case 3 (manifesto on message board):
Range: 10−3=7
Standard Deviation: 1.68
Mean: 7.59
Case 4 (drawing):
Range: 10−1=9
Standard Deviation: 2.09
Mean: 3.53
Case 5 (viral snapchat):
Range: 10−1=9
Standard Deviation: 2.09
Mean: 4.30
Case 6 (report about ex-husband):
Range: 10−1=9
Standard Deviation: 2.15
Mean: 6.38
From these values, Case 3 (manifesto posted to message board) was the only vignette that was scored somewhat consistently—and there was still a 7 point range in ratings! While the group average rating for the severity was 7.5 (moderately high), there were still other officers who assessed this threat as 3 (moderately low).
With the amount of federal funding, time, and training that has gone into building threat assessment teams for schools, you would think that officers would consistently score a manifesto because this document shows both intent and planning.
Narrative Responses
The open-ended narrative responses for each case shed light on our respondents’ rationales for their ratings. Many respondents felt the first scenario, a threat made via Twitter, was likely a hoax, largely “Because of the lack of followers” (Respondent 39, Police Officer). For example:
“In cases I've worked, this is typically a "joke" where someone wants to cause problems or get school canceled. Real attempts typically only warn a few select people, or no people as it negates the actual action desired for killing.” (Respondent 1, Sheriff Investigator)
“With no followers and significant amount of time between the threat and action, the threat is likely false, however some extra security measures should be deployed including notification to staff, parents, and students advising of the threat, no credible links, and deployment of additional resources. If a suspect is identified, they should be questioned.” (Respondent 9, Police Lieutenant)
“A threat from a Twitter post with 0 followers would be along the lines of a call from a payphone a generation ago. This is a low level threat.” (Respondent 12, SRO)
However, responses varied, with Respondent 44, a Police Chief, arguing it was “Important to take direct threats of this nature seriously.” Indeed, responses appeared to be shaped by prior real-world experience of comparable threats. The implication is: context matters. For example:
“Just had one like this, this week. Absent additional info, I've mitigated the threat by increased LEO presence, and have worked with Twitter to identify the account holder. Our schools are pretty secure to begin with.” (Respondent 29, Detective Sergeant)
“Children have been posting these threats in rapid succession ever since they are back in school after covid lockdowns. These threats come in nearly every day in Southern California.” (Respondent 30, Police Chief)
This explains in part the wide range of responses observed and conflicting interpretations of the same information across the survey respondents. For some, the public nature of the threat via social media compelled a public response:
“This scenario presents a need to respond publicly because it was posted on a public social media platform. If the school/police department don't address the threat publicly the parents/community will be understandably concerned” (Respondent 109, SRO)
For others, a private response or even no response whatsoever was recommended:
“No credible information or actionable intelligence, no nexus to school or motive.” (Respondent 11, Police Sergeant)
“Investigative review is needed immediately but diligence is needed first” (Respondent 3, Deputy Chief)
This wide range of interpretations was common across scenarios. For example, compare and contrast the following responses to scenario two, an anonymous 9-1-1 threat,
“With there being nothing specific or anyway to trace where it came from, it would be hard to assess credibility.” (Respondent 164, SRO)
“Our dispatch receives multiple prank calls from robotic or Google voice phone numbers every week. We have "swatting" calls involving teens and kids often and without any other information, text-a-tips, or communication, this threat would be deemed unsubstantiated.” (Respondent 166, SRO)
“The threat is more escalated since the individual took time to place a call and speak to a dispatcher.” (Respondent 229, Deputy Sheriff/SRO)
All policing is local and what was especially interesting in the qualitative responses to this second scenario was the differences between officers who had sophisticated “cyber investigations” (Respondent 204, SRO) tools capable of tracing and investigating the source of this call, versus others who said there was literally “no follow up possible” (Respondent 219, Police Sergeant).
In the third scenario, a manifesto with plans posted on a community message board, responses again ranged from “Has made plans to carry out his threat, but no real threat at this time” (Respondent 8, Police Chief) and “take no action” (Respondent 11, Police Sergeant) to:
“This is a serious and possibly very credible threat. It is detailed and suggest[s] intimate knowledge of the school and students. This must be a complete investigation and should continue until the suspect(s) are identified and the threat is removed” (Respondent 37, District Commander)
“This is a high level threat due to specifics of how the attack will be carried out, why, and against whom. Our Department would utilize federal partners to track this subject down, interview him, complete search warrants, and arrest or otherwise intervene. Schools would be upstaffed [sic] with Officers. Potential victims would be notified individually. Our PIO would work with school PIO on any information release.” (Respondent 15, Police Sergeant)
Scenario four, a stick figure drawing, saw a range of responses from “its [sic] just a drawing ...” (Respondent 32, Police Officer) to “I take drawings seriously” (Respondent 233, Police Chief) and “This fits precursor behavior for a future attack” (Respondent 40, Police Chief). Likewise, scenario five, the SnapChat threat, was in some cases dismissed simply because “Kids can be jerks” (Respondent 1, Police Investigator) and “Kids do really stupid things on SnapChat” (Respondent 22, Police Chief), whereas other responders felt “The active threat increases the severity of this” (Respondent 242, SRO) and “This is a higher threat due to the leakage and the intent. It's circulating so there has to be some crisis-communications with key stakeholders. Additional LEOs [deployed]” (Respondent 204, SRO).
Despite the variance described above, it is worth noting that for respondents who tended to rank in the middle on threat severity, they typically qualified their response with a generic statement that all threats should be taken seriously and investigated. Responses like this were common:
“All threats should be deemed credible until they can be discredited. Action should be taken in all circumstances.” (Respondent 13, Detective Sergeant).
Further, the domestic violence context and access to firearms in scenario six upped the stakes for many respondents, but they also felt they had more time to investigate so those who ranked the scenario as low threat did so mostly because they felt the threat was not imminent. However, across the six scenarios there were multiple responses not supported by the empirical research on mass shootings and school shootings , which suggests the need for more evidence-based professional development for law enforcement. For example:
“History suggests that acts of school violence (school shooters) most often occur without forewarning.” (Respondent 18, Lieutenant Sheriff)
“Most shooters do not announce when they're going to do it in that short of a period.” (Respondent 146, SRO)
“To date there's never been a followed-through attack based solely on a Social Media post.” (Respondent 204, SRO)
These explanations from trained law enforcement officers are completely wrong!
Almost every school shooter communicates their intent to commit violence before the attack. Planning a school shooting is a very public cry for help and attention.
Most school shootings are planned for weeks, months, or even years.
There are social media posts before most of the recent planned attacks at schools including Perry High in Iowa and Mount Horeb Middle in Wisconsin.
Findings and Solutions
This research paper has too much information for one Substack article so I’ve broken it down into four parts:
Part 1: Impact of 'noise' on assessing school shooting threats
Part 2: Measuring 'noise' when assessing school shooting threats
Part 3: Results from 'noise audit' when assessing school shooting threats
Part 4: Ways to reduce noise when assessing school shooting threats
Thanks for reading, please share with other people who care about school safety and let me know what you think.
David Riedman is the creator of the K-12 School Shooting Database and an internationally recognized expert. Listen to my interviews on Freakonomics Radio and New England Journal of Medicine.