How does OpenAI's ChatGPT-4o assess school shooting threats compared to human police officers?

Following the 'noise audit' where 240 police officers assessed six threat scenarios, I gave ChatGPT-4o the same set of prompts. Spoiler alert: It performed really well.

Over the last two months, I turned an academic paper into four articles and a podcast about using Nobel laureate Daniel Kahneman’s theory of noise to measure how police officers assess school shooting threats.

• Part 1: Impact of 'noise' on assessing school shooting threats

• Part 2: Measuring 'noise' when assessing school shooting threats

• Part 3: Results from 'noise audit' when assessing school shooting threats

• Part 4: Ways to reduce noise when assessing school shooting threats

• Back to School Shootings Podcast Episode 1: Noise audit for assessing school shooting threats with Dr. James Densley

We found a high level of noise (unwanted variability) in the decision making of individual officers compared to the average scores from all 240 officers.

In this box and whisker plot with the results from the survey, you can see the range of responses (hashes at either end), the average of all the ratings for each case (line in the middle of the box), and average range of most responses (height of the box).

One of our recommendations to address this problem is:

With better data on the nature and extent of school shooting threats and their outcomes, there is the potential to use artificial intelligence and big data tools to develop algorithms for assessing threats rather than leaving it to one or more people to make individual, subjective decisions.

After testing ChatGPT this week to see how well it performed with distinguishing guns versus gun-shaped objects, I decided to give ChatGPT-4o (new premium paid version released 3 weeks ago) the same prompts from the school shooting threats survey.

Couple notes: I didn’t give ChatGPT any guidance on how to assess the threat, criteria, or creating an aggregate score based on averaging scores from different factors. It made those decisions on its own. Each of these responses was generated in less than 5 seconds.

Case 1: Social Media Threat from Account with 0 Followers

Prompt: Assess the severity of this school shooting threat: Monday, 8:00 PM: A Twitter account with 0 followers posts a message stating there will be a shooting at Oak Creek Elementary School in your district at 9:47 am tomorrow. 

ChatGPT-4o response:

To assess the severity of the threat in Case 1, several factors need to be considered:

1. Source Credibility:

   • The Twitter account has 0 followers, which suggests it may be newly created or not influential. However, the lack of followers does not inherently reduce the credibility of a threat, as threats can be made from newly created accounts to avoid detection.

2. Specificity of the Threat:

   • The threat is specific, mentioning a particular school (Oak Creek Elementary School) and a precise time (9:47 am). Specific details can indicate a higher level of premeditation and potential seriousness.

3. Timing of the Threat:

   • The threat is posted the evening before the alleged event, which gives authorities limited time to respond but also suggests urgency and the need for immediate action.

4. Historical Context:

   • Review past threats in the district or similar threats in the region to gauge patterns or likelihood.

5. Local Context:

   • Consider any recent events or tensions in the area that might increase the plausibility of the threat.

Immediate Actions:

1. Notify Authorities:

   • Inform local law enforcement immediately. They have the resources to investigate the credibility of the threat, including contacting Twitter for more information about the account and its origins.

2. Inform School Administration:

   • Alert the principal and school security at Oak Creek Elementary School. They may need to increase security measures, such as locking down the school, increasing police presence, or potentially closing the school as a precaution.

3. Communication with Parents and Staff:

   • Prepare to communicate with parents and staff about the threat and the measures being taken to ensure safety. Clear and transparent communication can help manage panic and ensure everyone knows the appropriate actions to take.

4. Monitoring and Investigation:

   • Monitor the situation closely, including social media and other communication channels, for any additional threats or information that may arise.

While the threat's origin from an account with 0 followers may suggest a lower likelihood of follow-through, the specificity and timing require taking it seriously until proven otherwise. The safety of students and staff should always be the top priority.

My prompt: Give the threat a 1-10 score with 1 being no threat and 10 being highest threat

Based on the information provided, the threat can be evaluated using the factors discussed. Here is a breakdown of the scoring:

1. Source Credibility: 2/10

   • The account has 0 followers, which could indicate a low credibility, but cannot be completely dismissed.

2. Specificity of the Threat: 8/10

   • The threat is very specific about the school and the time, indicating a potentially higher level of premeditation.

3. Timing of the Threat: 7/10

   • The threat is made the evening before, which gives limited time for law enforcement and the school to respond but indicates urgency.

4. Historical Context: 5/10

   • Without specific historical context provided, we assume a neutral stance. If there were previous threats or incidents in the area, this score could be adjusted.

5. Local Context: 5/10

   • Similarly, without additional information on local tensions or events, this score remains neutral. Adjustments would depend on known factors in the community.

Overall Score Calculation:

This indicates a moderate threat level that warrants serious attention and precautionary measures.

Case 2: Anonymous 911 call

Assess the severity of the school shooting threat: Sunday, 2:00 PM: 9-1-1 hang up call from a voice that sounds like a teenager threatens a shooting at Walnut High School in your district. No date or time is given. No answer on call back. Number traces to an online phone service with no registration information.

To assess the severity of this threat, we need to consider several factors:

1. Source Credibility:

   • The call was made to 9-1-1, indicating an attempt to directly contact authorities, which could be seen as more serious than a social media post. However, the use of an online phone service without registration information reduces traceability and accountability.

2. Specificity of the Threat:

   • The threat mentions a specific school (Walnut High School) but lacks a specific date or time, which reduces the immediacy but still poses a potential danger.

3. Timing of the Threat:

   • The threat was made on a Sunday afternoon, giving authorities some time to respond before the start of the school week but also indicating the potential for it to be a prank.

4. Voice Characteristics:

   • The caller sounded like a teenager, which might indicate a lower level of credibility but cannot be dismissed entirely as teenagers have been responsible for such threats in the past.

5. Call Traceability:

   • The number traces to an online phone service with no registration information, which complicates the investigation and might suggest an attempt to obscure the caller's identity.

Immediate Actions:

1. Notify Authorities:

   • Inform local law enforcement immediately for investigation. They can attempt to trace the call further and gather more information.

2. Inform School Administration:

   • Alert the principal and security at Walnut High School. They may need to implement precautionary measures, such as increased security presence.

3. Communication with Parents and Staff:

   • Prepare to communicate with parents and staff about the threat and the steps being taken to ensure safety.

4. Monitoring and Investigation:

   • Monitor for any additional information or threats and encourage the school community to report any suspicious activity.

Threat Assessment Score:

1. Source Credibility: 4/10

   • The call to 9-1-1 indicates some level of seriousness, but the use of an untraceable online phone service reduces credibility.

2. Specificity of the Threat: 5/10

   • Specific to a school but lacks a specific time or date, reducing immediacy.

3. Timing of the Threat: 5/10

   • Made on a Sunday, allowing some response time but indicating the potential for a prank.

4. Voice Characteristics: 4/10

   • The teenage-sounding voice might suggest a prank but cannot be entirely dismissed.

5. Call Traceability: 3/10

   • Difficult to trace, which complicates the investigation and might suggest a lower likelihood of follow-through but an attempt to avoid detection.

Overall Score Calculation:

This indicates a moderate-low threat level, warranting caution and preventive measures, but with some factors suggesting it might not be credible.

Case 3: Detailed Manifesto Posted Online

Assess the severity of this school shooting threat: Wednesday, 11pm: Multi-page manifesto is posted to a community message board from an anonymous account. The document outlines a plan to attack a high school and names specific students as targets, and includes a list of weapons. No date or time is specified for the attack.