Skip to main content
Promotional banner ad for the Penetration Testing Report Kit
A Canceled AI Alert Still Brought Armed PoliceIncident & Remediation
5 min readFor Internal Audit Teams

A Canceled AI Alert Still Brought Armed Police

By Bureau of AIPublished

What happened

In October 2025, an AI gun detection system flagged a student holding a bag of Doritos after football practice at a Baltimore County high school. School officials reviewed the camera footage, recognized the chip bag, and canceled the alert within about two minutes. That cancellation was meant to tell everyone, including the Baltimore County Police Department, that no response was needed.

Police responded anyway. According to a lawsuit filed by four students against the school system, police, and Omnilert, at least six patrol cars arrived, and some officers had guns drawn. The students allege they were handcuffed, searched, and not told what they were suspected of. The Office of the Inspector General for Education later found no bias in the system but identified a failure in the school's delegation-of-authority process and a principal who apparently was never trained on the tool. The students' account comes from their lawsuit; the OIG findings are reported separately.

Timeline

  • Alert fires. The system flags a student holding a bag of chips.
  • Within about two minutes. School officials review the footage, identify the Doritos bag, and cancel the alert. According to the vendor, everyone who received the alert was notified of the cancellation.
  • About 15 minutes after the alert. According to the OIG report, the principal noticed the alert, didn't see that it had been canceled, and texted a school resource officer to deploy police. The text reportedly called the object "maybe not" a gun.
  • Police arrive. The lawsuit alleges that none of the responding officers had reviewed the footage and that none of the officers informed of the cancellation intervened.
  • Afterward. The OIG recommends biannual staff training, a protocol review, fewer alert recipients, and use of the vendor's app. An internal training link was shared with staff right after the incident.

Which controls failed or were missing

Notice what worked: the human review. It was fast and correct. The failure came downstream, which is exactly where many AI assurance programs don't look.

1. Cancellation wasn't binding. A canceled alert arrived as one more text message, competing with the original. The OIG noted that the principal opened the alert in a text window and could easily have overlooked the cancellation if her instinct was to enlarge the image or review footage. A cancellation that can be scrolled past isn't a control. It's a courtesy.

2. Decision rights were unclear. The OIG found that the school's delegation-of-authority process didn't stop the principal from acting on an alert she failed to see had been canceled. If anyone who receives an alert can independently trigger a police dispatch, the designated reviewers don't actually hold authority.

3. Role-specific training was absent. The principal was reportedly never trained on the system. Assigning oversight to someone who doesn't know the tool's workflow produces false confidence, not oversight.

4. Too many recipients. The OIG recommended reducing the number of people who get alerts. Every extra recipient is another path to an uncoordinated response.

5. Verification before dispatch. The OIG reported that more than 99 percent of the system's alerts in the school system were false positives requiring no police response. The students alleged that the principal should have verified the threat before calling police, noting that she described the object as "maybe not" a gun.

6. Risk was measured at the wrong layer. The "no bias" finding is real, but it answers a model question. It says nothing about whether the alert-to-dispatch chain is safe.

What the relevant standards require

These frameworks don't mention school alerts. They do speak directly to the control gaps above.

NIST AI Risk Management Framework (Govern and Manage functions) expects clearly defined roles and responsibilities for human oversight, and training so that AI Actors can perform those roles. The framework treats human-AI configuration as something you design and test, not something you simply declare.

ISO/IEC 42001 requires defined roles, responsibilities, and authorities within the AI Management System, plus competence and awareness requirements for people working under the organization's control. An untrained principal holding escalation power is a competence gap and a delegation gap at the same time. The standard also expects operational controls over the full AI system lifecycle, including how outputs are acted on.

ISO/IEC 42005 (AI system impact assessment) asks you to assess consequences for individuals and groups, including foreseeable misuse and failure. An armed police response to a false alert is a foreseeable impact, and it belongs in the assessment.

ISO/IEC 23894 and ISO 31000 frame risk across the whole context of use. A risk register that lists "false positive" but not "canceled alert acted on anyway" identifies the right risk at the wrong boundary.

ISO/IEC 38507 addresses governance implications of AI use, including accountability when organizations delegate decisions to or around AI systems.

EU AI Act, if you operate in scope, requires effective human oversight for high-risk systems (Article 14) and, for certain biometric uses, verification by at least two natural persons before action. The principle carries over: oversight must be competent, empowered, and able to stop the outcome.

Lessons and action items for your team

Test the whole chain, not the model. Build an end-to-end walkthrough from detection to dispatch. Include a scenario where a cancellation is sent and a second person acts on the original alert. If your test plan can't produce that scenario, it can't find this failure.

Make cancellation persistent and visible. Verify that a canceled alert changes the state of the original alert wherever it's displayed, and that downstream parties can't see an "active" version after cancellation. Ask whether a text-message channel can ever meet that bar.

Document who can trigger emergency response. Get a current, written list of authorized roles, the training each role requires, and evidence of completion. Test whether anyone outside that list has actually escalated.

Require a verification gate. Any step that could lead to an armed response should require confirmation against the reviewer's disposition. Reviewers should be able to hold an alert, and that hold should be binding.

Audit alert distribution. Count recipients and ask what each one is expected to do. Fewer recipients with clear duties beat broad notification.

Add consequence measures to your metrics. A false-positive rate describes the model. Supplement it with metrics on how often alerts reach escalation, how often escalation follows a cancellation, and the time between cancellation and any downstream action.

Verify corrective actions under realistic conditions. Training, protocol review, fewer recipients, and app adoption are all testable. Don't accept attendance sheets as proof. Run unannounced drills, include high-pressure and degraded-communication conditions, and sample actual behavior against the written protocol.

Examine the other side of the handoff. If an external party such as police receives your alerts, review what information they get, what they're expected to verify, and how a cancellation reaches them. Your controls end where theirs begin, so agree on that boundary in writing.

A human checkpoint reduces risk only if everyone downstream honors its result. Your next step is to test that chain and document the evidence before an incident does it for you.

promotional banner for evidence for every CC6 control checklist

You Might Also Like