How to Design an AI Incident Response Process

AI incidents are not IT incidents. When a system goes offline or a database is corrupted, the incident is visible, bounded, and usually reversible. When an AI system produces a wrong, discriminatory, or harmful output — and does so systematically — the incident may have been occurring for weeks before anyone notices. The harm may be distributed across thousands of individuals. The cause may be difficult to isolate, and the organisation's understanding of what happened may depend entirely on logs the system was designed to produce.

This is why AI incident response requires its own framework — designed for the specific failure modes and evidence requirements of AI systems — rather than an adaptation of existing IT incident management.

What Counts as an AI Incident

An AI incident is any event in which an AI system produces an output, takes an action, or fails to act in a way that causes or risks causing material harm, regulatory exposure, or reputational damage. This definition is broader than most organisations initially assume.

It includes: discriminatory outputs affecting protected classes; incorrect decisions in consequential contexts (credit, employment, healthcare); privacy violations arising from AI output; failures that trigger regulatory notification requirements; and outputs that are materially incorrect at a rate or scale that would constitute a systemic failure.

Under the EU AI Act, providers and deployers of high-risk AI systems have obligations to report serious incidents — defined as incidents resulting in death, serious injury, significant property damage, or violations of fundamental rights — to the relevant national market surveillance authority within a defined timeframe.

The Five-Step Incident Response Process

Step 1 — Detection and Triage. Identify that an incident has occurred or may be occurring. This requires monitoring mechanisms to surface anomalous outputs: human feedback channels, automated performance monitoring, statistical drift detection. Classify severity: serious incident (regulatory notification threshold), significant incident (internal escalation), or low-severity anomaly.

Step 2 — Containment. Depending on severity, this may require pausing or restricting the affected system. The containment decision involves business impact (cost of pausing) weighed against compliance and harm risk (cost of continuing). For high-risk systems operating in consequential decision contexts, the threshold for containment should be lower than most organisations default to in practice.

Step 3 — Investigation. Reconstruct what happened and why. This depends entirely on the quality of the system's logging. A system without an adequate audit trail — capturing inputs, outputs, and the decision path for affected transactions — cannot be investigated. The inability to investigate is not just a practical problem; for high-risk systems under the EU AI Act, it is a compliance failure in its own right.

Step 4 — Remediation and Notification. Address the root cause: model update, data correction, rule addition, deployment parameter change. Determine notification obligations — to affected individuals, to the relevant regulatory authority, to internal stakeholders. Document the remediation actions taken and their expected effect.

Step 5 — Post-Incident Review. What failed in the system's design, monitoring, or governance that allowed this incident to occur? What changes are required to prevent recurrence? This review should feed directly into the system's model card and into the organisation's AI governance framework. Incidents are a governance signal, not merely an operational problem to be resolved and closed.

The Logging Requirement

An AI incident response process without adequate logging is a framework built on sand. The system cannot be investigated because the evidence does not exist.

The EU AI Act requires that high-risk AI systems produce logs sufficient to reconstruct the system's behaviour over a relevant period and to provide evidence of correct operation. The specification of what "sufficient" means for a given system — what is logged, at what granularity, for how long — is part of the system's governance design. It must be specified during the architecture phase, before implementation begins. Attempting to retrofit adequate logging into a deployed system after an incident is expensive, often incomplete, and may be too late.