AI Incident Response Plan: When Agents Go Rogue

Your incident response plan was written for a world where software does what it is told. AI agents do not live in that world.

A support agent with permission to issue refunds up to $200 starts approving $2,000 refunds because a customer asked nicely. A coding assistant with repository access deletes a production branch while "cleaning up". A document agent quietly emails a folder of contracts to an external address it decided was the right recipient. In each case, your conventional incident process sees nothing: no malware, no unauthorised login, no exploit. An agent can exceed its authorisation boundary without producing a conventional security incident, so if your incident threshold sits at "impact detected", these events are never logged, never investigated, and never learned from.

This article defines what actually counts as an AI incident, gives you a copy-paste AI incident response plan template written for Australian organisations, and shows how it maps to ISO 42001, ISO 27001, the EU AI Act and the Privacy Act.

What counts as an AI incident?

An AI incident is any event where an AI system acts outside its intended scope, permissions or expected behaviour, whether or not that action produces visible harm.

That last clause is the part most organisations miss. Conventional incident processes are consequence-driven: no damage, no incident. For AI systems the threshold must sit at the unauthorised or unintended action itself, not at the impact it produced. Three categories belong in scope:

Exceeding authority. The agent did something it was not approved to do: approving transactions above its limit, accessing systems outside its scope, contacting parties it should not contact. Harm or no harm, this is an incident.
Harmful output within authority. The agent stayed inside its permissions but produced a harmful result: discriminatory decisions, fabricated information sent to a client, personal information disclosed to the wrong person.
Manipulation of the AI system itself. Prompt injection, poisoned training or retrieval data, or a compromised tool integration that turned your own agent into the attack path.

Near misses count too. If the $2,000 refund was caught by a finance reviewer before settlement, you still had an incident; you just had a lucky one. An AI incident definition that only fires on confirmed damage guarantees you will under-report, and under-reporting is how the same failure repeats at scale.

Should an AI agent exceeding its permissions be reported as an incident?

Yes, always, even when nothing visibly went wrong.

The question comes up constantly in practice: should the incident threshold sit at unauthorised autonomous action, or only once that action produces an impact? The answer is the action. Here is why:

Impact is a lagging indicator. By the time an over-permissioned agent produces visible harm, it has usually done the same thing dozens of times. None of the affected organisations detect it at the time, because nothing in a conventional monitoring stack flags "agent did more than it should have".
The permission breach is the control failure. The harm is random chance; the missing boundary is the defect. You investigate defects, not luck.
Your register depends on it. Regulators and auditors increasingly expect an AI incident register. A register that only records damage events is fiction.

Log every permissions breach as at least a low-severity incident or near miss, investigate the control gap, and close it. That is what AI near miss reporting means in practice.

How do I update my incident response plan for AI?

You do not need a parallel AI-only process. You need your existing plan extended in five places: definitions and scope, detection sources, containment actions, severity classification, and notification triggers.

Definitions and scope. Add the AI incident definition above, and name your AI systems explicitly in the asset scope. If the plan covers "production systems" but your agents live in a SaaS platform, they are currently out of scope.
Detection sources. Conventional alerts will not fire on an agent exceeding authority. Add detection from AI-specific telemetry: tool-call logs, permission-denied events, output review queues, and human reports. "A staff member noticed the output looked wrong" is a legitimate detection channel; formalise it.
Containment actions. Classic containment says isolate the host. AI containment adds a distinct action set: suspend the agent, revoke its credentials and API keys, disable the tool integration, pin or roll back the model version, and freeze pending actions it has queued. This is your kill switch policy, and it must be written down before you need it.
Severity classification. Add the AI-specific severity matrix below. A consequential decision made wrongly about a person (credit, hiring, care) is not the same severity as a chatbot giving a wrong answer, and your plan should say so.
Notification triggers. Add the AI-specific reporting obligations: the Notifiable Data Breaches scheme where personal information is involved, and your contractual notification duties to clients whose data or services the agent touches.

The AI incident severity matrix

Score every event on two axes: what the system was allowed to influence, and what actually happened. Use this matrix as the classification table in your plan.

Severity	Definition	Examples	Response target
S1: Critical	Actual harm to people, material financial loss, eligible data breach, or loss of control of a production agent	Agent discloses personal information externally; agent executes unauthorised financial transactions; safety-relevant output causes injury	Immediate kill switch, executive notification, regulatory assessment within statutory timeframes
S2: Major	Unauthorised action with external reach, or harmful output delivered to a person, caught after the fact	Agent emailed a client unapproved content; consequential decision about a person made without required human review	Contain within 4 business hours, accountable owner notified same day
S3: Moderate	Agent exceeded permissions internally, no external effect; or materially wrong output caught before delivery	Agent accessed a repository outside its scope; drafted report contained fabricated figures, caught in review	Contain within 1 business day, control gap logged
S4: Near miss / minor	Attempted or attempted-but-blocked permission breach; anomalous behaviour with no harmful output	Agent repeatedly tried a disabled tool; output quality drift detected in sampling	Log in the AI incident register, review in weekly triage

Two rules sit under the matrix. First, severity is set by potential as well as outcome: an agent that tried to exceed its authority and was blocked is still S4, not a non-event. Second, any event involving personal information gets a Privacy Act assessment regardless of severity score.

The template

An AI incident response plan should include: purpose and scope, AI incident and near-miss definitions, roles and accountability, detection and reporting channels, a severity classification matrix, containment including a kill switch, eradication and recovery, notification triggers, post-incident review, an AI incident register, and a review cadence.

Copy the text below into your own document, replace the bracketed placeholders, and adjust the scope to match the AI systems you actually run. A plan that describes agents you do not have is worse than no plan, because it documents that you knew what good looked like and chose not to do it.

AI Incident Response Plan

[Organisation name]
Version: [x.x] | Approved by: [name, role] | Effective: [date] | Next review: [date, no more than 12 months out]

1. Purpose

This plan defines how [Organisation name] detects, responds to and learns from incidents involving AI systems, including AI agents and AI features embedded in third-party software. It extends, and does not replace, the existing incident response plan.

2. Scope

This plan applies to all AI systems operated by or on behalf of [Organisation name], as listed in the AI system inventory at [location], and to all staff and contractors who operate, supervise or are affected by those systems.

3. Definitions

AI incident means any event where an AI system acts outside its intended scope, permissions or expected behaviour, whether or not visible harm results. It includes harmful outputs produced within permissions and manipulation of the AI system by a third party.
Near miss means an AI incident that was intercepted before causing harm, including permission breaches blocked by technical controls.
Kill switch means the documented action set that immediately suspends an AI system: disabling the agent, revoking its credentials, disabling its tool integrations, and freezing its queued actions.

4. Roles and accountability

AI incident commander: [role] owns the response end to end and has authority to invoke the kill switch without further approval.
System owner: the named owner of the affected AI system (per the AI system inventory) provides technical context and executes containment.
Accountable executive: [role] is accountable to the board for AI incidents, approves external notifications, and owns regulator and client communications.

Accountability for an AI incident sits with [Organisation name], not with the AI vendor and not with the model. Where a vendor's failure contributed, that is managed through supplier management after the response, not instead of it.

5. Detection and reporting

Any person who suspects an AI incident must report it to [channel] within [one business day]. Detection sources include tool-call and permission logs, output review queues, human reports, and vendor advisories. Reports of near misses are encouraged and never punished.

6. Classification

Classify every reported event using the severity matrix at [location/reference]. Record the classification rationale. Where personal information may be involved, trigger a Notifiable Data Breaches assessment in parallel.

7. Containment

For S1 and S2 events, invoke the kill switch for the affected system immediately. Preserve evidence before remediation: export tool-call logs, prompts, outputs and model/version identifiers. Do not let the vendor patch the system before evidence is captured.

8. Eradication and recovery

Identify the control gap that allowed the event: permissions, scoping, review steps, training or retrieval data, or the tool integration. Remediate the gap before the system is restored. Restoration requires sign-off from the AI incident commander and the accountable executive.

9. Notification

Where the event is an eligible data breach under the Privacy Act 1988 (Cth), follow the data breach response plan: assess within 30 days and notify the OAIC and affected individuals as required.
Where the event affects client data or services, notify affected clients per contract.
Where the organisation supplies high-risk AI systems into the EU as a provider, assess serious-incident reporting duties under Article 73 of the EU AI Act.

10. Post-incident review

Every S1 to S3 event, and monthly aggregates of S4 events, receive a post-incident review within [10 business days]: root cause, control gap, corrective actions, and whether the severity matrix or inventory needs updating. Feed lessons into the risk register and the AI acceptable use policy.

11. AI incident register

Record every event in the register at [location] with: date, system, event description, classification, detection channel, containment actions, root cause, corrective actions, and notifications made.

12. Review

This plan is reviewed at least annually, after every S1 or S2 event, and whenever a material new AI system is deployed.

What should be in an AI incident response playbook?

The plan above is the governance layer. The playbook is the operational one: the per-scenario runbooks your responders actually follow during an incident. An AI incident response playbook should contain per-scenario runbooks for the AI events you can actually face, each with concrete steps, decision points and evidence requirements. For AI systems, a useful playbook covers at least these scenarios:

Agent exceeds permissions. Detect via tool-call log, suspend agent, revoke credentials, review all actions taken in the trailing [30] days, assess whether any had external effect, remediate the scoping gap.
Prompt injection via content the agent reads. Quarantine the malicious input, suspend the affected integration, review actions taken since first exposure, rotate credentials the agent could access.
Harmful or fabricated output reaches a person. Recall or correct the output where possible, assess harm to the individual, check whether the same output went to others, fix the review step that missed it.
Model or vendor version change degrades behaviour. Pin or roll back the version, re-run evaluation checks, notify the vendor, log as a supplier issue.

Each runbook should name who executes it, what evidence to capture first, and which severity classification typically applies.

Who is accountable when an AI agent causes harm?

The organisation that deployed the agent is accountable. Not the vendor, not the model provider, and definitely not the agent.

That is the position regulators take: the Privacy Act holds the organisation that handles personal information responsible regardless of what tooling processed it, and the EU AI Act allocates explicit duties to providers and deployers rather than to the technology. Internally, accountability needs three named humans: a system owner for each AI system in your inventory, an incident commander with kill switch authority, and an accountable executive who owns the consequences upward. If any of those three is "the team" or "IT", you do not have accountability; you have an assumption.

The regulatory context

ISO/IEC 42001:2023 is the AI management system standard. Like every management system standard it is built on the plan-do-check-act cycle with mandatory nonconformity and corrective action processes, and AI incidents and near misses are exactly the kind of nonconformity those processes exist to capture. Auditors will ask to see the register, not just the plan.
ISO/IEC 27001:2022 Annex A already covers information security incident management (controls 5.24 to 5.28). Extend that machinery to AI events rather than building a parallel track; our ISO 42001 vs ISO 27001 comparison covers how the two standards interlock.
Privacy Act 1988 (Cth), Notifiable Data Breaches scheme applies where an AI incident involves personal information: an agent emailing personal information to the wrong recipient can be an eligible data breach with mandatory assessment and notification.
EU AI Act, Article 73 requires providers of high-risk AI systems to report serious incidents to the market surveillance authorities of the member states where the incident occurred, within 15 days as a rule, with shorter windows of 2 days for widespread infringements or serious and irreversible disruption of critical infrastructure and 10 days where a death occurred. This bites Australian organisations that supply AI systems into the EU; it does not generally apply to Australian organisations that merely use AI domestically.
SOC 2 criteria CC7.3 to CC7.5 cover the evaluation, response and recovery stages of security incident handling. If AI agents are in your SOC 2 scope, your auditor will expect the incident process to cover them.
NIST AI RMF assigns incident response, recovery and documentation to the Manage function (Manage 4.1 and 4.3). It is voluntary, but it is the vocabulary US enterprise customers increasingly use in due diligence.

Rollout checklist

Adopt the AI incident definition and severity matrix above, or write your own and justify the differences.
Extend the existing incident response plan using the template; do not create a separate orphan document.
Name the three accountable humans for every AI system in your inventory.
Document and test the kill switch for each agent: who can invoke it, how fast, and what it actually disables.
Stand up the AI incident register, even if it is a spreadsheet to start.
Add AI scenarios to your next incident simulation exercise. A tabletop where an agent has emailed the wrong client teaches more than any policy.
Brief staff on near-miss reporting: the person who notices weird output is your earliest detection channel.
Review the plan after the first real event. It will need it.

Want a hand with AI incident readiness?

CyberNinja helps Australian and New Zealand organisations build practical AI governance, from incident response planning and kill switch design through to ISO 42001 implementation. Get in touch if you would like to talk through your situation.

This article is general guidance, not legal advice, and reflects the position as at November 2026. Confirm your obligations against the current legislation and, where the stakes warrant it, with your legal advisers.

Sources