The Incident Commons: AI Enters an Aviation-Style Safety Era

What just happened

Europe just turned AI failure into public infrastructure. On September 26, 2025, the European Commission released draft guidance and a reporting template for what the AI Act calls serious incidents, and opened a consultation through November 7, 2025. The guidance prepares providers of high risk AI for mandatory reporting that becomes applicable in August 2026. It clarifies definitions, explains how to file, and sketches how incident data should be reused across regimes. This is the first substantial outline of how a structured, repeatable AI postmortem could work at scale in a major market. You can read the announcement and download the template directly from the Commission draft guidance.

The reporting rule sits inside Article 73 of the AI Act. The short version is simple, even if the details are not. When a high risk AI system is linked, or likely linked, to significant harm, providers must notify national authorities quickly, follow with a complete investigation, and cooperate on remediation. For widespread or very severe incidents, timelines get tighter. The guidance also points to alignment with international initiatives and stresses that providers should be preparing their processes now, rather than waiting for the deadline.

The shift from safety talk to safety plumbing

This is not another trust mark or pledge. It is plumbing. Aviation learned decades ago that safety improves through structured learning loops that capture near misses, defects, and design mistakes, then feed those lessons back into procedures and equipment. The closest United States analog is NASA’s Aviation Safety Reporting System, a confidential pipeline that turns frontline mistakes into community knowledge. If you want to understand the template for this mindset, read the NASA safety reporting overview.

AI is now entering that era. The EU guidance makes failure a shared object. Harm is no longer a public relations event, it is a structured data record. Once the market can read that record, failure starts to behave like a price. Vendors will be nudged to compete on real reliability, not just on benchmark charts or staged demo moments.

Why mandatory reporting beats voluntary trust marks

Voluntary trust marks reward marketing. They certify intent and process snapshots, but they rarely surface what actually breaks in production. A badge does not tell you whether a customer support agent hallucinated refund policies last Thursday, or whether a ranking system quietly downgraded a minority dialect last month.

Mandatory incident reporting does three things that trust marks cannot:

It produces negative signals. A count of real incidents, with severity, affected users, and time to containment, tells buyers what to avoid or price in.
It creates common vocabulary. A reporting template that forces impact, root cause, and corrective actions into comparable fields lets buyers read across vendors.
It forces postmarket learning. Reliability improves where feedback loops are tight. Reporting closes the loop between failure, fix, and prevention.

If you have been following how governance is shifting from promises to permissions, this aligns with our view in permissions for agentic AI.

Telemetry before policy: black boxes for models and agents

Aviation safety improved when aircraft carried black box recorders. AI needs the same concept, translated into software. That means tamper evident telemetry that allows investigators, regulators, and customers to reconstruct what happened. It also means instrumenting the human and tool context around model calls, not just the prompts.

A practical black box for AI should capture at least the following:

Versioned model and policy identifiers, including training snapshot and fine tuning lineage.
Full prompt and tool call traces, with timestamps, truncation markers, and token counts.
Decision state, not chain of thought. For example, which tools were considered, which was chosen, and why according to a defined policy. Represent this as structured policy events rather than raw internal reasoning.
Context pack inputs, such as retrieved documents, feature flags, and system instructions, with content hashes for privacy and integrity.
Environment metadata, for example rate limits, timeouts, dependency versions, and degraded mode indicators.
Outcome signals, such as user corrections, human in the loop overrides, and downstream exceptions.
Redaction layers, so sensitive user content can be masked while preserving structural integrity of the trace.
Cryptographic signing of logs, plus immutable storage with rotation and retention windows aligned to legal obligations.

Most modern stacks already sit on observability rails. OpenTelemetry based pipelines or application monitoring tools can carry these events if teams agree on a minimal schema. Agent frameworks and custom orchestrators can expose event emitters at tool boundaries. Cloud providers can expose signed trace exports as a first party feature. This instrumentation also pairs well with the ideas in our piece on the software social contract, where agents act across user interfaces and need audit grade traces.

Turning mistakes into market signals

An incident record that is comparable across providers behaves like a price tag on failure. Buyers can ask simple, powerful questions and get answers rooted in real data, not anecdotes:

Incident rate per thousand sessions for my use case over the last quarter.
Median time to containment and median time to full fix.
Recurrence rate after remediation, and the half life of similar incidents across your fleet.
Percentage of incidents discovered by vendor monitoring versus customer reports.

Those numbers will flow into procurement, insurance, and platform policy. Cyber insurers will write endorsements that discount premiums for vendors with low recurrence rates. Marketplaces will promote apps with longer incident free streaks. Enterprises will push vendors toward reliability service level agreements that include safety metrics, not only latency and uptime.

The accident internet for AI

If aviation taught us anything, it is that isolated logs do not create safety. Shared memory does. Here is what an accident internet for AI should look like:

A two tier registry. Tier one is confidential and regulator facing, where full details live. Tier two is public, where de identified summaries, root cause categories, and corrective actions are published.
A common incident schema. Borrow from product safety and cybersecurity to define fields like incident type, system function, harm surface, human in the loop presence, detection channel, mitigations, and verification.
A root cause taxonomy. Move beyond generic model error. Include data defects, retrieval drift, guardrail bypass, tool integration failure, human review gaps, and operator interface flaws.
Safe harbor for rapid notice. Offer reduced penalties for early, good faith reports and near misses, to encourage signal rich reporting rather than legalistic silence.
A registry of corrective actions. Allow anyone to search for how similar incidents were fixed, and whether those fixes held over time.
Privacy and security controls. Separate identities from narratives, publish hashed artifacts, and allow independent audits in secure enclaves when sensitive content is involved.
Interoperability with sector systems. Map AI incident fields to health, finance, or transport reporting, so one postmortem can satisfy multiple obligations without copy paste.

The Commission guidance points in this direction by pairing a reporting template with references to other regimes and international tools. The next step is an implementation community that treats the template like a living standard, tested against real incidents and improved with every cycle.

How this changes vendor incentives

Vendors have historically optimized for benchmark wins and launch moments. In a world with incident led memory, the winning move changes.

Transparent by default. It becomes rational to ship with robust logging, public status pages for safety incidents, and monthly reliability notes. Secrecy looks like risk, not mystique.
Design for investigation. Teams will pick models, vector stores, and toolchains that are easy to trace. Libraries that emit clean events will gain share. Expect a boomlet in agent tracing products.
Product management meets safety engineering. Roadmaps will include not only features but quantitative reliability targets and the instruments to measure them.
Marketing aligned to reality. Sales will lean on independent incident histories and third party attestations rather than bespoke case studies that cannot be compared.

If your agents depend on memory of past interactions, make sure your incident design respects user consent and retention limits. The patterns in the opt in memory divide are essential when you are logging context and outcomes.

What buyers should ask for now

You do not need to wait for August 2026 to get real benefits. Add three clauses to your next request for proposal or renewal.

Incident readiness clause. The vendor must maintain an incident response playbook that covers AI systems, with named roles, a unified incident timeline, and a process to issue preliminary notices within a fixed number of hours after suspected harm.
Telemetry and access clause. The vendor must capture agent traces with the data points listed earlier, store them in a tamper evident system, and grant the customer audit rights for incidents that affect that customer or materially similar customers.
Reliability service levels. Define thresholds for incident rates, time to containment, and recurrence. Tie credits, holdbacks, or termination rights to missing those targets.

Add two reporting expectations:

Publish de identified postmortems for incidents above a defined severity, including root cause, fix, verification steps, and time to prevention.
Participate in the public registry once it exists, and commit to backfilling summaries for significant historical incidents.

What builders should do this quarter

If you develop or deploy AI systems, this is the preparation list to start today.

Inventory your agents. For each system, map inputs, tools, decision points, and human in the loop steps. You cannot log what you cannot see.
Implement a minimal incident schema. Use a consistent set of fields across teams, even if the schema is simple at first. A rough but consistent schema is better than none.
Add event emitters. Emit structured events at policy decisions, tool calls, guardrail evaluations, and human overrides. Start with a low rate of trace sampling if cost is a concern, then scale up.
Set retention and signing. Pick a retention period that satisfies likely obligations and sign logs to make tampering obvious.
Run a fire drill. Pick a past bug or hallucination, pretend it is a serious incident, and run the playbook. Measure time to detection and notice, then fix the gaps.
Prepare a redaction layer. Build or buy tools that can mask personal or confidential content in logs while preserving structure and hashes for correlation.

Spillovers into the United States

The United States excels at sectoral rules. Expect incident reporting to appear first where it already exists for safety critical systems.

Health. The Food and Drug Administration expects postmarket surveillance for medical devices, including software. As clinical AI moves from decision support to more autonomous workflows, incident led reporting will blend into those channels.
Finance. Banks live with model risk management and supervisory exams. A shared incident vocabulary will make it easier to compare vendors and to set capital or provisioning expectations for model failures that cause financial harm.
Transport. Advanced driver assistance and autonomous vehicles report disengagements and crashes in some jurisdictions. Expect harmonization of agent telemetry with those reports.
Consumer protection. The Federal Trade Commission can treat deceptive or unfair AI claims as consumer harm. Incident data will give it faster signals and clearer remedies.

A federal safe harbor would accelerate adoption. A program styled after NASA’s confidential reporting, with liability protections for prompt and complete reports, could boost volume and quality without waiting for a single cross sector law. State attorneys general and sector regulators can pilot it while Congress debates broader rules.

A concrete picture of public postmortems

Imagine reading a two page incident summary for a widely used enterprise agent. It might look like this.

System. Customer support agent, version 1.9.3, fine tuned on retrieval augmented data from January 2025.
Incident. On September 12, 2025, the agent issued unauthorized refunds to 413 customers due to a tool selection error triggered by ambiguous policy language.
Impact. Direct financial loss of 1.2 million United States dollars, with secondary impact that included delayed service for 8 percent of users that day.
Detection. Anomaly detected by vendor monitoring within 14 minutes due to a spike in refund tool invocations per session.
Containment. The tool was disabled within 23 minutes and human agents took over. A full fix shipped in 48 hours.
Root cause. Retrieval drift introduced stale refund rules, the policy evaluator lacked a negative constraint for maximum refunds per session, and the human in the loop step was bypassed because the agent was incorrectly tagged as low risk for returning customers.
Fix. Added hard caps, strengthened the policy evaluator with a new negative constraint, retrained the retrieval index daily, and restored a human checkpoint for any refund over 100 United States dollars.
Verification. No recurrence in 30 days across 2.1 million sessions, with a 98 percent reduction in false refund attempts.

That is what a useful incident summary looks like. It is specific, it exposes mechanisms, and it teaches others how not to repeat the mistake.

The business upside no one talks about

Reliable incident telemetry reduces the cost of shipping new features. Teams hesitate to deploy agent skills when they cannot predict failures or reconstruct them. With a black box trace, you can ship, watch for new error modes, and roll back with evidence. Your regression tests become richer because they copy real incident patterns. Your operations team resolves tickets faster, because they can search for the same root cause in minutes instead of days. Your customers feel safer because they know what you will tell them if something goes wrong, and when.

This is how engineering maturity compounds. The first postmortem is painful. The fifth creates a library of fixes. The fiftieth becomes a set of playbooks that new teams can follow, and it becomes a selling point. Over time, reliability stops being a mystery and becomes a measurable discipline.

A note on confidentiality and competition

Two fears often block incident sharing. One is that incidents reveal trade secrets. The other is that bad actors will learn to exploit failure modes. Both fears are manageable with design.

Summaries can protect secrets. De identified users and redacted proprietary context still convey the learning that matters. They show causes and fixes without exposing confidential materials.
Delay lowers exploitation risk. Publishing older incidents lowers the chance of copycat exploitation while still teaching the community. A ninety day delay for public summaries is a sensible default in many cases.
Use trusted intermediaries. A neutral third party can hold sensitive details. Aviation uses trusted intermediaries and secure repositories while still publishing lessons. The same model can work here.

The bottom line

The EU’s draft guidance does not only set up a compliance chore. It invites the industry to build an incident commons. If we treat postmortems as shared infrastructure, the market will reward vendors who show their work and improve quickly. The result will look a lot like aviation after systematic reporting took hold. Fewer surprises, faster learning, and a culture where mistakes are turned into memory rather than marketing crises.

This is the moment to lay the rails. Adopt black box telemetry, write public postmortems, and wire incident metrics into your contracts. When the rules take effect, you will already be fluent. More important, your systems will be safer because they will be learning in public.