Who's Liable When the AI Decides? The Accountability Gap in Autonomous Systems

The accountability gap in autonomous AI is not a hypothetical future problem. It is a live legal, commercial, and governance challenge playing out in courtrooms, regulatory offices, and boardrooms right now. As AI systems move from assisting human decisions to replacing them, the frameworks we built to assign responsibility — product liability law, professional negligence doctrine, corporate governance standards — are straining against a reality they were not designed to handle.

This article examines the accountability gap from first principles: what it is, why existing legal frameworks fall short, what the EU AI Act attempts to do about it, and what governance structures — human-in-the-loop, human-on-the-loop, and emergency controls — mean for how liability is assigned and managed in practice.

The Core Problem: Decisions Without a Deciding Human

Traditional liability frameworks are built on a simple model: a human makes a decision, an action follows from that decision, harm results, the human who made the decision is responsible. This model is not just a legal convenience — it reflects a moral intuition that responsibility tracks decision-making authority. Whoever controlled the choice that caused the harm bears the consequences.

Autonomous AI disrupts this model at a fundamental level. An AI system trained on historical data develops decision-making patterns that emerge from the training process rather than from the deliberate choices of any individual. When that system makes a consequential decision, several actors contributed to the outcome without any one of them making the decision:

• 🏗️
  The AI developer — designed the model architecture, chose the training methodology, curated or selected training data, and made the engineering decisions that shaped the system's behaviour But the developer typically does not control what decisions the system makes in deployment, how it is configured, or what use cases it is applied to.
• 🏢
  The deploying organisation — integrated the AI system into their workflow, configured its parameters, chose to rely on its outputs, and decided whether humans would review its decisions But the organisation did not build the underlying model and may have limited visibility into why it produces particular outputs.
• 👤
  The individual operator — may have enabled or configured an automated decision workflow without fully understanding the AI system's behaviour But they may have been following organisational procedures, not making independent technical choices.
• 📊
  The data providers — whose historical data encoded the patterns the AI learned, including any biases or errors embedded in that history But they typically had no knowledge of how their data would be used to train an AI system, let alone the downstream decisions it would make.

When a human makes a negligent decision, we can identify the moment of choice and examine whether it met the standard of care. When an AI system makes a harmful decision, there is no moment of deliberate choice to examine — only a probabilistic output from a system that no one designed to produce that specific outcome. This is the accountability gap: the space between the harm that occurred and the identifiable human decision that caused it.

Where Existing Legal Frameworks Fall Short

Three bodies of law are most commonly invoked when AI harms occur: product liability, professional negligence, and general tort law. Each creates significant problems when applied to autonomous AI systems.

Product Liability: The "Defect" Problem

Product liability law holds manufacturers responsible for harm caused by defective products. In theory, it should apply cleanly to AI systems: if the AI system has a defect that causes harm, the developer is liable.

The problem is the definition of defect. Traditional product liability identifies three categories: manufacturing defects (the product was made incorrectly), design defects (the product design itself is unreasonably dangerous), and warning defects (the user was not adequately warned of known risks).

None maps cleanly onto AI harm. An AI system that produces discriminatory outputs may not be "defective" in any conventional sense — it may be working exactly as designed, making predictions that reflect patterns in its training data. A self-driving vehicle that makes a reasonable driving decision that happens to cause an accident may not have a design defect — it may have encountered a genuinely ambiguous situation that a reasonable driver might also have navigated poorly.

"The fundamental problem with applying product liability to AI is that AI does not fail in the way products fail. It does not break. It generalises. When the generalisation is wrong or harmful, there is no manufacturing defect to point to — there is only a probabilistic outcome that, in hindsight, was the wrong one."

— Analysis consistent with EU Product Liability Directive reform discussions, 2024–2025

Professional Negligence: The "Expertise" Problem

Professional negligence law applies when a person with specialised expertise — a doctor, lawyer, financial adviser — fails to meet the standard of care expected of a competent professional in their field. It is the body of law most naturally suited to consequential decisions, because it asks whether the decision was reasonable given what the professional knew and should have known.

The problem with applying this to AI is dual. First, AI systems are not legal persons — they cannot be professionals, they cannot be held to a standard of care, and they cannot be subject to professional sanctions. Second, when a professional uses an AI tool to assist their decisions, the question of where the professional's judgment ends and the AI's recommendation begins becomes genuinely difficult to answer.

A radiologist who uses an AI system to screen scans and misses a tumour that the AI flagged has committed negligence by ignoring a correct recommendation. A radiologist who uses an AI system and trusts a false negative — a scan the AI incorrectly assessed as clear — is in more ambiguous territory. They relied on a tool, as professionals routinely do. Was the reliance itself unreasonable? That depends on the tool's known error rate, the radiologist's training in interpreting its outputs, and whether any professional standard requires additional verification.

Tort Law: The "Causation" Problem

General negligence tort requires establishing that the defendant owed a duty of care, breached that duty, and that the breach caused the harm. Causation is the most problematic element for AI liability. In complex AI systems — particularly those involving large language models, ensemble methods, or multi-agent pipelines — tracing a specific output back to a specific causal factor is often technically impossible.

This is not merely a practical evidential problem. It is a structural feature of how modern AI systems work. The output of a transformer model is not the product of a traceable reasoning chain — it is the result of billions of weighted parameters interacting in ways that are not interpretable at a human level. When no one can explain why the system produced the output it did, establishing causal responsibility becomes a philosophical as much as a legal challenge.

Real-World Cases: Who Was Actually Held Responsible

The accountability gap is not theoretical. A growing body of decisions, settlements, and regulatory findings shows how institutions are currently navigating it — and the pattern is instructive.

The EU AI Act: Liability Provisions vs Reality

The EU AI Act is the most significant attempt to date to systematically address AI liability through legislation. It does not create a comprehensive AI liability regime on its own — that work is being done through companion legislation, the EU AI Liability Directive — but its requirements have direct liability implications that are already reshaping how organisations think about accountability.

What the AI Act Actually Does on Liability

The AI Act creates liability exposure primarily through its compliance obligations, not through direct liability rules. When a deployer of a high-risk AI system fails to comply with the Act's requirements — inadequate human oversight, missing technical documentation, failure to register the system — and harm results, that non-compliance becomes powerful evidence of negligence in civil proceedings.

Article 9 requires high-risk AI systems to have risk management systems in place throughout their lifecycle. Article 13 requires transparency and provision of instructions adequate for deployers to implement proper human oversight. Article 14 — the human oversight provision — requires that high-risk AI systems be designed to allow effective oversight by humans with appropriate knowledge, and that humans be able to override, disregard, or reverse system outputs.

Each of these requirements creates a corresponding liability risk: failure to implement them shifts moral and potentially legal responsibility squarely onto the deploying organisation.

The Companion AI Liability Directive

The EU AI Liability Directive, under development alongside the AI Act, proposes two major changes to the liability landscape that address the accountability gap directly:

• ⚖️
  Rebuttable presumption of causation: if a claimant can show that a defendant violated the AI Act's requirements and that the violation is plausibly linked to the harm, causation is presumed — the burden shifts to the defendant to disprove it This directly addresses the causation problem in AI tort claims. Claimants no longer need to prove the technical chain of causation from AI architecture to harm — they need to prove the regulatory violation and the plausible link.
• 📂
  Disclosure obligations: courts can order defendants to disclose evidence and documentation about high-risk AI systems, including technical documentation, logging data, and conformity assessments This addresses the practical evidentiary problem that claimants rarely have access to the technical details needed to prove AI liability claims. The black-box problem becomes less dispositive if courts can compel disclosure.

The Accountability Gap in High-Risk Categories

The AI Act's Annex III identifies specific high-risk use cases where the liability implications are most acute. These categories are high-risk precisely because autonomous AI decisions in these domains carry the greatest potential for irreversible harm:

What Governance Frameworks Mean for Liability

The practical governance question for any organisation deploying autonomous AI is how the choice of oversight model affects both the probability of harm and the allocation of liability when harm occurs. Three oversight models are in common use, and they carry meaningfully different liability implications.

Human-in-the-Loop (HITL)

In a human-in-the-loop model, a human reviews and approves each consequential AI decision before it takes effect. The AI system generates recommendations; the human decides. This is the highest-oversight model and, correspondingly, allocates the greatest degree of individual human responsibility to the review process.

HITL provides the strongest protection against the accountability gap — because a human made each consequential decision, conventional negligence doctrine applies clearly. If the human approved a harmful decision, they (and their employer) are responsible, and the AI developer's liability is limited to cases where the AI system itself was technically defective.

The cost is speed and scale. HITL is not viable for high-volume automated decisions — credit scoring, fraud detection, content moderation — because human review rates cannot match AI processing rates. Organisations choosing HITL for specific high-stakes decisions are making an explicit accountability trade-off: accepting operational limitations in exchange for conventional liability allocation.

Human-on-the-Loop (HOTL)

In a human-on-the-loop model, the AI system acts autonomously but a human monitors the system's behaviour and can intervene if needed. Decisions are made by the AI; humans watch for problems and override when they identify them. This is the model required by the EU AI Act's Article 14 for high-risk AI systems that cannot practically operate with full HITL oversight.

HOTL creates a more complex liability picture. The human monitor has a duty to identify problems and intervene. If they fail to do so — if the monitoring is inadequate, the alert thresholds are too high, or the human is monitoring too many systems to catch anomalies — the liability for those failures lies with the deploying organisation. The question is not whether a human approved each decision but whether the monitoring system was reasonably designed and whether the responsible human met their monitoring obligations.

Emergency Controls and Kill Switches

Emergency controls — the ability to halt an autonomous AI system immediately when problems are detected — are a governance minimum for any organisation deploying autonomous AI in consequential domains. They do not prevent the accountability gap from arising, but they limit the harm that can accumulate before the gap is closed.

From a liability standpoint, the absence of effective emergency controls is increasingly treated as evidence of organisational negligence. An organisation that deploys an autonomous system capable of causing cascading harm without a tested and functional kill switch has failed to take a basic precaution. When harm occurs, that failure makes the liability case significantly simpler for plaintiffs and regulators.

Conversely, organisations that can demonstrate they had effective emergency controls, that those controls were tested, and that they responded promptly when problems were detected are in a substantially better liability position — even when harm occurs — because they can show they took their oversight obligations seriously.

The Accountability Gap in Multi-Agent Systems

Single autonomous AI systems already strain liability frameworks. Multi-agent architectures — networks of AI systems that interact with, delegate to, and build on each other's outputs — create a compounded accountability gap that current frameworks have barely begun to address.

Consider a common enterprise AI architecture: an orchestrating agent receives a business goal, breaks it into subtasks, delegates each to specialist agents, and assembles the results into a recommendation that triggers automated actions. If that chain of autonomous decisions leads to harm — a discriminatory outcome, a security breach, an erroneous financial transaction — the causal chain runs through multiple AI systems, potentially from multiple vendors, operating under logic that no single human designed or approved.

The Singapore IMDA agentic governance framework is the only governance document that addresses multi-agent accountability directly — identifying inter-agent trust, prompt injection vulnerabilities, and agent verification as specific governance domains. The EU AI Act covers multi-agent systems by implication rather than by design. Neither framework fully resolves the accountability gap for complex multi-agent harms, which may require injured parties to pursue claims against multiple entities simultaneously.

Who Actually Pays: A Practical Framework

Based on the current state of law, regulation, and enforcement practice, the following allocation of accountability is the most accurate description of where AI liability actually lands:

What Organisations Should Do Now

The accountability gap will not be closed by legislation alone — it requires proactive governance decisions by organisations deploying autonomous AI. The pattern from existing cases and regulatory guidance is consistent: organisations that demonstrate deliberate governance choices, documented oversight processes, and functional emergency controls are in a significantly better position than those that deploy AI systems without these structures.

• ✅
  Classify your AI systems against EU AI Act Annex III before deployment Knowing whether your system is high-risk determines your compliance obligations — and, increasingly, your liability exposure in civil claims. Self-assessment is currently permitted for most categories; get it documented.
• ✅
  Implement human oversight that is technically functional, not just policy-documented Article 14 compliance requires that humans can actually monitor, understand, and override the system — not merely that your policies say they should. The distinction between "policy says humans oversee this" and "humans can technically intervene" is where liability cases are decided.
• ✅
  Build and test emergency controls before deployment A kill switch that has never been tested is a documentation exercise, not a governance control. Demonstrated emergency stop capability — tested under realistic conditions, with trained operators — provides meaningful liability protection.
• ✅
  Maintain audit logs that support accountability investigations When harm occurs, the question of who is responsible will turn on what the system did and whether the relevant humans met their oversight obligations. Logs are not just compliance infrastructure — they are your primary evidence in any liability dispute.
• ✅
  Clarify contractual allocation of responsibility across your AI supply chain If you deploy AI systems built on third-party foundation models, your contract with the model provider should specify what representations are made about system behaviour, what indemnities apply, and how responsibility is allocated if those representations prove wrong. Most current AI contracts are inadequate on this point.

The Accountability Gap Is a Governance Problem

The accountability gap in autonomous AI is, ultimately, a governance problem — not a purely legal one. Legislation can shift the burden of proof, create disclosure rights, and define the compliance standard below which negligence is presumed. It cannot do what governance does: create the oversight structures, the monitoring processes, and the organisational accountability that make individual liability decisions make sense.