The most serious risks posed by AI systems do not arise when they deceive, persuade, or manipulate. They arise when decision authority over violence is partially delegated to systems that cannot bear responsibility, and when that delegation is diffused across institutions such that no single actor can be held accountable.

This is not a future problem. It is already underway.

Delegation, not autonomy, is the core issue

Public discussion often fixates on the word autonomous, as if the danger lies in machines acting entirely on their own. In practice, the more significant shift is delegated judgment: systems that narrow options, rank targets, recommend actions, or compress uncertainty for human operators under time pressure.

The human remains “in the loop,” but the loop is structurally distorted.

When systems pre-filter perception and prioritize outcomes, human judgment becomes reactive rather than deliberative. Responsibility remains nominally human, while meaningful choice is increasingly upstream.

Concrete deployments already in use

This pattern is visible in existing military and security systems.

The Israeli Defense Forces’ reported use of AI-assisted targeting systems during operations in Gaza, including tools described as supporting target generation and prioritization, illustrates the risk. Even when human authorization is formally required, the scale and speed of recommendations can overwhelm deliberation.

Similarly, the U.S. Department of Defense’s Project Maven, which applies machine learning to analyze drone imagery, was explicitly framed as decision support rather than autonomous action. Yet internal critiques noted that automation bias and workload pressures could lead operators to defer excessively to system outputs.

In both cases, the system does not “decide” to strike. It structures the decision space in ways that materially affect outcomes.

Responsibility diffusion as an institutional failure mode

When harm occurs, responsibility becomes difficult to assign.

Was the fault in:

  • the model’s training data,
  • the system’s thresholds,
  • the operator’s judgment,
  • the commander’s rules of engagement,
  • or the institution’s procurement decisions?

Each layer can plausibly deflect responsibility to another. The result is not deniability in the narrow sense, but responsibility diffusion: no actor is clearly accountable for outcomes that were jointly produced.

This diffusion is not accidental. It is a predictable consequence of complex socio-technical systems deployed under operational pressure.

Why speed and scale make correction unlikely

Unlike earlier dark patterns, delegated violence operates under conditions where reversibility is minimal.

Errors in recommendation systems for content or commerce can be corrected after the fact. Errors in targeting systems cannot. The cost of a false positive is immediate and irreversible.

Moreover, the incentives favor scale. Systems that accelerate operations are valued precisely because they compress time and expand capacity. That same compression reduces opportunities for reflection, dissent, or refusal.

Once embedded, these systems are difficult to slow without accepting strategic disadvantage.

Existing international humanitarian law assumes identifiable decision-makers and discrete acts. Delegated systems blur both.

Rules about proportionality, distinction, and military necessity presume a human agent capable of contextual judgment. When that judgment is partially offloaded to systems optimized for pattern recognition and throughput, legal frameworks strain to apply.

Attempts to ban “fully autonomous weapons” miss the point. Most harm arises before full autonomy, in hybrid systems that formally preserve human control while functionally reshaping it.

A concrete institutional signal

The United Nations’ ongoing debates within the Convention on Certain Conventional Weapons have repeatedly stalled on definitions. States disagree on what constitutes autonomy, meaningful human control, or acceptable delegation.

This definitional paralysis is itself a signal. It reflects not ignorance, but institutional discomfort with confronting delegation directly, because doing so would require rethinking command responsibility, procurement incentives, and operational doctrine.

Why this is a dark pattern, not a technical flaw

From an ACP perspective, delegated violence is a dark pattern because it:

  • preserves the appearance of human control,
  • diffuses accountability across layers,
  • and embeds irreversible risk in routine operations.

No single actor needs to intend harm. The system produces it through aligned incentives and structural opacity.

What containment would require (and why it is rare)

Effective containment would require:

  • explicit limits on delegation depth,
  • auditable records of how recommendations shape decisions,
  • and institutional willingness to accept slower operations in exchange for accountability.

These are costly commitments. They conflict with strategic incentives and bureaucratic norms that reward speed and deniability.

As a result, most institutions proceed incrementally, normalizing delegation while assuring themselves that humans remain in charge.

The point of no return

The danger is not a sudden leap to machine-controlled warfare. It is a gradual erosion of responsibility, where outcomes are jointly produced but individually disowned.

Once violence is mediated through systems designed to optimize rather than judge, reclaiming accountability becomes extraordinarily difficult.

At that point, the question is no longer whether AI should be used in lethal contexts, but whether institutions can still claim meaningful responsibility for what they do.