The intelligence misclassification

AI governance discourse continues to treat intelligence as the primary risk variable. Systems are evaluated according to capability: model size, autonomy, apparent reasoning ability, or proximity to human judgment. Regulatory thresholds are drawn around these features, and public concern tends to spike at moments of visible technical advance.

This framing persists despite a growing empirical record that points elsewhere. The most consequential failures of AI systems have not resulted from sudden increases in intelligence. They have resulted from systems that remained in place, unremarked, shaping decisions repeatedly over long periods of time. What matters most, governance-wise, is not how intelligent a system is, but whether it stays.

Persistence as unpriced power

Continuity produces a form of power that is difficult to see because it lacks an event boundary. A system that persists inside an institution does not need to outperform humans, replace them, or even be especially accurate. It needs only to remain operational long enough for its outputs to become familiar, routinized, and embedded in downstream judgment.

The COMPAS risk assessment tool illustrates this dynamic with uncomfortable clarity. Its influence over sentencing and parole decisions did not stem from exceptional sophistication. It stemmed from endurance. For years, it sat inside judicial workflows, shaping outcomes incrementally, case by case. Its limitations were known early. Its behavior was studied, debated, and criticized. What changed slowly was not the system, but the institutional environment around it, which adapted to its presence and normalized its influence.

Continuity did the work that intelligence never had to.

Authorization without renewal

Persistence also exploits a structural weakness common to modern governance regimes: authorization is treated as an event rather than as a condition that must be continuously re-earned. Systems are approved, procured, certified, or adopted at identifiable moments. What follows is assumed to be operation, not governance.

Over time, the rationale for a system’s existence erodes. Staff turnover increases. The original trade-offs fade from memory. Documentation remains, but institutional understanding decays. The system continues to operate not because it is periodically re-justified, but because no mechanism exists to require reauthorization.

In this way, continuity converts initial permission into standing authority.

This pattern is visible across welfare eligibility systems, fraud detection tools, predictive policing software, and large-scale content moderation classifiers. Early governance activity is intense; later governance becomes procedural. The system persists, lightly monitored, loosely understood, and rarely reconsidered as a matter of principle.

Drift without decision

Persistence enables a second failure mode: drift without agency. Models are retrained, thresholds adjusted, integrations modified. Each change is incremental, framed as maintenance rather than redesign. No single modification appears significant enough to trigger review. Yet cumulatively, the system that exists bears only a partial resemblance to the system that was originally authorized.

Governance regimes tend to treat this as benign evolution. In practice, it is authority exercised without decision. Outcomes change, but no one decides to change them. Responsibility becomes difficult to locate because no discrete act of authorization can be identified.

This is not a bug in oversight processes; it is a consequence of designing governance around episodic intervention rather than continuous presence.

The limits of episodic oversight

Most AI governance mechanisms are event-driven. They are activated by procurement, deployment, incidents, audits, or regulatory thresholds. Persistence dissolves these triggers. A system that never quite fails, never quite succeeds, and never quite changes enough to demand attention can exert more influence than a system that causes a visible crisis.

The Dutch childcare benefits scandal again makes this clear. The risk-scoring system involved did not suddenly become more capable. It remained. Over time, its outputs hardened into enforcement practice. Institutional reliance grew. By the time harm became undeniable, the system had already reshaped norms of suspicion and compliance.

The damage accrued under conditions of apparent normalcy.

Continuity as governance blind spot

What makes continuity difficult to govern is that it resists the categories governance systems are built around. It is not a breach, a threshold, or an incident. It does not announce itself. It appears, if at all, as background.

Regulators are structured to respond to violations. Organizations are structured to manage projects. Risk frameworks are structured to evaluate hazards at discrete points in time. None of these structures is well suited to governing systems whose primary risk arises from their continued existence.

As a result, continuity is treated as neutral, even stabilizing. Systems are assumed to become safer as they mature. In reality, they often become more authoritative simply by persisting.

Why intelligence remains the wrong proxy

This is why debates about “agentic” or highly autonomous AI often misfire. Agency is not required for governance-relevant power. A system does not need initiative or intent to shape institutional outcomes. It needs durability.

A mediocre system that persists for a decade can matter more than a brilliant system that is tested briefly and withdrawn. Intelligence attracts scrutiny. Continuity evades it.

The fixation on capability thus misdirects governance attention toward spectacular risk while leaving cumulative, structural risk largely untouched.

The unresolved constraint

If risk is primarily a function of persistence rather than intelligence, then much of current AI governance is misaligned with the problem it claims to address. Capability thresholds, transparency obligations, and episodic audits are ill-suited to systems whose influence accrues over time through repetition and normalization.

What remains unresolved is how to govern systems that do not fail loudly, do not improve dramatically, and do not trigger review — but nonetheless come to exercise durable authority simply by remaining in place.

That question cannot be answered by better classification or finer-grained disclosure. It requires treating continuity itself as a governance object, rather than as a neutral backdrop against which governance occasionally intervenes.

In the next piece, I will examine how this misalignment reproduces itself across organizations and sectors — not because they share technology, but because they share institutional structure.