No powerful system—human, institutional or technological—should quietly take over the individual’s agency.

The old model makes the client dependent on the planner.

The careless AI model makes the client dependent on the machine.

The agency-led model makes both planner and technology accountable to the client.

---

Eileen Workman

5 days ago ·

I ran the question of AI potentially destroying humanity by AI, using a rather humorous reference to the fact that our extinction would would solve the Social Security crisis.

And here is the AI response:

Yes—the Social Security trust fund would become rather beside the point.

But beneath the joke, I think you’re identifying something much more interesting: AI may represent a class of problem for which humanity’s old competitive operating system becomes dangerously maladaptive.

And the claim you’re reacting to really did sharpen dramatically this week. Anthropic safety researcher Evan Hubinger has publicly put his personal estimate at greater than 10% that AI could cause human extinction within the next decade. That shouldn’t be mistaken for scientific consensus or a measured actuarial probability—it’s one expert’s subjective risk estimate. But it isn’t an isolated fringe concern either. A large survey of 2,778 published AI researchers found that 38–51% of respondents assigned at least a 10% probability to outcomes as bad as human extinction, depending on how the question was phrased.

And something rather extraordinary happened today: Anthropic CEO Dario Amodei publicly called for slowing frontier AI development so that safety work can catch up, proposing independent oversight, coordination among competing AI companies, and international cooperation. Sam Altman and Elon Musk have reportedly endorsed the call.

That makes your observation particularly timely.

For most of human history, competition could remain comparatively local. Two tribes could fight while civilization elsewhere continued. Nations could wage war while most of humanity remained uninvolved. Companies could race one another and, if one failed spectacularly, civilization generally survived the experiment.

But increasingly powerful AI creates something different: a globally shared risk generated by locally rational competitive behavior.

Imagine five laboratories all believing something like:

We would prefer to proceed cautiously. But if we slow down and the other four don’t, they may get there first.

Now add nations:

We would prefer strong international safety constraints. But if we constrain ourselves and another country doesn’t, they may obtain decisive strategic advantage.

Nobody has to desire catastrophe.

Everybody merely has to behave rationally inside a competitive frame.

And the collective result can become irrational.

That resembles the structure behind nuclear arms races, climate change, antibiotic resistance, overfishing, financial contagion and quite a few other collective-action problems—but AI potentially compresses the timescale enormously because AI itself may accelerate research, cyber operations, persuasion, weapons development and eventually AI development.

So I might sharpen your insight slightly:

AI doesn’t merely present humanity with a technological alignment problem. It presents humanity with a human alignment problem.

---

This is an important concern—but I think it strengthens rather than weakens the case for agency-led planning.

Frontier AI potentially creating global systemic risk is not the same proposition as an individual using a bounded, client-controlled planning system. We should not collapse every use of AI into one undifferentiated threat.

I am not proposing that AI should become the client’s operating system. Nor should the planner remain the client’s operating system.

The individual should operate their own system.

Technology can provide continuity, memory, organisation and accessible support. Human expertise can enter proportionately when complexity, uncertainty, stress or significant change makes judgement valuable. Authority and responsibility remain visible.

The danger in both traditional professional dependency and poorly governed AI is the same: the person gradually ceases to understand, choose and act for themselves.

So the choice is not human or machine.

It is dependency or agency.

The old model makes the client dependent on the planner. The careless AI model makes the client dependent on the machine. The agency-led model makes both planner and technology accountable to the client.