What's Missing From Every AI Safety Conversation (And Why Nobody Wants to Bring It Up)

The AI safety debate has a peculiar structure. On one side: a group of researchers and advocates who argue that advanced AI poses existential risks to humanity, that aligning AI systems with human values is a hard unsolved technical problem, and that the most important work anyone can do is on preventing a future superintelligent system from pursuing goals that catastrophically diverge from human interests. Eliezer Yudkowsky, the Machine Intelligence Research Institute, a significant portion of Anthropic’s founding team — this is the cluster.

On the other side: a group of researchers, industry figures, and commentators who argue that current AI systems are just statistical tools, that the existential risk framing is science fiction that distracts from concrete harms happening right now, and that the “safety” establishment is a self-serving apparatus that primarily functions to protect incumbent AI companies from competition and regulation. Yann LeCun, a significant portion of the academic ML community, most of the venture capital industry — this is the other cluster.

Both groups are having an interesting argument. Both groups are, in different ways, avoiding the most important question in the room.

The question is: who decides what AI systems are optimized to do, under what accountability structure, using what governance mechanism, and with what enforcement capability? This is not a technical question. It’s a political economy question. And the answer to it will determine more about AI’s actual effects on the world than any amount of technical alignment research or existential risk mitigation work.

Let me explain what I mean with specifics, because abstract governance talk is easy to dismiss.

An AI system deployed in credit scoring makes decisions about who gets loans. It is optimized for some combination of default probability prediction, regulatory compliance, and business profitability. These optimization targets are chosen by a small group of data scientists and product managers at the deploying company, subject to some legal constraints, but not subject to any democratic input from the people whose loan applications are being decided. The people denied loans don’t know the criteria. They often don’t know that an algorithmic system made the decision. There is no meaningful mechanism for them to contest the decision or understand why it was made.

An AI system deployed in social media content ranking decides what information billions of people see. The optimization target — some combination of engagement, time-on-platform, advertiser preferences, and legal constraints — is set by a small group of product managers and ML engineers at one or two companies. These people are not elected. They’re not regulated beyond basic platform obligations. Their decisions about what to optimize for shape the information environment of political democracies, yet there is no mechanism by which the populations affected can meaningfully influence those decisions.

An AI system deployed in predictive policing scores neighborhoods or individuals for crime risk. It is optimized for some combination of historical crime data, demographic correlates, and precinct-level performance targets. The people who live in high-scored neighborhoods didn’t set those scoring criteria. They often don’t know they exist.

These are not hypothetical scenarios. They’re existing systems, operating now, affecting tens or hundreds of millions of people. The decisions made by the small groups who set the optimization targets have enormous distributional consequences — who gets resources, who gets credit, what information populations receive, who gets surveilled more heavily — yet these decisions are made without meaningful public accountability.

The existential risk community’s response to this concern is essentially: yes, those are real problems, but they’re small problems compared to what a misaligned superintelligence could do, so we should focus resources on the big risk.

This argument fails in multiple ways.

First, it treats the near-term political economy problem as someone else’s responsibility. But the political economy of the next decade — who controls AI systems, under what accountability, with what governance structures — will determine what kind of regulatory and oversight environment exists when (if) more powerful systems arrive. Ceding that ground to “we’ll sort out governance after we solve alignment” is giving away the most important near-term leverage. Alignment solutions that exist in a world where AI deployment is completely ungoverned will be ignored. Alignment solutions that exist in a world with robust governance infrastructure will have somewhere to land.

Second, the alignment research program is not demonstrably making progress toward a problem that is clearly defined. The alignment problem, as typically formulated, involves preventing a hypothetical future system from pursuing goals that diverge from human values in a world where we can’t fully specify human values and where the system is more capable than us at pursuing its goals. This is a genuinely hard problem. It’s not clear that the current research programs — RLHF, constitutional AI, scalable oversight — are addressing it in ways that will remain relevant when the systems they’re studying become much more capable. The researchers doing this work are smart and well-intentioned. The connection between the current work and the stated goal is not tight.

Third, focusing existential risk concern on a hypothetical future superintelligence is emotionally and intellectually convenient in a way that should produce skepticism. The concrete, medium-term, tractable governance question is uncomfortable to engage with because it requires navigating real political conflicts about power and accountability. The superintelligence framing substitutes a technically interesting but politically contentless question — how do we align a hypothetical system with human values? — for the politically difficult question — who decides what AI optimizes for right now, and who can hold them accountable?

The “AI is just a tool” camp is no better, though it makes different errors.

The dismissal of safety concerns sometimes comes with an implicit claim that current AI harms are what matters, and that the safety establishment is distracting from them. This is partly right — current harms are real. But the “AI is just a tool” framing imports an assumption that’s empirically questionable: that AI systems are neutral tools that simply execute whoever gave them instructions, and that any harm is attributable to those instructions.

This isn’t accurate. An AI system trained on a dataset with certain statistical properties will have systematic biases built into its outputs. Those biases emerge from design decisions made by a small group of engineers — decisions about what data to train on, what objectives to optimize, how to handle edge cases, what outputs to penalize. These decisions are normative choices with distributional consequences. Calling the system a “tool” obscures the fact that those choices were made, that they could have been made differently, and that they produce systematic effects that can’t be attributed solely to whoever gave the immediate instruction.

The “just a tool” framing also gets deployed in a way that’s suspiciously convenient. When it’s useful to argue that AI is just executing instructions (so the company isn’t responsible for outputs), companies call it a tool. When it’s useful to argue that AI is making intelligent decisions (so the product is worth the enterprise contract price), companies describe it as a cognitive system capable of autonomous reasoning. You can’t have both framings available on demand.

What would taking the political economy question seriously actually look like in practice?

It would mean treating “what does this AI system optimize for, and who decided that?” as a primary question in evaluating any AI deployment — not secondary to “is the technology impressive?” or “is the technology dangerous in a science fiction sense?” Primary. This requires building audit and evaluation infrastructure that doesn’t currently exist in most jurisdictions.

It would mean building institutional capacity to answer that question for specific systems in specific deployments. This requires auditability requirements — AI systems in high-stakes domains need to be inspectable in ways that allow third-party verification of optimization targets and assessment of distributional effects. Most current deployments are not auditable by anyone outside the deploying organization. The technical capability for meaningful auditing exists; the legal requirement to submit to it doesn’t.

It would mean treating AI governance as a continuous political process rather than a one-time regulatory decision. The FDA doesn’t approve drugs and then stop paying attention. It monitors ongoing safety, requires post-market surveillance studies, and has authority to revise approvals when evidence changes. An equivalent structure for AI systems in consequential domains would require sustained political will and institutional investment that nobody has yet committed to seriously.

It would mean having an honest public conversation about which entities currently have the power to set AI optimization targets and what accountability they face. The current answer: a handful of technology companies and a handful of large institutional users, accountable primarily to shareholders and subject to legal liability that’s diffuse and hard to prove. This is a political economy structure that systematically produces outcomes that favor the companies deploying the systems over the people affected by them.

Why doesn’t the AI safety establishment lead on this? The incentive structures are not hard to identify.

The technical alignment research community exists in a complex symbiotic relationship with the major AI labs. Anthropic was founded by safety-focused researchers and has a large alignment team. OpenAI has a safety function. Google DeepMind has an ethics group. These teams are funded by the same organizations whose deployment decisions create the political economy problem. No one’s funding gets maximized by loudly arguing that the core governance question is “who in your organization sets the optimization targets for your models, under what accountability, and why should anyone outside your organization trust that process?”

The research is more fundable when it’s framed as technical — alignment as a machine learning problem, safety as a capability problem, governance as a downstream question that other people will handle once the technical work is done. This framing conveniently positions the researchers as the critical part of the problem, and it’s a framing that labs find comfortable because it doesn’t imply that their current deployment practices require external accountability.

The “AI is just a tool” camp has its own incentive structure. If AI systems are just tools, companies face less regulatory scrutiny, less disclosure requirements, less liability exposure. The framing is commercially convenient.

The political economy question doesn’t have a natural institutional home or a natural funding constituency. Civil society organizations that could push it are underfunded relative to the labs. Governments that could regulate effectively on it are outpaced by the technology and are often short on the technical expertise needed to regulate intelligently rather than performatively. Academic researchers who study the governance question exist — there’s good work being done in this space — but they don’t have the funding, the political access, or the narrative dominance of the alignment research community.

So the conversation continues to be dominated by two camps that are, in different ways, avoiding the question that matters most. One camp is focused on hypothetical future systems. The other is focused on denying there’s a serious problem. Neither of them is primarily asking: who decided this, and who can make them answer for it?

That question is not cinematic. It doesn’t produce good keynote talks. It doesn’t attract the kind of funding that “prevent superintelligence from destroying humanity” attracts. But it’s the question that will determine whether AI development over the next decade produces something that democratic societies can meaningfully control — or something they’re handed as a fait accompli.