When 'High Risk' Means the Wrong Thing

Somewhere in the drafting of the EU AI Act, someone made a sensible-sounding decision that is now producing genuinely strange outcomes. They decided that the riskiness of an AI system should be assessed primarily by its application domain — not its technical architecture, not its error rate, not its actual deployment context. An AI system used in hiring is high-risk. An AI system used to recommend movies is minimal-risk. The assessment hinges not on what the system does or how well it does it, but on what sector it operates in.

This is, in a narrow sense, reasonable. The consequences of a bad hire recommendation are more serious than the consequences of a bad movie recommendation. Nobody is arguing otherwise.

But the application-domain approach has a structural problem that is now producing outcomes nobody intended. It treats identically a mediocre AI hiring system operated by a careless small company and a state-of-the-art hiring system operated by a responsible large company. Both are high-risk. Both face the same conformity assessment requirements. The quality of the system is irrelevant to its classification.

The result: companies that were already being careful are drowning in compliance paperwork. Companies that were being careless are not necessarily improving — some are finding clever ways to redraw the boundaries of what their system is “for.”

The Domain Trap

The EU AI Act’s Annex III lists the high-risk application domains with some specificity. Management and operation of critical infrastructure. Education — specifically systems that determine access or assess performance. Employment — systems used in recruitment, selection, promotion, task allocation. Essential private services, including credit. Law enforcement. Migration. Administration of justice.

These domains share a characteristic: they’re areas where AI errors can compound existing inequalities or constrain individual freedom in ways that are hard to reverse. Being incorrectly denied a loan, incorrectly screened out of a job application, incorrectly flagged by a border control system — these are harms that can follow a person for years.

The logic is sound. The implementation creates something paradoxical.

Consider an HR tech company in the Netherlands — let’s call it what it actually is, because several exist — that sells a resume screening tool to medium-sized employers. This company’s tool is high-risk under Annex III. They face conformity assessment, mandatory human oversight documentation, data governance requirements, transparency obligations to applicants, and registration in the EU AI database. The cost and complexity of compliance may well drive them out of the market.

Who replaces them? The larger HR platform companies — SAP SuccessFactors, Workday, Oracle HCM — which were already selling AI-augmented hiring tools before the Act came into force. They have compliance departments. They have legal teams. They can absorb the conformity assessment cost without breaking stride. The regulation, designed to protect job applicants from harmful AI, may in practice consolidate the hiring AI market among exactly the vendors who have the most to gain from scale and the least incentive to improve.

This isn’t a conspiracy. It’s a structural feature of compliance economics. Large incumbents often quietly support regulations that raise barriers to entry, because barriers to entry protect incumbents. The EU AI Act isn’t unique in this regard — every major regulatory framework eventually gets captured by some version of this dynamic. But the speed at which it’s happening here is striking. The regulation isn’t even fully enforced and the market consolidation is already visible.

Gaming the Taxonomy

The more immediately interesting dynamic is what’s happening at the classification boundaries.

A system used to “determine access to” educational resources is high-risk. A system that “recommends” educational resources is presumably not, since it doesn’t determine access. The line between determination and recommendation is, in practice, a legal distinction that engineers can architect around. If your system outputs a ranked list rather than a binary yes/no, is it determining or recommending? This is not an idle philosophical question. It is currently the subject of paid legal advice across Europe.

Similarly: a system used in employment “recruitment or selection” is high-risk. A system that helps HR professionals “organize their time” or “prioritize their attention” during a recruitment process may not be. If your resume screening tool outputs a “relevance score” that a human then uses to prioritize their review queue, is the human making the decision or the AI? Some companies are restructuring their products specifically to answer this question in the direction that avoids high-risk classification.

This is not hypothetical. In June 2026, a prominent Belgian employment law publication ran a detailed analysis of how three major HR tech vendors had amended their product terms and user documentation to reframe their AI systems as “decision support” rather than “decision-making” tools — a reframing with meaningful legal consequences under the Act and essentially no change to the underlying system behavior.

The EU AI Office is aware of this. They’ve published guidance on “functional equivalence” that attempts to prevent cosmetic reframing from altering classification. But guidance is not law, and the guidance itself contains enough ambiguity that legal teams can still find room. This is the eternal problem of principled regulation meeting adversarial compliance: the principle is clear; the boundaries are contestable; the contest is decided by whoever has better lawyers.

The Quality-Blindness Problem

Perhaps the most counterproductive feature of the current classification system is its complete indifference to system quality.

A high-risk AI system must meet documentation, transparency, and oversight requirements. It does not have to be accurate. There is no minimum accuracy standard for employment AI systems in the EU AI Act. There is no requirement to demonstrate that a hiring AI system actually improves hiring outcomes. The regulation is entirely procedural in its requirements — conform to the documentation and oversight framework, and you can deploy a high-risk system regardless of its actual performance.

This creates a peculiar incentive structure. Companies building genuinely good AI systems face the same compliance burden as companies building genuinely bad ones. The regulation doesn’t reward quality. It rewards compliance. These are different things.

Contrast this with the approach the FDA takes to medical devices. Device approval requires clinical evidence of both safety and effectiveness. You can’t launch a medical device by documenting your quality management system and providing transparency information. You have to demonstrate the device works. The EU AI Act’s approach to “high-risk” AI is far closer to the ISO 9001 quality management certification model — prove you have a process, not that the process produces good outcomes — than to the FDA efficacy model.

There are reasons the Act doesn’t require efficacy demonstration: measuring AI system quality is genuinely hard, and defining what “better hiring” even means is philosophically contested. But the procedural-only approach means that a company can deploy a racially biased hiring AI that reliably screens out protected groups, document their human oversight process meticulously, and be technically compliant with the EU AI Act while causing exactly the harms the Act was designed to prevent.

Who’s Actually Struggling

The enforcement data from the first seven months of 2026 tells a specific story. National Competent Authorities have opened formal inquiries into 23 organizations across the bloc, according to the EU AI Office’s July 2026 transparency report. The breakdown is revealing.

Fourteen of the 23 are companies with fewer than 100 employees. Seven are in employment or education AI verticals. Only one is a company most people outside the industry would recognize. None of them are foundation model providers.

The small companies are struggling with compliance not because they’re bad actors but because the conformity assessment process assumes a level of organizational infrastructure that most small companies don’t have. The Act requires a “quality management system” for high-risk AI — procedures for risk management, data governance, post-market monitoring, corrective actions. A large company has all of this by default because they have compliance departments. A 40-person startup has founders who code and a part-time operations person.

The inquiry into the one recognizable company — a major European bank’s automated customer assessment system — has proceeded at glacial pace through the German NCA since March 2026. No enforcement action has yet been taken. The bank’s legal team has filed responses running to hundreds of pages. This is the compliance asymmetry made legible: small companies face existential pressure from early inquiries; large companies face expensive but manageable friction.

Regulatory systems are always imperfect instruments. The EU AI Act will improve as the EU AI Office publishes more guidance, as Notified Bodies develop expertise, as courts interpret ambiguous provisions, and as the first major enforcement actions clarify what the rules actually mean in practice. The GDPR is genuinely better regulation in 2026 than it was in 2018. The same will likely be true of the AI Act in 2034.

But “it will improve” is cold comfort for the Tallinn fintech that is shutting down its lending AI platform rather than navigating a nine-month Notified Body queue, or the Dutch HR tech startup that is pivoting to the US market where the compliance overhead is a fraction of what Europe demands. The first year of enforcement is not a preview of what the regulation will eventually become. It’s the regulation’s actual introduction to the companies it governs — and introductions, as any psychologist will tell you, are hard to walk back.

The EU AI Act’s drafters wanted to protect people from harmful AI. They’ve mostly succeeded in creating an industry of compliance professionals and a generation of clever legal reframings. The harmful AI is, in most cases, still running.

The Missing Feedback Loop

There is a design problem embedded in the EU AI Act that the risk classification debate tends to obscure: the regulation has no systematic mechanism for learning whether it’s working.

The Act requires post-market monitoring by AI developers — ongoing tracking of deployed systems’ performance. It requires the EU AI Office to produce annual reports on implementation. It creates a database of high-risk AI systems where providers register their applications. These are genuine data collection mechanisms.

What they don’t capture is the counterfactual: the AI systems that weren’t built because of compliance costs, the products that were modified to avoid classification, the companies that left the EU market rather than comply. The regulation measures what’s there. It cannot measure what’s missing.

This asymmetry means the regulatory feedback loop is systematically biased. Enforcement actions and compliance assessments tell the AI Office about applications that exist and face scrutiny. The office can measure whether certified high-risk AI systems meet their documentation requirements. It cannot measure whether beneficial high-risk AI applications are being foregone because the certification pathway is too costly. The cost of under-innovation is invisible to the instrument designed to measure regulatory effectiveness.

Fixing this would require the EU AI Office to actively track not just compliance but the investment, product, and market dynamics in affected sectors — and to commission independent research that explicitly tries to quantify what isn’t being built. Some version of this will eventually be required by the Act’s five-year review clause, which requires the Commission to assess the regulation’s actual impact. But a five-year review catches consequences five years after they’ve accumulated.

The GDPR’s five-year review, published in 2023, was the first official acknowledgment that the regulation had significantly impacted European digital ecosystem competitiveness. By then, several European ad-tech and data analytics sectors had substantially restructured — not necessarily for the worse, but with effects that had been accumulating since 2018 without systematic measurement.

The EU AI Act’s five-year review is due in 2030. The risk classification consequences described above will be thoroughly apparent by then. The question is whether they’ll be apparent in time to make adjustments that matter for the companies facing them today.