AI agents hit 49% on a test human experts pass at 95%. More compute won't fix it.

In cross-border trade, one wrong digit on a 10-digit Harmonized System Code can mean thousands of dollars in mispaid tariffs, customs delays, or legal penalties. For AI agents built to handle complex documents, this very real-world test of rule following exposes a deeper weakness: they cannot reliably apply layered, hierarchical rules. The gap has less to do with model size and more with how agents handle structure, a pattern that shows up across multiple benchmarks, including tests of instruction-following and verification, as the verification horizon analysis and IFBench's instruction-following tests have shown.

Alibaba's HSCodeComp benchmark, unveiled this week, puts a hard number on the gap. The top-performing AI agent, itself a Qwen-based framework Alibaba designed, scores 65.0% accuracy on 632 real-world products across 32 categories. The best closed-source agent tested manages 49.4%. Human experts: 95.0%. The disconnect between prototype speed and production reliability is a known theme in AI deployment, as vibe coding critiques have documented.

The 45-point chasm

The gap is not about throwing more compute at the problem. The paper's experiments show that inference-time scaling, a widely touted path to better reasoning, does not improve performance on HSCodeComp. "Hierarchical rule reasoning requires a new architectural approach rather than simply increasing compute," the researchers write. That points to a structural bottleneck that extends far beyond customs clearance, echoing findings from SkillCoach's process-level evaluation.

The benchmark forces agents to interpret tariff rules from sources like the eWTP and official customs rulings databases. The rules are layered: a base tariff for a category, exceptions for subcategories, further modifications for materials or use cases, and conditional overrides for trade agreements. The language is often ambiguous. The logic is implicit. For an agent, parsing that hierarchy without hallucinating a condition or collapsing a sub-rule is the core test. And it fails. Similar failures have been observed in enterprise Java migration tasks, as an analysis of AI agent performance on enterprise Java migrations shows.

Where agents go wrong

The paper identifies three main failure modes. First, excessive reasoning: agents spin up unnecessary self-correction chains that spiral into dead ends. Second, reasoning hallucinations: the model invents rule conditions that do not exist in the source text. Third, domain knowledge gaps: the agent lacks the background understanding of trade classification conventions that a human expert picks up over years. The ACL Area Chair, in the peer review, called the benchmark "a rich testbed in a niche domain for structural reasoning, rule following, domain grounding, and diagnostic analysis of agent failures." The paper won the Best Resource Paper Award at the 64th Annual Meeting of the Association for Computational Linguistics in San Diego.

Beyond customs

The implications go well beyond trade. Hierarchical rule application is central to legal compliance, medical diagnosis, and tax auditing, domains where one misapplied rule has real consequences. Insurance claims processing, regulatory filings, and government benefits eligibility all follow similarly structured decision trees. An agent that cannot handle a 10-digit tariff code is unlikely to pass a tax audit logic test. Alibaba's own Qwen-based agent framework, built for digital customs clearance, leads the benchmark at 65.0%. That is a big jump over standard open-source agents, which cluster around 30-40%, but still 30 points below human experts. The framework is open-source, available on Hugging Face and GitHub alongside the HSCodeComp dataset. The challenge of bridging such gaps in agent training has been explored in Alibaba's open-source world model for agent training.

A new testing ground for agent architectures

The ACL award signals that the research community sees HSCodeComp as a rigorous diagnostic tool. As AI agents move out of chat interfaces and into enterprise workflows, processing invoices, routing customer service tickets, auditing compliance documents, the ability to apply hierarchical rules becomes a make-or-break capability.

The paper's finding that more compute does not help is particularly rough for current reasoning methods. Chain-of-thought, self-consistency, and similar techniques that rely on scaling inference-time effort assume that thinking longer yields better results. HSCodeComp shows that for hierarchical rule application, longer thinking often produces more hallucinated exceptions and longer, more fragile argument chains. This reinforces the case for open evaluation workbenches that can test such hypotheses systematically, as Olmo Eval's open evaluation workbench demonstrates.

For now, human experts keep their 95% crown. But the benchmark draws a clear target: any agent framework that can bridge that 30-point gap on a test of structural reasoning will have demonstrated a genuinely new capability, one with direct commercial value in every domain governed by layered regulations.