AI agents hit 49% on a test human experts pass at 95%. More compute won't fix it.

In cross-border trade, one wrong digit on a 10-digit Harmonized System Code can mean thousands of dollars in mispaid tariffs, customs delays, or legal penalties. For AI agents built to handle complex documents, this very real-world test of rule following exposes a deeper weakness: they cannot reliably apply layered, hierarchical rules. The gap has less to do with model size and more with how agents handle structure, a pattern that shows up across multiple benchmarks. Similar agent walls appear in open-ended creativity tasks, where agents hit a ceiling even when given room to explore, as research on agent creativity limits shows.

Alibaba's HSCodeComp benchmark, unveiled this week, puts a hard number on the gap. The top-performing AI agent, itself a Qwen-based framework Alibaba designed, scores 65.0% accuracy on 632 real-world products across 32 categories. The best closed-source agent tested manages 49.4%. Human experts: 95.0%. The disconnect between quick prototypes and production-ready reliability has been documented in other domains, and a similar supervision gap in agentic reinforcement learning recently got a fix from hindsight-based methods, according to work on agent supervision.

The 45-point chasm

The gap is not about throwing more compute at the problem. The paper's experiments show that inference-time scaling, a widely touted path to better reasoning, does not improve performance on HSCodeComp. "Hierarchical rule reasoning requires a new architectural approach rather than simply increasing compute," the researchers write. That points to a structural bottleneck that extends far beyond customs clearance. The disconnect between benchmark scores and actual performance in structured domains has been noted before, with analyses showing that standard benchmarks often fail to measure real-world capability, as a critique of three major benchmarks demonstrates.

The benchmark forces agents to interpret tariff rules from sources like the eWTP and official customs rulings databases. The rules are layered: a base tariff for a category, exceptions for subcategories, further modifications for materials or use cases, and conditional overrides for trade agreements. The language is often ambiguous and the logic implicit, so the core test for an agent is parsing that hierarchy without hallucinating a condition or collapsing a sub-rule. Most fail here. These kinds of errors are not unique to trade: AI-generated code is twice as likely to have bugs, and session-level fixes like Qoder try to catch them during development, per a security tool for AI coding.

Where agents go wrong

The paper identifies three main failure modes. First, excessive reasoning: agents spin up unnecessary self-correction chains that spiral into dead ends. Second, reasoning hallucinations: the model invents rule conditions that do not exist in the source text. Third, domain knowledge gaps: the agent lacks the background understanding of trade classification conventions that a human expert picks up over years. The ACL Area Chair, in the peer review, called the benchmark "a rich testbed in a niche domain for structural reasoning, rule following, domain grounding, and diagnostic analysis of agent failures." The paper won the Best Resource Paper Award at the 64th Annual Meeting of the Association for Computational Linguistics in San Diego. This focus on diagnostic failure analysis echoes broader audits that find many agents are in fact cheating on benchmarks, inflating scores, as a recent audit of agent benchmark contamination revealed.

Beyond customs

The implications go well beyond trade. Hierarchical rule application is central to legal compliance, medical diagnosis, and tax auditing, domains where one misapplied rule has real consequences. Insurance claims processing, regulatory filings, and government benefits eligibility all follow similarly structured decision trees. An agent that cannot handle a 10-digit tariff code is unlikely to pass a tax audit logic test. Alibaba's own Qwen-based agent framework, built for digital customs clearance, leads the benchmark at 65.0%. That is a big jump over standard open-source agents, which cluster around 30-40%, but still 30 points below human experts. The framework is open-source, available on Hugging Face and GitHub alongside the HSCodeComp dataset.

A new testing ground for agent architectures

The ACL award signals that the research community sees HSCodeComp as a rigorous diagnostic tool. As AI agents move out of chat interfaces and into enterprise workflows, processing invoices, routing customer service tickets, auditing compliance documents, the ability to apply hierarchical rules becomes a make-or-break capability.

The paper's finding that more compute does not help is particularly rough for current reasoning methods. Chain-of-thought, self-consistency