Treating SOPs as code: why compilation alone lifts strong agents by 16 points

A bank's call center agent has to follow a standard operating procedure: verify the customer's identity, check account status, assess eligibility, then either process the request or refuse. Every enterprise agent faces this task. The dominant approach dumps the entire SOP into the prompt as text, which has well-known failure modes: resident prose of roughly 10,000 characters per turn competes with the dialog for attention, and verbalization strips executable structure such as alternative order and gate short-circuits. That pattern of prompt-bloat is exactly what vibe coding's hidden costs warn about: text-only scaffolding looks cheap until it breaks.

A new paper from the Hong Kong Polytechnic University, the University of Hong Kong, and Zhejiang Normal University, titled "Compile, Then Page: Executable SOP Programs and a Capability-Gated Runtime for Procedural LLM Agents", takes a different approach. The authors treat the SOP as a program rather than text. An offline, deterministic compiler translates the benchmark's machine-readable dependency constraints into a two-layer pseudo-code program: process functions for each user goal, and rule subroutines that pair each constraint with a verification recipe and an evidence-bearing return. An online program-guided (PG) runtime executes this as a two-layer virtual machine, where a symbolic stack machine handles bookkeeping (stack, cursor, variables, recovery) and pages only the active frame into context, while the LLM converses, calls tools, and judges evidence. The runtime mirrors the discipline that VitaBench 2.0 demands of persistent collaborators: evidence tracking, not just memory.

The three-arm design

The study uses a three-arm design on SOPBench, a benchmark that casts SOP compliance as tool-use tasks over domain simulators. The three arms (official text, compiled text, and compiled program plus runtime) separate representation from runtime on identical tasks across six models spanning capability tiers.

• offtext: the benchmark's official verbalized constraint text, resident every turn (the SOP-as-prompt baseline).
• flat: the full text of the compiled program, resident every turn (compiled representation without the runtime).
• pg: the same compiled program executed by the PG runtime with just-in-time instruction paging.

Results on Bank, the primary attribution domain (125 clean tasks), show that replacing the official SOP text with the compiled program text lifts DeepSeek-V4-Flash's pass rate from 70.4% to 86.4% (+16.0 points; 26:6, p < 0.001). Enabling the PG runtime on the same program adds another 6.4 points, reaching 92.8% on the clean subset, with 100% refusal correctness (86/86). The decomposition reveals that compilation and paging mainly buy a strong model correct refusal behavior: execution moves little (74.4% to 71.8% to 76.9%), while refusal climbs from 68.6% to 93.0% to 100%. That refusal gain echoes the safety-first architecture that critical-system AI frameworks emphasize: a guard that says no correctly is better than one that guesses yes.

A capability gate emerges

Across six models, the runtime effect is not noise around zero. It spans a signed range from +6.4 (DeepSeek-V4-Flash, p = 0.021) and +3.2 (n.s.) for strong models, through a same-vendor probe pair on opposite sides of zero, down to -14.4 (p = 0.011) for Qwen2.5-7B and -26.4 (GPT-4o-mini, p < 0.001). Pooling the two strong models yields 15:3 discordants (p = 0.016); the two weak models are independently significantly negative. Identical-configuration re-runs keep every model on the same side of zero.

The pattern extends to all seven domains. For the strong models V4 and Plus, pooled discordants reach 58:19 (p about 10^-5) and 75:31 (p = 2.3 x 10^-5), with no domain reversing its sign significantly. The weak models are harmed throughout, pooling to 59:237 for the 7B (p about 10^-26). The gate is real, and it is sharper than the binary ability split that sandbox benchmarks typically reveal: real-world procedural compliance demands a line that sandboxes never surface.

Decoupling compilation: content vs. format

Compilation both adds content (verification recipes, verifier-gate closure, evidence discipline) and re-formats the policy as code. A 2x2 design (official or compiled content x prose or code) separates the two. The format effect flips sign: re-syntaxing official prose into code gains the strong model +14.4 points yet costs the weak model -12.8 points. The content effect at fixed prose format is +17.6 for the strong model but not detected for the weak one (+2.4, n.s.). Once content is compiled, format ceases to matter (flat vs. c-prose: -1.6 and +0.8, both n.s.). For practitioners: the compiled representation never significantly hurts; isolated content pays above a low capability line; code syntax should be reserved for models it demonstrably helps.

Why the runtime is gated: discipline, not reconstruction

Directed state-reconstruction probes score all four probed models near ceiling (0.91 to 0.99), including both runtime losers. The gate is not inability to reconstruct state when asked. An unprompted audit, measuring mean domain calls issued after the goal action already succeeded, separates harmed from unharmed with a single sign rule: the two significantly harmed models are exactly the ones with positive signatures (+0.37, +0.57); all four unharmed carry negative values (-0.14 to -0.49). Reconstruction capability is present wherever probed; spontaneous state discipline is what the runtime gate tracks.

Attention-level measurements confirm the gap. DeepSeek-V4-Flash spontaneously grants its 2.1%-of-context frame an at-parity share (1.05 vs. flat's 0.96), and ablating the entire dialog history raises the gold action's log-probability by 1.9 nats. Qwen2.5-7B gives the frame only 0.15x parity on its own trajectories yet 0.87 re-presented cleanly. That is a capability-deployment gap at the attention level.

Deployment guidance

The results reject the idea that more scaffolding is always better. Compile when official prose underperforms; use code syntax only after a model-level check, and paging only after a discipline check. Strong models benefit from active-frame salience (most of the refusal gain is recoverable by placing the active frame first while retaining the full program), with a smaller refusal-centered gain from selective visibility. For weak models, use the strong execution harness without active-frame guidance. Paging is soft and auditable, so irreversible actions still require hard tool-layer guards. That final caution aligns with the separation of concerns that Mistral Studio's prompt governance enforces: a soft agent runtime can guide, but only a hard policy can protect against irreversible error.