LFM2.5-350M gained 14 points on JSON. YAML barely budged

The 7-point jump, and the 70% that still fails

The baseline run served LiquidAI's LFM2.5-350M through llama.cpp in BF16 and put it against all 2,000 IFStruct test items. It passed 452 of them, or 22.6%, at an average latency of 1,453ms. The IFStruct release blog reports 21.1% for the same model, close enough that the notebook reuses the local figure as its fixed baseline: every number after this point comes from an identical serving stack, so the comparison measures training rather than infrastructure.

Then came 100 GRPO steps on roughly 500 training samples. The merged checkpoint passed 594 of 2,000, or 29.7%, at 1,518ms. That is a 7.1-point gain, and the tuned model still returns something the evaluator rejects about seven times out of ten.

The error tally shows where those failures sit. Both runs are dominated by one complaint: "required field missing," logged 7,228 times on the base model and 7,331 times after training. Wrong item counts rose from 738 to 890, type mismatches from 540 to 555. Two error types do vanish. The base run logged 317 unclosed code blocks and 170 missing code blocks; neither appears in the tuned tally, which is the result the prompt design was built to produce.

Why schema compliance gets its own scoreboard

Structured output is one of the most common things people ask a language model to do, and one of the least measured on its own. Most suites fold validity into a broader reasoning or extraction score, so a model that produces the right answer in an unparseable shape can score the same as one that never found the answer. IFStruct measures the shape. It tests whether output is valid, parseable, and matched to the requested schema, which is the question that decides whether a model can be wired into a downstream system at all. It is one more entry in the broader turn away from static, one-size-fits-all benchmarks.

The harness and dataset are open: the code sits in the Liquid4All/ifstruct repository, and version 1.0 of the dataset is published as LiquidAI/ifstruct-v1.0. The source is explicit that this notebook is not the pipeline behind the IFStruct blog's RL model, and that it does not try to reproduce the published score. It is a demonstration that narrow, task-specific fine-tuning can push a small model toward the performance of a much larger one.

Six million trainable parameters on a free-tier GPU

All of it fits on a free-tier 16 GB Colab or Kaggle GPU. Training data comes from nvidia/Nemotron-RL-instruction_following-structured_outputs, a set that pairs each prompt with a target JSON Schema and an expected field count; about 500 samples were used. The adapter is LoRA at rank 16 with alpha 32, and because LFM2.5 runs a hybrid attention and convolution architecture, it targets module names specific to that design: q_proj, k_proj, v_proj, out_proj, in_proj, w1, w2, w3. That trains roughly 6 million parameters, about 1.66% of the model, a footprint that lines up with the small-model boom running alongside the frontier's billions.

The Nemotron distribution does not match IFStruct's, so the prompts were altered first. Forty percent got a fenced-code-block instruction appended. A separate 20%, not overlapping with that group, were rewritten as top-level array tasks with a required item count. The second change is where the bare-list results later come from.

Three reward functions score every completion on a 0 to 1 scale. json_format_reward pays full credit for the requested form, 0.2 for a parseable answer in the wrong form, and nothing for output that will not parse. field_count_reward pays 1.0 for an exact top-level field count and decays linearly as the count drifts. schema_validation_reward counts constraint violations and gates partial credit on how many required keys are present. They combine as a weighted sum of 1.0, 0.5, and 2.0, which puts twice the weight on schema validity as on format alone. Training runs at temperature 1.1 with a KL penalty of 0.01 against the reference model.

The 16 GB budget shapes other choices too. Completions are capped at 1,024 tokens, which is not generous for nested JSON, and the author reports that the truncated-completion fraction stayed near zero across the run. Token budgets are a known fault line for models this size, as Boris-2's uneven 90B and 60B pretraining allocations showed.

The regressions hiding inside the average

IFStruct group	Base	GRPO-tuned	Change
Overall	22.6%	29.7%	+7.1
JSON	18.0%	31.9%	+13.9
YAML	27.2%	27.5%	+0.3
Wrapper key	28.5%	29.7%	+1.2
Bare list	16.6%	29.7%	+13.1

Split by requested format, JSON pass rates nearly doubled while YAML held still. Split by top-level structure, bare lists jumped from 16.6% to 29.7% and wrapper-key objects crept from 28.5% to 29.7%. That pattern tracks the prompt augmentation directly, since bare-list output is exactly what 20% of the training prompts were rewritten to teach.

Entity type	Base	GRPO-tuned	Change
Screenplay scene	17.4%	37.0%	+19.6
Log parser examples	29.2%	45.8%	+16.6
Support ticket batch	37.0%	49.3%	+12.3
Event ticket booking	45.8%	57.9%	+12.1
Interview transcript segment	26.2%	16.2%	-10.0
Camera review	7.2%	6.0%	-1.2

Entity-level results swing harder in both directions. Camera reviews slipped from six passes to five out of 83 items, and interview transcript segments fell ten points. The five weakest entity types are the same five before and after training, only reordered: recipe, gpu_review, camera_review, customer_email_thread and conference_schedule occupy the bottom of both runs. Recipe more than doubled, from 4.3% to 10.0%, and still ends near the floor. Whatever the reward signal taught, it did not teach the tasks that were already hardest, the same selective movement abliterated Qwen3.8-27B produced when its refusals fell to zero while everything else held still.

The comparison that matters is the one the source draws. The tuned 350M model lands below Qwen3.5-2B at 33.15%, a model several times its size. Closing most of a gap that wide with 6 million trainable parameters and 100 steps is the argument. The caveat sits right beside it. IFStruct rewards form, and form is what a narrow reward signal can buy. A test that scores the shape of an answer says nothing about whether the answer is useful, the same blind spot HumanEval left between coding and real work. The 7,331 missing required fields in the tuned run were never the target.