The structured-outputs-integration-gate Skill — The Test That Schema Units Miss

Adding schema-bound response parsing to an LLM pipeline is straightforward in isolation. The new validator compiles, the schema unit tests pass, and the implementation looks clean. The failure mode appears later, at the boundary between the new validator and the pre-existing post-processing code that was quietly cleaning responses before the validator existed.

The structured-outputs-integration-gate skill exists to catch this specific class of regression before it reaches a code review. Its trigger is narrow: the task migrates from prompt-engineered JSON extraction to an API's structured-output or typed-parse endpoint, or it adds a post-parse validation layer alongside existing response-cleaning helpers.

Why schema unit tests are not enough

A schema unit test exercises the validator in isolation. It mocks the provider call, sends a valid payload, an invalid-shape payload, and some edge-of-range fields. This correctly locks the schema contract. It does not, however, see what happens when the legacy response cleaner and the new validator run on the same payload in sequence.

The problem is sequencing. The new validator typically runs against the parsed model object before the old filters get a chance to run. If the legacy contract was «silently remove these tokens from the response», the new validator flips that to «hard-fail the whole response the moment one of those tokens appears». The schema tests were green because they never introduced that exact payload. Production behaviour was broken because legacy response payloads contain exactly those tokens.

The wrapper-path test

The gate requires a second test alongside the schema unit tests. The wrapper-path test exercises the full handler from its entry point through every existing post-processing filter and the new validator. The critical step is injecting a payload that the legacy filters were designed to strip, then asserting that the historical behaviour is preserved — silent removal and success — not the schema-unit behaviour, which would be a hard rejection.

This test is the gate. The skill states plainly that neither kind of test alone is sufficient. A schema unit test without the wrapper-path test leaves the contract boundary unobservable. A wrapper-path test that does not inject the legacy strip payload misses the failure mode entirely.

What to do when the gate fires

If the plan for the touched handler does not list both kinds of test, the skill requires returning to plan generation and adding them before any implementation begins. If implementation is already underway, the skill requires holding the merge until the wrapper-path test is both green and has deliberately attempted the legacy strip scenario.

The anti-patterns that the gate refuses are specific: schema unit tests alone even when exhaustive, a wrapper-path test that does not inject the legacy filter payload, and a validator that fires before the legacy filter runs. The third case has a structural fix: split the validator into a schema-only check and a post-filter safety-net helper, so the legacy filter runs first and the no-residual check runs after.

Where the rule comes from

The skill documents its own origin. A silent contract regression of this exact shape shipped to a feature branch, passed schema-unit review, and was caught only at human merge request review by a maintainer who held the legacy filter contract in memory. The schema tests were green; the production behaviour was broken. One wrapper-path test would have closed the gap before review reached a human.

Read what Datarim is for background on how skills embed lessons into the workflow, or see the self-verification skill for how these kinds of gaps are caught at the verification layer.