Skip to content

test(validator): drop the R4 baseline entry that #432 already fixed - #517

Merged
smunini merged 1 commit into
mainfrom
fix/stale-r4-conformance-baseline
Aug 7, 2026
Merged

test(validator): drop the R4 baseline entry that #432 already fixed#517
smunini merged 1 commit into
mainfrom
fix/stale-r4-conformance-baseline

Conversation

@smunini

@smunini smunini commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

The failure

r4_spec_examples_match_baseline fails with a single divergence — an improvement the ratchet refuses to accept silently:

FIXED  questionnaire-cqf-example.json: validates clean now.
       Good -- remove it from the baseline so the ratchet holds.

This is not specific to any one PR. It reproduces on main, it fails the scheduled validator-conformance.yml runs (2026-08-05, 2026-08-06), and it fails the Pack smoke + spec example corpus job on every validator PR since — #461, #465, #472 among them.

Root cause

The baseline recorded exactly one entry whose only issue kind was unknown-schema:

questionnaire-cqf-example.json — "Documented gap: core extension StructureDefinitions (extension-definitions.json) are not in the vendored spec bundles, so these extension canonicals cannot be resolved."

#432 (feat/363-extension-catalogue, 55e3f6d, merged 2026-08-04) vendored resources/R4/extension-definitions.json and regenerated fhir_schemas_r4.json.gz (180KB → 208KB). The file's three canonicals — cqf-expression, cqf-library, cqf-questionnaire — resolve now, so it validates clean. The R4 sweep no longer emits the unknown-schema kind at all. That PR did not update the baseline, so the ratchet has been firing on its own success ever since.

The change

Delete the entry. 220 → 219 known failures. resourcesValidated stays at 2912 and no other manifest field moves, so this is the whole fix.

R5 still reports 1934 unknown-schema issues — its pack was never given extension definitions — and that baseline is untouched here.

Verification

$ cargo test -p helios-fhir-validator --test spec_examples -- --ignored
R4: validated 2912 resources, 219 with issues
test result: ok. 1 passed; 0 failed

https://claude.ai/code/session_015Xv3QofZPCr89AgHjnATGR

`questionnaire-cqf-example.json` was recorded as a known failure for a
single `unknown-schema` issue: the core extension StructureDefinitions
were not in the vendored spec bundles, so `cqf-expression`,
`cqf-library` and `cqf-questionnaire` could not be resolved.

PR #432 (feat/363-extension-catalogue, 55e3f6d) vendored
`resources/R4/extension-definitions.json` and regenerated the R4 schema
pack, which closed exactly that gap -- the R4 sweep no longer reports
the `unknown-schema` kind at all. The baseline was never updated, so
the ratchet has been firing on its own success since 2026-08-04,
failing the scheduled run on main and every validator PR since.

Remove the entry. 220 -> 219 known failures; `resourcesValidated`
stays at 2912, so nothing else in the manifest moves.

Claude-Session: https://claude.ai/code/session_015Xv3QofZPCr89AgHjnATGR
@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@smunini
smunini merged commit d1588bd into main Aug 7, 2026
20 checks passed
@smunini
smunini deleted the fix/stale-r4-conformance-baseline branch August 7, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant