test(validator): drop the R4 baseline entry that #432 already fixed - #517
Merged
Conversation
`questionnaire-cqf-example.json` was recorded as a known failure for a single `unknown-schema` issue: the core extension StructureDefinitions were not in the vendored spec bundles, so `cqf-expression`, `cqf-library` and `cqf-questionnaire` could not be resolved. PR #432 (feat/363-extension-catalogue, 55e3f6d) vendored `resources/R4/extension-definitions.json` and regenerated the R4 schema pack, which closed exactly that gap -- the R4 sweep no longer reports the `unknown-schema` kind at all. The baseline was never updated, so the ratchet has been firing on its own success since 2026-08-04, failing the scheduled run on main and every validator PR since. Remove the entry. 220 -> 219 known failures; `resourcesValidated` stays at 2912, so nothing else in the manifest moves. Claude-Session: https://claude.ai/code/session_015Xv3QofZPCr89AgHjnATGR
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
r4_spec_examples_match_baselinefails with a single divergence — an improvement the ratchet refuses to accept silently:This is not specific to any one PR. It reproduces on
main, it fails the scheduledvalidator-conformance.ymlruns (2026-08-05, 2026-08-06), and it fails thePack smoke + spec example corpusjob on every validator PR since — #461, #465, #472 among them.Root cause
The baseline recorded exactly one entry whose only issue kind was
unknown-schema:#432 (
feat/363-extension-catalogue, 55e3f6d, merged 2026-08-04) vendoredresources/R4/extension-definitions.jsonand regeneratedfhir_schemas_r4.json.gz(180KB → 208KB). The file's three canonicals —cqf-expression,cqf-library,cqf-questionnaire— resolve now, so it validates clean. The R4 sweep no longer emits theunknown-schemakind at all. That PR did not update the baseline, so the ratchet has been firing on its own success ever since.The change
Delete the entry. 220 → 219 known failures.
resourcesValidatedstays at 2912 and no other manifest field moves, so this is the whole fix.R5 still reports 1934
unknown-schemaissues — its pack was never given extension definitions — and that baseline is untouched here.Verification
https://claude.ai/code/session_015Xv3QofZPCr89AgHjnATGR