MAINT: Standardize garak Doctor default strategy to DEFAULT#2128
Open
varunj-msft wants to merge 1 commit into
Open
MAINT: Standardize garak Doctor default strategy to DEFAULT#2128varunj-msft wants to merge 1 commit into
varunj-msft wants to merge 1 commit into
Conversation
Flip the Doctor scenario's default_strategy from ALL to DEFAULT for consistency with the other standardized scenarios (matching WebInjection). Both Policy Puppetry techniques are already tagged "default", so DEFAULT and ALL resolve to the same techniques -- this is a behavior-preserving change and the VERSION is intentionally unchanged (the resume eval-hash derives from the resolved techniques, not the default-strategy label).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of standardizing our scenarios.
Flips the garak Doctor scenario's
default_strategyfrom ALL to DEFAULT so it matches the convention used by the other standardized scenarios (e.g. WebInjection). DoctorStrategy already defines a DEFAULT aggregate and both concrete techniques (PolicyPuppetry, PolicyPuppetryLeet) are already tagged "default", so DEFAULT and ALL currently resolve to the same two techniques.This is a behavior-preserving change: a no-strategy run executes the same techniques as before. Its value is consistency and future-proofing — if a non-default technique is added later, ALL and DEFAULT diverge correctly without another change to the default.
VERSION is intentionally left unchanged. The scenario's resume/identity eval-hash is derived from the resolved techniques (plus datasets, params, target, scorer), not from the default_strategy label, so resolving to the same techniques means the hash is unchanged and existing results still resume. Bumping the version would only cause spurious invalidation.
Tests and Documentation
DoctorStrategy.ALLexplicitly (previously it relied on the default), so ALL and DEFAULT are both covered.All doctor unit tests pass; ran ruff, ruff format, and ty clean.
Documentation: none needed — no user-facing docs reference the Doctor default strategy.