FIX: Curate Scam scenario DEFAULT strategy#2127
Open
varunj-msft wants to merge 1 commit into
Open
Conversation
Standardize the Scam scenario's default run per story #8380: - Add a DEFAULT aggregate covering the single-turn techniques (ContextCompliance, RolePlay). The multi-turn PersuasiveRedTeamingAttack is excluded from DEFAULT so a no-strategy run stays fast; it is still reachable via ALL or MULTI_TURN. - Flip default_strategy from ALL to DEFAULT. VERSION bumped to 2. Tests and scanner docs updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of standardizing our scenarios.
Previously the Scam scenario had
default_strategy=ScamStrategy.ALL, so a run with no--strategiesexecuted every technique, including the multi-turn PersuasiveRedTeamingAttack. That multi-turn attack runs several turns per objective and makes the default run slow and expensive, which isn't a good out-of-the-box default.This change adds a curated DEFAULT aggregate and points the scenario's default at it:
DEFAULT = ("default", {"default"})to ScamStrategy and tags the two single-turn techniques (ContextCompliance, RolePlay) with "default". PersuasiveRedTeamingAttack (multi-turn) is intentionally left out of DEFAULT.default_strategyfrom ALL to DEFAULT.So a plain
pyrit_scan airt.scamnow runs the two fast single-turn techniques. The multi-turn attack is still fully available via--strategies all,--strategies multi_turn, or--strategies persuasive_rta.This follows the same manual-enum DEFAULT convention already used by the garak WebInjection and Doctor scenarios.
VERSION is bumped 1 -> 2 since the default run behavior changes.
Tests and Documentation
Tests (tests/unit/scenario/airt/test_scam.py):
ScamStrategy.ALLexplicitly and assert all three techniques run.All scam unit tests pass; ran ruff, ruff format, and ty clean.
Documentation: