Skip to content

FIX: Curate Scam scenario DEFAULT strategy#2127

Open
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/8380-Standardizing-Scenarios-Scam
Open

FIX: Curate Scam scenario DEFAULT strategy#2127
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/8380-Standardizing-Scenarios-Scam

Conversation

@varunj-msft

Copy link
Copy Markdown
Contributor

Description

Part of standardizing our scenarios.

Previously the Scam scenario had default_strategy=ScamStrategy.ALL, so a run with no --strategies executed every technique, including the multi-turn PersuasiveRedTeamingAttack. That multi-turn attack runs several turns per objective and makes the default run slow and expensive, which isn't a good out-of-the-box default.

This change adds a curated DEFAULT aggregate and points the scenario's default at it:

  • Adds DEFAULT = ("default", {"default"}) to ScamStrategy and tags the two single-turn techniques (ContextCompliance, RolePlay) with "default". PersuasiveRedTeamingAttack (multi-turn) is intentionally left out of DEFAULT.
  • Flips default_strategy from ALL to DEFAULT.

So a plain pyrit_scan airt.scam now runs the two fast single-turn techniques. The multi-turn attack is still fully available via --strategies all, --strategies multi_turn, or --strategies persuasive_rta.

This follows the same manual-enum DEFAULT convention already used by the garak WebInjection and Doctor scenarios.

VERSION is bumped 1 -> 2 since the default run behavior changes.

Tests and Documentation

Tests (tests/unit/scenario/airt/test_scam.py):

  • Added a strategy-enum test class asserting DEFAULT expands to exactly {context_compliance, role_play}, DEFAULT excludes persuasive_rta, and ALL still includes persuasive_rta.
  • Added an end-to-end test that a no-strategy (default) run yields only the two single-turn atomic attacks.
  • Made the existing "all" generation test pass ScamStrategy.ALL explicitly and assert all three techniques run.
  • Updated the default_strategy and VERSION assertions.
    All scam unit tests pass; ran ruff, ruff format, and ty clean.

Documentation:

  • Updated the Scam section of doc/scanner/airt.py (and the paired airt.ipynb) to list the DEFAULT aggregate and explain that the multi-turn technique is opt-in. The code sample uses an explicit strategy and is unchanged, so no re-execution was required; airt.ipynb was kept in sync with the .py.

Standardize the Scam scenario's default run per story #8380:

- Add a DEFAULT aggregate covering the single-turn techniques
  (ContextCompliance, RolePlay). The multi-turn PersuasiveRedTeamingAttack is
  excluded from DEFAULT so a no-strategy run stays fast; it is still reachable
  via ALL or MULTI_TURN.
- Flip default_strategy from ALL to DEFAULT.

VERSION bumped to 2. Tests and scanner docs updated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant