feat: marketplace & local plugins experiment toggle#736
Open
Groenbech96 wants to merge 21 commits into
Open
Conversation
…gins experiment toggle Adds a design spec and a task-by-task implementation plan for a new experiment toggle that installs agent plugins (a marketplace pinned to a git commit, or a local marketplace) into the running CLI via its own 'plugin marketplace add' + 'plugin install' commands, records them in ExperimentConfiguration, and removes them after the run. Docs only; no code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…iew) Addresses review: execution-based categories run entries as a parallel matrix (max-parallel 64) on a shared self-hosted runner, and the CLI plugin store is user-scope/global, so concurrent installs/teardowns would race. Fix: install into a fresh per-entry config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under .bcbench), applied to both the plugin commands and the agent launch. This removes teardown entirely (isolated home discarded with the checkout) and moots the teardown_plugins signature comment. Verified live on both CLIs (fresh home: install + skill load + headless session, real config untouched). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…r-entry home) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es, dedupe marketplace clone Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Move the terminal AgentError out of 'case _' to after the match block in _materialize and _home_env_var, matching the existing pattern in types.py (EvaluationCategory). Behavior unchanged; clears the github-code-quality mixed-returns advisory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Groenbech96
commented
Jul 7, 2026
Per PR #736 review: move the shutil.which(...) resolution + not-found check to the top of run_copilot_agent / run_claude_code (before the 'Running...' log and all setup work), so a missing CLI fails fast. Behavior otherwise unchanged; the resolved binary is still passed to setup_plugins_from_config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI's pre-commit 'ty check' hook failed: add_call[1]/install_call[1] were typed
dict | None (kwargs.get('env')) and thus not subscriptable. Record env as a
non-optional dict (kwargs.get('env') or {}). ty now exits 0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a self-contained example marketplace (bcbench-example-marketplace + bcbench-example-plugin bundling one no-op skill) under instructions/microsoft-BCApps/plugins/bcbench-example/, referenced by a disabled local entry in config.yaml. Flip enabled: true to smoke-test that a plugin installs into the per-entry isolated home and its skill loads (verified live: marketplace add + install + skill list all succeed). Guarded by a test tying config.yaml -> example dir -> marketplace.json -> plugin.json so the example can't silently rot. EXPERIMENT.md documents it and the built-in marketplace-name collision caveat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…(revert before merge) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, install) Adds INFO logging around plugin setup so CI logs clearly show each step — 'Installing N plugin marketplace(s) into isolated COPILOT_HOME=...', 'Cloning plugin marketplace repo@commit' / 'Copying local plugin marketplace ...', 'Registering marketplace ... from ...', and 'Installing/Installed plugin ...'. Previously only the post-install line (and clone_at_commit's clone log) showed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Local `plugins` entries resolved their `path` under instructions/<owner>-<repo>/, which broke for dataset entries whose repo (e.g. microsoftInternal/NAV) differs from where the example was stored. Resolve `local` sources from a fixed, repo-agnostic base (src/bcbench/agent/shared/plugins/) instead, and move the bundled bcbench-example there. Update tests, spec, plan, EXPERIMENT.md, and the config.yaml comments/path accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `marketplace` plugin source now accepts an optional `path` (marketplace root within the clone), mirroring `local`, so repos that host a marketplace in a subdirectory are supported. Enable a self-contained marketplace example that clones this repo at a pinned commit and points `path` at the bundled example, so an evaluation run visibly logs the plugin clone under bcbench.operations.git_operations (the local copy path logged no clone). The local example is kept as a disabled, test-validated template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a real-world marketplace plugin example (obra/superpowers, marketplace "superpowers-dev", plugin "superpowers") pinned to v6.1.1, enabled for a smoke run so the agent can discover and invoke a plugin skill (e.g. brainstorming) via the `skill` tool in headless -p mode. The prior bundled examples are kept as disabled, test-validated templates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the marketplace & local plugins experiment toggle designed in #733. This branch includes the design spec + implementation plan (from #733) plus the full implementation.
What
A config-driven
pluginsexperiment toggle (peer ofinstructions/skills/agents/mcp) that installs agent plugins into the running CLI so their skills are native during a run — measured like the other experiment levers.marketplace(repo + pinnedcommit) orlocal(path); per-entryenabled; multiple at once.plugin marketplace add+plugin install) — config injection (extraKnownMarketplaces/enabledPlugins) is trust-dialog-gated and ignored in headless mode, so it cannot be used (verified + corroborated by an independent write-up).<repo>/.bcbench/plugins/.COPILOT_HOME/CLAUDE_CONFIG_DIRunder.bcbench/), applied to the plugin commands and the agent launch — so themax-parallel: 64matrix on the shared self-hosted runner never shares the user-scope plugin store. No teardown; nothing touches the developer's real config.ExperimentConfiguration.plugins: list[str]("name@commit"/"name@local"), mirroringmcp_servers.How it was built
Executed the plan task-by-task (subagent-driven, fresh agent per task + per-task review + final whole-branch review):
ExperimentConfiguration.pluginsfieldclone_at_commitgit helper (pinned shallow clone)plugin_operations.pymaterialize + marketplace-name helperssetup_plugins_from_config(per-entry isolated home install)Verification
marketplace add+installinto a freshCOPILOT_HOME/CLAUDE_CONFIG_DIR, skill loads, realcopilot -psession runs (auth via env token, no onboarding), real config untouched.Notes / follow-up
CLAUDE_CONFIG_DIRauthenticated headless session (viaANTHROPIC_API_KEY) is the one residual runtime check to confirm in CI (Copilot's is verified); noted in the spec's open questions.