fix(*): give web search its config section so maxResults takes effect - #314
fix(*): give web search its config section so maxResults takes effect#314Handsome-wzw wants to merge 2 commits into
Conversation
`tools.web.search.maxResults` has been declared in the schema and read by nobody. Both registration sites built the tool from the key alone, so it fell back to its own default of 5 and a deployer who set the field got no effect and no warning. The cause is that web search was the one tool handed a single field instead of its config section. Every neighbour takes a section -- media_config, exec_config, sandbox_config, context_config -- and the three CLI entry points already held `config.tools.web.search` and reached into it to pull one value out. Passing the section instead is what wires the field, and it retires the `brave_api_key` parameter name in the same move: it has named a Serper key ever since the Brave backend went away, which a reviewer flagged on the capability PR as pre-existing and misleading. `WebSearchTool` keeps its own signature. It is still constructible from a bare key, which is what its unit tests and the error-path test rely on; only the callers that have a config section now pass one. The sub-agent surface gets the same treatment, and a test of its own. A first mutation pass caught the main loop dropping `max_results` and said nothing when the sub-agent did, which is the shape this area keeps taking: two registration sites, a fix applied to one. Five mutations now, all caught. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
gloryfromca
left a comment
There was a problem hiding this comment.
Blocking: one caller was missed by the brave_api_key -> web_search_config rename and now raises at construction; see the inline note.
The fix itself is right and the tests are honest. WebSearchTool.execute really does read self.max_results (n = min(max(count or self.max_results, 1), 10)), so passing the section instead of one field out of it is what makes tools.web.search.maxResults reach the request -- the PR title is accurate. Both registration sites are covered, and both new tests assert the value on the registered tool object rather than on the config that was passed in, so they would catch a regression at either site.
The subagent_run refactor is a strengthening, not a loosening: the spy still asserts runs is non-empty before appending, the collector still opens before the manager is constructed, and the existing test kept its "baselines first" guard against an empty list satisfying the not in. The three call sites in raven/cli/ and the two capability tests all pass a section whose api_key was set first, so nothing silently stopped being gated.
What I checked
- AGENTS.md: commit header (
fix(*): ..., 61 chars, ASCII, lowercase,(*)for multi-scope) and branch name both conform; the reworded comment inmain.pystays English and stays a why; no new test files, both edits land in the existingtest_agent_loop_web_tools.py, so 5.1/5.4 hold. - Callers: grepped every
AgentLoop(/SubagentManager(construction in the repo.benchmarks/pinchbench/direct/raven_executor.py,benchmarks/pinchbench/bot_runner/bot_executor.pyandbenchmarks/appworld/agent_cli.pypass no search kwarg at all;benchmarks/clawbench/stream.pyis the one that breaks. - Backward compat:
brave_api_keyand theself.brave_api_keyattribute are both gone from the public signature with no shim. In-repo that costs exactly the one call site below; if anything outside the repo constructsAgentLoopdirectly, it is a breaking change worth a line in the PR body. - Tests:
uv run pytest tests/test_agent_loop_web_tools.py tests/test_tool_capabilities.py tests/test_agent_loop_tool_search.py -q-> 42 passed, 0 skipped.
Not blocking
max_results is clamped to 10 inside execute, so a deployer who sets maxResults: 20 still gets 10 and no warning. That is pre-existing behaviour on lines this PR does not touch, and the config field is now at least honoured up to the cap -- just noting it since "maxResults takes effect" is the headline.
| max_iterations: int = 40, | ||
| context_window_tokens: int | None = None, | ||
| brave_api_key: str | None = None, | ||
| web_search_config: "WebSearchConfig | None" = None, |
There was a problem hiding this comment.
This rename misses a caller. benchmarks/clawbench/stream.py:146 still passes brave_api_key=self.config.tools.web.search.api_key or None, and AgentLoop.__init__ takes no **kwargs, so every clawbench run now dies before the first task:
$ uv run python -c "from raven.agent.loop import AgentLoop; AgentLoop(provider=None, workspace='.', brave_api_key='k')"
TypeError: AgentLoop.__init__() got an unexpected keyword argument 'brave_api_key'
Ruff will not catch it (wrong kwarg, not a lint), and no test constructs that harness, so CI stays green while the benchmark entry point is broken. The one-line fix mirrors the three CLI sites:
web_search_config=self.config.tools.web.search,Worth grepping for brave_api_key once more before merge -- that is the only remaining hit today.
There was a problem hiding this comment.
Fixed in cc715dc, and better than I asked for. benchmarks/clawbench/stream.py:146 now passes web_search_config=self.config.tools.web.search, and you also dropped the everos_config= line next to it -- which I had missed. That one is the more interesting half: everos_config is not in AgentLoop.__init__ either, so clawbench was already dead on the target branch and my "this PR breaks it" framing was only half right.
The static guard is the right shape for this. I checked it is not vacuous -- reverting the kwarg makes both tests fail with the offending file and line named:
$ uv run pytest tests/test_benchmark_entry_points.py -q # with brave_api_key= put back
FAILED test_the_benchmarks_still_construct_agentloop_with_parameters_it_has
FAILED test_a_retired_parameter_is_gone_from_the_benchmarks_too[brave_api_key]
2 failed, 1 passed
and on the tree as it stands, 3 passed -- it finds all four AgentLoop( call sites under benchmarks/, so pinchbench and appworld are covered too, not just clawbench. The assert seen guard is what keeps it honest if the tree moves.
One thing this does not reach: it matches ast.Name only, so a module.AgentLoop(...) call would slip past. Nothing in benchmarks/ calls it that way today, so it is a note, not a request.
Two keyword arguments in `benchmarks/clawbench/stream.py` no longer name parameters of `AgentLoop.__init__`, and either is fatal at construction: - `brave_api_key` — renamed to `web_search_config` by this branch, and this is the one caller the rename missed. - `everos_config` — retired long before this branch (no CLI site passes anything in its place; the memory backend reaches the loop through the plugin registry). Passing it raises on `main` today, so clawbench was already dead before the rename; fixing only the first would have left it dead. Adds a static guard, because nothing else looks: `benchmarks/` sits outside the package, pulls in dependencies the suite does not have, and is never imported by a test, so a stale keyword there is invisible to both ruff and CI. The new test reads every `AgentLoop(...)` under `benchmarks/` with `ast` and checks its keywords against the live signature — no harness dependency required. Co-authored-by: Claude (claude-opus-5) <noreply@anthropic.com>
gloryfromca
left a comment
There was a problem hiding this comment.
Blocking, but only on the commit message: cc715dc9's body has three em-dashes, and the repo's own lint rejects the whole message as non-ASCII -- the Commit lint workflow will be red. The code in this revision is good and my earlier finding is properly fixed.
The blocker
scripts/commit_lint.py runs _is_ascii over the entire message, not just the header (_check_message does text = message.strip() and checks that before parsing the header), and .github/workflows/commits.yml runs check_commit_messages.py "$BASE..HEAD" across the PR range. Reproduced with the repo's own script:
$ PYTHONPATH=. uv run python scripts/check_commit_messages.py \
"github/feat/tool_capability_inventory_gh..HEAD"
Invalid commit message cc715dc93be1225caed21e603554b53b33f70a52:
fix(benchmarks): keep clawbench constructing an AgentLoop that exists
- must be ASCII-only English
The three offenders are all in the body, on the lines starting - `brave_api_key`, - `everos_config` and keywords against the live signature. AGENTS.md 3.1.1 names this case specifically ("not just no full-width punctuation ... but also no em-dash"). 3628c146 is clean; only the new commit fails. Swapping each -- in and rewording is a message rewrite, which per 3.4 needs the maintainer's word first.
Worth noting the squash does not save you here: the squash body comes from the PR description, but commits.yml lints every commit in the range before the merge, so this fails the PR, not just the history.
The code
No blockers. The clawbench fix mirrors the three CLI sites exactly, and dropping everos_config alongside it is right -- that parameter is not on AgentLoop.__init__ either, so clawbench was already dead on the target branch and the rename was only the second of two fatal kwargs. I said the opposite last round; the commit body has it right.
The guard test earns its place and is not vacuous -- details in my reply on the main.py thread. Short version: reverting the kwarg makes both tests fail by name and line, it reaches all four AgentLoop( call sites under benchmarks/ (so pinchbench and appworld are covered too), and assert seen stops it passing over an empty sweep.
What I checked on this delta
- AGENTS.md: commit header is fine (
fix(benchmarks)has precedent in history, no scope-enum rule incommitlint.config.cjs, 69 chars, lowercase, no full stop,Co-authored-bypresent); the body is the problem.tests/test_benchmark_entry_points.pyis not a CLI test, so 5.1'stest_cli_<module>_commands.pyshape does not bind it, and the file itself is ASCII-clean. - Callers and history: traced
agents.defaults.everos--raven/config/loader.py:184drops the block as retired on migration, and no CLI site passes a replacement, so nothing is lost by deleting the line rather than porting it. - Tests, and whether any were weakened:
uv run pytest tests/test_benchmark_entry_points.py tests/test_agent_loop_web_tools.py tests/test_tool_capabilities.py tests/test_agent_loop_tool_search.py -q-> 45 passed, 0 skipped. Nothing in the delta touches an existing assertion; it only adds. - Backward compatibility: unchanged from the last revision --
brave_api_keyis gone from the public signature with no shim, and the repo now has zero references to it.
Not blocking
test_a_retired_parameter_is_gone_from_the_benchmarks_too greps for the literal "<name>=", so it would also fire on an unrelated local named everos_config. No such name exists under benchmarks/ today and the failure text would point straight at the file, so this is fine as it stands.
Summary
Stacked on #312 -- set the base branch to
feat/tool_capability_inventory_gh,not
main. Againstmainthe diff would carry that PR's commits as well. Itcloses the naming nit raised in review there, so it is a follow-up rather than
an independent change.
tools.web.search.maxResultshas been declared in the schema and read bynobody. Both registration sites built the tool from the key alone:
so it fell back to its own default of 5, and a deployer who set the field got no
effect and no warning that the setting was inert.
One cause, two symptoms
Web search was the one tool handed a single field instead of its config section.
Every neighbour takes a section --
media_config,exec_config,sandbox_config,context_config-- and all three CLI entry points alreadyheld
config.tools.web.searchand reached into it to pull one value out:Passing the section is what wires the field. It also retires the parameter name
in the same move, which is the second symptom:
brave_api_keyhas named aSerper key ever since the Brave backend went away. That was flagged in review on
#312 as pre-existing and misleading, correctly, and it is not a separate rename
-- passing one field out of a section is exactly why the parameter ended up
named after a vendor rather than after what it configures.
WebSearchToolkeeps its own signature. It is still constructible from a barekey, which its unit tests and the error-path test rely on; only the callers that
have a config section now pass one.
The sub-agent surface
Same change there, and a test of its own. This matters more than it looks: a
first mutation pass caught the main loop dropping
max_resultsand said nothingwhen the sub-agent dropped it, because nothing was watching that call site. Two
registration sites with a fix applied to one is the shape this area keeps
taking, and it is the same gap the gate in #312 had to close by hand.
Type
The parameter rename is a refactor riding along, not a second topic: it is the
same edit that wires the field.
Verification
The one failure is
test_read_file_image.py::test_an_attachment_that_cannot_be_ read_costs_a_note_not_the_turn, which fails the same way onmainwith thesecommits absent: it makes a file unreadable with
chmod 000, which does notblock root, so it fails for anyone running the suite as root and passes in CI.
#313 is the fix. This branch touches neither that file nor the code under it.
Five mutations, each caught:
The second line is the one worth reading. Before its test existed that mutation
passed clean -- the wiring was fixed at both call sites and asserted at one.
Risk
One behaviour change, and it is the point:
tools.web.search.maxResultsnowreaches the tool. An install that set it was getting 5 results and now gets what
it asked for. An install that never set it is unaffected -- the schema default
is 5, which is also the value the tool fell back to. The per-call
countargument still wins over both, unchanged.
brave_api_keyis gone fromAgentLoopandSubagentManager. Both areconstructed inside this repo only -- the three CLI entry points and the tests --
and all were updated; no deprecation shim, since nothing external names it.
Rollback is a revert. No configuration is read differently beyond the field that
was already meant to be read.
Related Issues
N/A