Fix issues that made long runs fail or hang - #182
Open
chandrakananandi wants to merge 3 commits into
Open
Conversation
A long authoring turn (Opus thinking over a large prompt) can exceed the SDK's 600s non-streaming ceiling, and `timeout=None` explicitly DISABLES the SDK's timeouts (an explicit None is not not-given), so a socket that died silently mid-call hung the session forever — both observed on Crucible solana_vault runs. Stream every request so bytes keep flowing (no ceiling, no idle window for NAT killers to hit), and bound each httpx phase at 300s so a dead socket surfaces in minutes; for a streamed response that bounds the silence between chunks, not the whole turn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nomic model's remote code caches positional tensors per sequence length, so concurrent encodes race: every concurrent crucible_docs_search died with tensor-shape mismatches (the tool degrades to "no results", so authoring ran ungrounded and hallucinated the crucible API), and on Apple Silicon the auto-picked MPS backend segfaulted the whole process inside torch's Metal shader cache. One process-wide lock serializes encodes -- queries are short, so contention is noise -- and COMPOSER_EMBED_DEVICE=cpu lets a Mac host opt out of MPS entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
State read back through the Postgres checkpointer can carry raw dicts where the schema declares models (the serializer's fallback when it cannot reconstruct the class). The readback then died on "'dict' object has no attribute 'property_title'" -- after every component session had already finished its paid authoring -- and the campaign reported empty with exit 0. Revalidate at the boundary so the readback is typed either way. Symptom fix: why the serializer falls back at all is still open, and other checkpoint readers may want the same guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
I ran into this myself today. It looks like it's an issue with the new "tool family" thing. I don't think this is the right fix though; I will open a separate PR for this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was trying to run AutoProver (crucible on the solana_vault demo) via Fable and it had to do these fixes to get it working. I wanted to make a PR with the fixes, in case these might help others too.
From Claude:
LLM calls died at 10 min, or hung forever. Non-streaming calls hit the Anthropic SDK's 10-minute limit, and our
timeout=Nonedisables timeouts completely, so a dropped connection hung the run indefinitely. Fix: stream responses (no limit) and time out after 5 min of silence. This will affects every Anthropic call in the repo. Is that ok?Doc search: broke under parallel agents. The embedding model isn't thread-safe: concurrent searches returned no results on CPU and segfaulted on Mac GPUs. Fix was to run one encode at a time; add COMPOSER_EMBED_DEVICE=cpu to skip the GPU.
Report: empty despite successful work. State read back from Postgres sometimes arrives as plain dicts instead of typed objects, crashing the final step after all the paid LLM work was done. Fix was to re-validate on read. (Why the round-trip does this, and why an all-failed run still exits 0, are left open.)