feat(flow): postpone-and-auto-resume steps on transient provider errors

[BenderRodrigez]

Problem

A flow step whose agent run fails with a transient LLM-provider/gateway error currently fails the whole flow (routes to fallback.to). But these are self-healing capacity blips, not real failures. Three consecutive developer-react-on-jira runs for one Jira issue died this way, each on the first model call of a step (no output produced):

failed to process events: maximum retry attempts reached for HTTP 429: 8 retries
failed to process events: stream error: stream ID 147; INTERNAL_ERROR; received from peer

The provider layer already retries these (8× on 429, 3× on RST) — but when the upstream (EU-regional Bedrock via the gateway) is saturated for the whole window, the budget exhausts and the error becomes fatal. There's no way to say "the endpoint is busy, come back later."

Fix

When a step opts in via resume_after, a transient provider error parks it as postponed instead of failed, so the orchestrator's existing postpone sweep re-enters it after the delay — when the endpoint has recovered. No orchestrator change is needed: it already reads the same resume_after key to schedule the timed resume, and its resume-chain loop-breaker already bounds a persistently-degraded upstream. Steps without resume_after keep their current terminal-failure behaviour.

• flow.go — add Step.ResumeAfter. It's parsed here now so the runner can gate on it; the orchestrator already parses the same YAML key. Both sides use lenient yaml.Unmarshal, so each tolerates the field the other owns — no cross-repo lockstep.
• service.go — isTransientProviderError (string-match on the surfaced message, since the typed provider error is wrapped before it reaches the flow layer; mirrors provider.isRetryableRSTStreamError/isTransientStreamError shapes) + postponeStepForTransientError. Both step-failure paths are guarded: the retry-loop exhaustion path and handleStepError.
• This is a pause, not a failure: no AgentEvent error is emitted and no fallback is routed, so the run ends terminal-postponed and auto-resumes.

Opt-in gate is resume_after (single key, no footgun): a step can't ask to postpone but forget to set the wake timer. Deliberately not applied to must-run-now steps (e.g. a salvage step that pushes local WIP before the workspace is torn down) — a resume runs in a fresh workspace where that state is gone.

Tests

internal/flow/transient_postpone_test.go: classifier table (both observed error strings, wrapped errors, overloaded/ThrottlingException/ServiceUnavailableException, and negatives incl. a client-sent stream error that must NOT match) + the resume_after opt-in gate. gofmt, go build ./..., go vet, and the full go test ./internal/flow/... pass locally on go 1.25.8.

The consumer flow change (adding resume_after to the developer-react-on-jira work steps) is a separate MR in piano/ai-agents/piano-developer.

Companion to #19 (MySQL deadlock retry). Independent branch off main; touches only internal/flow.

🤖 Generated with Claude Code

[obukhovaa]

Review — postpone-and-auto-resume on transient provider errors

Solid, well-scoped change. I checked out the branch and independently ran go test ./internal/flow/... — green. The core mechanics hold up:

• No dangling goroutine / hang. The postpone paths just return, and defer wg.Done() (service.go:361) covers every runStep return, so the flow still converges.
• Consistent with existing postpone. The normal postpone path (service.go:520-522) already returns without emitting an agentEvent, so "pause, not a failure — no AgentEvent error, no fallback route" matches established behaviour rather than inventing a new one.
• Resume re-runs the step. collectResumableSteps re-enters a postponed row as postpone:false → status=running (service.go:1061-1067), so the parked step actually re-executes at its carried iteration. Output is nulled on park, which is correct here since a transient failure produced none.
• Classifier is conservative on the right axis. context.Canceled / context.DeadlineExceeded (graceful shutdown, step-timeout) don't match any signature, so a cancelled ctx or a step that blew its own Timeout still fails terminally instead of parking — good.

A few notes, none blocking:

1. The only real bound on a park→resume→park cycle is external (please confirm)

Within opencode nothing caps repeated transient-postpones: postpone doesn't bump maxIterations, and each resume is a fresh Run with a fresh flow timeout. A persistently-throttled endpoint would park and auto-resume indefinitely. The PR delegates this to "the orchestrator's resume-chain loop-breaker" — worth confirming that breaker actually keys on this resume chain, since it's the sole thing standing between a bad-day upstream and an unbounded resume loop.

2. resume_after is gated on presence only, not parsed as a duration (opencode side)

stepPostponesOnProviderError accepts any non-blank string. opencode never parses the duration (the orchestrator owns that), so a typo like resume_after: "15minutes" still opts the step into parking, but may hand the orchestrator a value it can't schedule a wake from — parking with a broken wake timer. Since the two repos are intentionally decoupled by lenient yaml.Unmarshal, consider validating it parses as a Go duration at flow-load so a malformed value fails fast on opencode's side too. Optional.

3. Broad substring classifier — small but nonzero false-positive surface

"rate limit", "service unavailable", "overloaded", "too many requests" match anywhere in the surfaced message. Blast radius is small today because a step-level error originates from the agent runtime (result.Error), the empty-output guard, or a ctx error — not from tool output. Just flagging that any future error surface carrying those words on a resume_after step would park rather than fail.

4. Nit: guard sits above the row-exists precondition in handleStepError

handleStepError is also called from the pre-persist error paths (NewAgent / resolveSession / resolveInteractionTarget, service.go:418/431/437) before the entry-time flow_states row is written. The new guard there would UpdateFlowState a possibly-missing row. Not reachable today (those errors carry no provider signatures) and it degrades gracefully (warn + emit postponed), so purely a note in case those error surfaces ever change shape.

---

I pushed one commit to this branch: docs(flow): document resume_after in flow-creator skill — ports the flow-spec.md reference change (the field entry + the resume_after section) so the runtime behaviour is documented alongside the code.

[BenderRodrigez]

Thanks for the thorough review — and for porting the flow-spec doc. Addressed in e7c1996 + 7aaca79:

1. Unbounded park→resume chain — you were right, it was not bounded

I traced breakResumeLoop and it bails at postpone_sweep.go:41 unless orig.MatchKey has the TeamCity build-await prefix. A transient-postpone keeps the work-item match_key, so the orchestrator breaker never fired for it — the chain was genuinely unbounded. My PR text was wrong to lean on it.

Fixed in opencode rather than depending on an orchestrator change: postponeStepForTransientError now declines to park (returns false → caller fails the step terminally) once the step's flow_states row is older than maxTransientPostponeAge (2h). created_at is set on insert and preserved across the UPDATE-on-resume, so it measures total park→resume wall-clock without a per-resume counter — which matters because, as I found tracing the resume path, neither store round-trips: collectResumableSteps walks with the caller args (they overwrite the postponed row's args at re-entry, service.go:272), and iteration is semantically overloaded with self-loop/maxIterations. created_at was the one stable hook. Happy to also relax the build-await gate in breakResumeLoop on your side if you'd prefer the bound live there too — but the flow is now self-bounding either way.

2. Validate resume_after as a duration on the opencode side — done

Added Step.ResumeAfterDuration() + a check in validateFlow, mirroring the existing TimeoutDuration() load-time validation. A typo like resume_after: "15minutes" (or a zero/negative) now fails at flow-load instead of silently handing the orchestrator a value it falls back to default on. Blank/unset stays valid (bare opt-in).

3. Broad substring classifier — acknowledged, left as-is

Agreed the blast radius is small: a step-level lastErr only comes from the agent runtime's result.Error, the empty-output guard, or a ctx error — none of which carry tool output, and ctx errors don't match any signature (verified). I kept the substrings rather than tighten-and-risk missing a real provider error shape; flagging it here so it's on record if a new step-error surface ever carries those words.

4. Guard above the row-exists precondition in handleStepError — fixed

postponeStepForTransientError now Get-or-Creates the flow_states row instead of a blind UpdateFlowState, so it's correct even from the pre-persist setup paths (unreachable today since those errors carry no provider signature, but no longer a latent no-op).

Tests: added TestStepResumeAfterDuration; gofmt / go build ./... / go vet / go test ./internal/flow/... all green on go 1.25.8.