SPIKE: draft flow specs from a screen-recording video (see #227) - #242
Draft
HappyDevs1 wants to merge 2 commits into
Draft
SPIKE: draft flow specs from a screen-recording video (see #227)#242HappyDevs1 wants to merge 2 commits into
HappyDevs1 wants to merge 2 commits into
Conversation
Context: docs/recording.md and issue #227 both state a video should never be a machine-parsed input - the machine surface is structured data, video is the human surface. This module exists anyway, as a deliberate, scoped experiment (not a decided direction), to find out concretely whether video-derived drafts are accurate enough to be worth the ambiguity that position warns about. Extracts key frames from a screen recording (ffmpeg, shelled out - never runs in CI, only locally at authoring time) and asks a vision-capable model to describe the single action bridging each frame pair, in the same step vocabulary flowproof's existing rules/author already use. Never infers assert steps: a recording shows behaviour, not a belief about correctness, so the draft has zero assertions by construction and still needs a human to add at least one, then survive one live `flowproof record` pass, before it produces anything replayable. New `flowproof author-from-video` CLI subcommand, marked EXPERIMENTAL in its own help text.
Confirmed via direct API reproduction: current claude-sonnet-5 rejects any temperature field outright (400 "temperature is deprecated for this model"), so every describe_transition call was failing before this spike could be tried against a real recording. The same deprecation also hits the actual authoring path (llm.rs), tracked and fixed separately there. Live-tested against a real SAP screen recording: correctly inferred the transaction navigated to and the exact value typed, from pixels alone.
HappyDevs1
force-pushed
the
feat/author-from-video
branch
from
July 31, 2026 14:13
2e8ba66 to
7b74edd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A deliberate, scoped experiment — not a decided direction, not intended to merge as-is.
docs/recording.mdand issue #227 both state the repo's standing position: structured data is the machine surface, video is the human surface, and nothing programmatic ever parses pixels. Issue #227 explicitly lists video parsing as a non-goal, and proposes session recordings + requirements documents instead.What it does and does not do
ffmpeg, shelled out — never bundled, never run in CI, only ever local at authoring time).assert:steps. A recording shows behaviour, not a belief about correctness — same reasoning Authoring: derive flows from real material, with an acceptance gate that the spec can actually fail #227 makes for session recordings. A human adds at least one assertion by hand..flow.yamlonly. It still has to survive one liveflowproof recordpass against the real app before it produces anything replayable — that pass is what actually catches a misidentified field, not a read-through of the draft text.flowproof author-from-videosubcommand, markedEXPERIMENTALin its own--helptext.Known gap: diff size
This diff is ~425 changed lines against the repo's 400-line ratchet cap (
scripts/gate/ratchets.sh), so that check is expected to fail here. It's one cohesive module (frame extraction, vision inference, draft assembly) plus the CLI surface needed to actually exercise it end-to-end — splitting the vision-inference piece into a separate follow-up PR would defer the exact question this spike exists to answer. Flagging it rather than gaming it.What would still be needed to actually evaluate this
Real end-to-end validation needs an actual SAP screen recording,
ffmpeg, and a live API key — none of which exist in this sandbox. The unit tests cover what's deterministic (draft assembly round-trips throughFlowSpec::parse, no assertions are ever emitted, missing-ffmpeg errors clearly), but the actual question — is the vision-inferred draft any good — needs a real video to answer.