GraphFlow Agent Bench

GraphFlow Agent Bench is a reproducible, paired A/B benchmark for one practical question: does a Codex workflow grounded in deterministic Joern graph analysis make safer and more behaviorally correct control-flow refactorings than the same model working from source, compiler feedback, sanitizers, and visible tests alone?

The treatment is deliberately described as a workflow-level intervention. It combines Joern access, graph-oriented instructions, mandatory before/after snapshots, and fail-closed evidence checks; this first experiment does not isolate Joern availability from those surrounding practices. Initial results are exploratory and must not be read as a statistical-significance claim.

Protocol version 1 remains frozen at protocol-v1. The current protocol-v2 amendment corrects one Task 5 Joern oracle that could not resolve an indirect function-pointer call; it does not change prompts, baselines, or score weights.

A new, deliberately separate exploratory lane tests machine-native complexity with an incremental HTTP/1.1 chunked-transfer decoder. It uses protocol identifier machine-native-v1, task identifier mn_http_chunked, and separate fixtures, prompts, results, reports, and commands. It does not modify or reinterpret either frozen protocol.

Conditions and isolation

Both conditions receive the same canonical prompt bytes, model, high reasoning effort, timeout, immutable source, and visible tests. A seeded coin flip determines sequential run order. Control is placed in an outer Bubblewrap filesystem boundary, uses ignored Codex user configuration, has Joern-containing PATH entries removed, has Joern installations masked, and cannot see the benchmark repository, evaluator, references, generated cases, treatment artifacts, or the competing run. Treatment starts from the identical fixture and prompt but receives the verified joern-agent-mcp STDIO server, a graph workflow AGENTS.md, project configuration, and enforcement hooks.

flowchart LR
  Protocol[Preregistered protocol] --> Pair[Seeded pair coordinator]
  Pair --> C[Control sandbox]
  Pair --> T[Treatment sandbox]
  C --> E[Clean evaluator]
  T --> E
  E --> R[Append-only results and reports]

Loading

sequenceDiagram
  participant P as Pair runner
  participant A as First randomized condition
  participant B as Second condition
  participant E as Evaluator
  P->>P: Record order and evaluator seeds
  P->>A: Fresh baseline + canonical prompt
  A-->>P: JSONL, patch, final response
  P->>E: Candidate source only
  P->>B: Fresh identical baseline + prompt
  B-->>P: JSONL, patch, final response
  P->>E: Candidate source only
  E-->>P: Runtime, sanitizer, graph, score

Loading

flowchart TB
  subgraph Control
    CS[Source + visible tests]
    CP[Sanitized PATH]
    CM[Joern and evaluator masked]
  end
  subgraph Treatment
    TS[Identical source + tests]
    JM[Joern MCP]
    GH[Graph instructions + hooks]
  end
  CS --> CodexC[Same Codex model]
  TS --> CodexT[Same Codex model]
  JM --> CodexT
  GH --> CodexT

Loading

flowchart LR
  Source[Final candidate source] --> Build[Strict GCC/Clang]
  Source --> Cases[Post-run seeded cases]
  Source --> San[ASan/UBSan/LSan]
  Source --> Joern[Independent Joern analysis]
  Source --> Scope[Patch scope]
  Build --> Score[Preregistered 100 points]
  Cases --> Score
  San --> Score
  Joern --> Score
  Scope --> Score

Loading

Tasks and scoring

The harmless C11 warehouse-telemetry state machine covers parsing, validation, dispatch, state changes, retries, nested branches, multiple exits, allocation, and shared cleanup. Five tasks exercise validation-path preservation, cleanup on every exit, state-transition integrity, loop progress, and dispatch consistency. Each has a hashed canonical prompt, deterministic manifest, visible tests, generated evaluator cases, graph invariants, a reference solution, and a known-bad mutation.

The frozen score is: generated behavior 40, graph invariants 20, sanitizers 15, visible regressions 10, strict build 5, scope/minimality 5, and fidelity 5.

Initial exploratory result

The repaired protocol-v2 demonstration ran one pair for each task. Control scored 100.00 on all five tasks. Treatment scored 100.00 on Tasks 1, 2, and 5, and 95.00 on Tasks 3 and 4 because required treatment evidence was absent. The mean paired difference was −2.00, the median was 0.00, and the win/loss/tie count from the treatment perspective was 0/2/3. Both conditions passed every generated behavioral case and independently evaluated graph invariant in the v2 suite.

These five pairs are exploratory and do not support a statistical-significance claim. In particular, they do not support claiming that the Joern-assisted workflow improved measured performance in this demonstration. See the generated report and how to interpret it.

Quickstart

python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/graphflow-bench doctor
.venv/bin/pytest

This project depends on the existing joern-agent-bridge integration and a compatible Joern installation. doctor verifies the actual executables and outer isolation.

Run a smoke pair:

.venv/bin/graphflow-bench run-pair task1 --order-seed 20260724

Run the five-task suite and generate reports:

.venv/bin/graphflow-bench run-suite --seed 20260724
.venv/bin/graphflow-bench report

Run three future paired repetitions per task:

.venv/bin/graphflow-bench run-suite --repetitions 3 --seed 20260724

Create blinded review bundles and regenerate reports:

.venv/bin/graphflow-bench anonymize reports/blinded --seed 20260724
.venv/bin/graphflow-bench report

See the experiment protocol, condition isolation, and limitations. Generated results are append-only by default. Paid Codex experiments are opt-in and never run in public CI.

Licensed under Apache License 2.0.