Skip to content

formal: compositional adequacy of differential dataflow operators#783

Merged
frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:formal-compositional-adequacy
Jul 4, 2026
Merged

formal: compositional adequacy of differential dataflow operators#783
frankmcsherry merged 2 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:formal-compositional-adequacy

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

Adds formal/Differential/Compositional.lean: a machine-checked account of the axis
orthogonal to the interesting-times/compaction work already in Coverage/Model — that
differential computation abstractly produces the right thing, and composes.

The single claim: each operator, as a transformation of difference traces, computes what
re-evaluating its mathematics moment by moment would — Adequate D impl := acc ∘ impl = D ∘ acc
— and these squares compose, so any program built from the operators maintains the value it was
meant to compute. All axiom-clean (propext, Classical.choice, Quot.sound; no sorryAx),
nothing sorryed.

What's proved

  • Spine. acc_injective (a trace is determined by its accumulation, so the implementing
    trace is unique), the Adequate square, and its composition law Adequate.comp — correctness
    of a program reduces to correctness of its pieces.
  • Operator squares. linImpl_adequate (LINEAR/SUM), timeImpl_adequate (enter/temporal,
    via a Galois adjoint), reduce_adequate (REDUCE — its existence half is exactly
    Coverage.exists_diff_trace_comp), and join_adequate (JOIN as a bilinear convolution =
    bilinearity + sup_le_iff).
  • Capstone. Program.adequate over id / linear / reduce / par / join / seq, by induction
    (JOIN is Adequate.join, a bilinear par).
  • Incremental refinement. Refines (per-batch increment) composes by a discrete chain rule
    (Refines.comp) — this is why difference traces are the composable interface between operators;
    the locality that makes the increment cheap for the nonlinear REDUCE is precisely the
    interesting-times argument of Coverage/Model.
  • ITERATE. Over an iteration coordinate T × ℕ: round_matches/iterate_program (the loop's
    accumulated output is the from-scratch iteration of the body's denotation), leave_stabilizes
    (stabilization is free from finite support — no termination hypothesis), leave_fixpoint with
    re-add (leave_fixpoint_enter) and start-from-X (leave_fixpoint_impulse) regimes, and, for a
    DELAY-ing (Causal) body, acc_unique — any two solutions accumulate identically, i.e.
    evaluation-order independence is the adequacy statement. acc_prod_fst shows mutual recursion
    needs no product lemma.

Two commits

  1. Compositional adequacy — the new file (purely additive: one file + one import line).
  2. Extract the accumulation core into Differential/Basic.leanacc and its group-hom API
    move out of Model.lean (where the abstraction layer was borrowing a primitive) into a
    foundational file imported by both Model and Compositional. Semantics-preserving:
    Model.streamCorrectness_holds and everything else stays axiom-clean. Coverage's
    Represents/Cut/Simple are left in place.

Each commit builds on its own. Deferred (noted in the file header): a typed Program X Y for
verified program transformations and key/value-changing joins, and the confluence question of
when different arrival schedules compute the same value (distinct from the adequacy proved here).

Nothing in the Rust workspace depends on formal/, and no CI gates on it. The README's
Contents and axiom-check instructions are updated.

🤖 Generated with Claude Code

frankmcsherry and others added 2 commits July 3, 2026 21:49
A new axis, orthogonal to the interesting-times/compaction work in
Coverage.lean and Model.lean: each operator's update-trace implementation
computes the same thing as its mathematics applied pointwise to accumulated
data (Adequate: acc ∘ impl = D ∘ acc), and this composes, so any program
maintains the value it was meant to compute, moment by moment.

Fully proved, axiom-clean (propext, Classical.choice, Quot.sound; no sorryAx):

- Spine: acc_injective (acc determines its trace), the Adequate square and
  its laws Adequate.comp / Adequate.id / Adequate.add.
- Batch operator squares: linImpl_adequate (LINEAR/SUM), timeImpl_adequate
  (enter/temporal, via a Galois adjoint), reduce_adequate (REDUCE, bridged to
  Coverage.exists_diff_trace_comp), join_adequate (JOIN convolution =
  bilinearity + sup_le_iff).
- Incremental refinement: Refines (per-batch increment) and the discrete
  chain rule Refines.comp, with Refines.of_additive for the linear case.
- Capstone Program.adequate: a deep-embedded program (id/linear/reduce/par/
  seq) interpreted two ways agrees by induction.
- ITERATE (delay-free, arbitrary seed s, round-wise body): iterate_unroll,
  round_matches (round-n state = n-th from-scratch iterate), leave_stabilizes
  (free from finite support), leave_converged, leave_loop / leave_fixpoint,
  with the re-add (leave_fixpoint_enter) and start-from-X
  (leave_fixpoint_impulse) regimes.
- ITERATE of a lifted Program body: denote_slice, round_wise (round-wise
  action = outer denotation), iterate_program; plus acc_prod_fst showing
  mutual recursion is linImpl_adequate for the projection hom.
- ITERATE with DELAY: Causal bodies (a forward delay = JOIN against a fixed
  collection, causal by sup_le_iff), acc_unique (any two loop solutions
  accumulate identically — evaluation-order independence is adequacy) and
  leave_acc_unique.

Deferred: the joinConst operator realizing LINEAR(F) as a fixed-collection
JOIN, and the confluence question (when different arrival schedules agree).

Nothing in the Rust workspace depends on this; no CI gates on it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The accumulation homomorphism `acc` and its group-hom API (`acc_add`,
`acc_zero`, `acc_single`, `acc_sub`, `acc_eq_finsupp_sum`, `acc_mapDomain`,
…) moved from `Model.lean` into a foundational `Differential/Basic.lean`
(`namespace Trace`), imported by `Model` and `Compositional`.  It is the
shared vocabulary of the whole directory, so it should not be owned by the
reduce-implementation file; `Compositional` no longer borrows a primitive
from `Model` (only the genuine bridge `Model.acc_represents` remains).

Semantics-preserving: every theorem still reports only propext /
Classical.choice / Quot.sound, `Model.streamCorrectness_holds` included.
`Coverage`'s `Represents`/`Cut`/`Simple` are left in place.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@frankmcsherry frankmcsherry marked this pull request as ready for review July 4, 2026 14:43
@frankmcsherry frankmcsherry merged commit 80f4b13 into TimelyDataflow:master-next Jul 4, 2026
6 checks passed
@frankmcsherry frankmcsherry deleted the formal-compositional-adequacy branch July 4, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant