Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .claude/board/EPIPHANIES.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,41 @@
## 2026-08-02 — E-THE-DURABLE-UNIT-IS-THE-CYCLE-NOT-THE-CAST-ONE-WAL-WRITE-PER-SWEEP-1 — the persistence seam is reshaped to the WAL-amortized generation; per-cast durability retired

**Status:** FINDING (operator-ruled 2026-08-02 — the decision on the fork Correction 2(c) of `E-THE-PAIRED-MOVE-MUST-BE-DURABLE-CO-LOCATED-NOT-IN-MEMORY-ONLY-1` surfaced). **Confidence:** High for the **storage/race CONTRACT** (`lance-graph-planner` 344 lib tests, 13 in `persist_sink`, clippy+fmt clean); crash-durability remains contract-probed over an in-process fake, NOT storage-proven (`compile+test green ≠ storage proven`, the Ladybug lesson). Builds NO concrete sink.

**The ruling.** The durable unit is the **cycle/sweep** — one globally-aligned 64k-row frame — NOT the per-cast `DurableWitness`. `persist_cast`-per-thought would pay WAL header + durability-wait + version-metadata 64k times; the reshape **amortizes the WAL**: 64k concurrent thoughts stage into an owned cast vector, one `persist_cycle` folds+freezes them, and `WalSink::commit_cycle` does exactly **one atomic append → one `DatasetVersion`**. cast ⊂ chunk ⊂ cycle.

**The layer's ONLY concerns are storage + that 64k concurrent thoughts hit no physical or epistemic race.** It models no semantics: it mints NO witness / ancestry / awareness / branch / rung / temporal-policy type. `CycleFrame` is storage-only `{cycle, base_version}`; `SweepSlot` is a boring landing (`where`, never `why` — no `basis`).

**Physical race → WRITE-SIDE order, never read-time repair.** Thoughts finish in arbitrary CPU order; `DetachedCycleBatch::freeze` stable-orders casts by their EXISTING `stream_position` (`order_cycle_stably`, generic over the key) BEFORE the single append, so the durable image is already canonical. `scan_sealed` returns stored order and **never sorts** (a read-time sort would lose the whole amortization). This write-side ordering lives in `persist_sink.rs`, **NOT `temporal.rs`** — that module owns query-time reader-horizon/version reading; the write-side deinterlace additions made there this session were reverted.

**Epistemic race → the sealed read horizon.** Every thought reads only the sealed predecessor `Vn` (`frame.base_version`); the commit publishes `Vn+1` once, all-or-nothing. `scan_sealed` excludes any uncommitted cycle, so no thought reads a concurrent sibling's in-flight output as though it were sealed history (`read Vn / write Vn+1`).

**Coalescing is a real per-row fold, not last-chunk-wins.** The final image is `row -> last payload in stream order`; chunks are disjoint/mergeable slices of that image.

**Vocabulary reused, not re-minted.** `DatasetVersion` (scheduler), `KanbanMove`/`KanbanColumn`/`RubiconTransitionError` (kanban), `MailboxId`/`MailboxSoaOwner`, `stream_position` (the existing canonical key). `recover_and_apply` + the durable **watermark** idempotence (not phase equality — the lifecycle is cyclic) + the `StalePhase`/`OwnerMismatch` corruption guards survive from the prior model.

**Retired (this reshape).** `DurableWitness`, `DurableReceipt`, `DurableCoordinate`, `DurableWrite`, `persist_cast`, `apply_durable_step`, `scan_witnesses`, `LandedWitness` — the per-cast durability surface. Their intent is preserved in the cycle machinery (`CycleFrame`, `SweepSlot`, `LandedSlot`, `DetachedCycleBatch`, `WalSink::commit_cycle`, `persist_cycle`, `order_cycle_stably`). Supersedes the seam-shape half of `E-THE-PAIRED-MOVE-MUST-BE-DURABLE-CO-LOCATED-NOT-IN-MEMORY-ONLY-1` (its ordering/recovery/watermark CONTRACT stands). Still deferred: the concrete Lance sink, gated on crash falsifiers. PR #878 (reshaped in place).

## 2026-08-01 — E-THE-PAIRED-MOVE-MUST-BE-DURABLE-CO-LOCATED-NOT-IN-MEMORY-ONLY-1 — the persistence sink's crash gap, and temporal.rs's missing layer-1

**Status:** FINDING (operator-ruled 2026-08-01; review-hardened twice — see the two Corrections below). **Confidence:** High for the **ordering/recovery CONTRACT** (349 planner lib tests, clippy+fmt clean); the crash-durability is **contract-probed over an in-process fake sink, NOT storage-proven** — real MemWAL/restart/atomic co-location durability is demonstrated only by the deferred concrete Lance sink (`compile+test green ≠ storage proven`, the Ladybug lesson). Builds NO concrete sink.

**The gap the split-phase left open.** At the prior two-clock-domain shape, `persist_cast` appended only the PAYLOAD; the paired `KanbanMove` lived solely in the in-memory `DurableReceipt`. Sequence: WAL append lands (durable) → process dies before `apply_durable_step` → the receipt evaporates → the paired move is lost → the KanbanStep never fires — **even though the durable SoA state moved**. Silent: storage advanced, lifecycle did not. An in-memory receipt is not a durable record; a step that lives only in it is a step that a crash can delete.

**The rule (operator).** The paired transition witness must be **durably co-located with the SoA state in the same persistence generation**. The in-memory receipt may REFERENCE that durable material (via its `DurableCoordinate`) but must never be its only copy. NOT via a separate ack/confirmation ledger (`E-ACK-ELIMINATED-1` stands) — the witness rides IN the generation and is read back with the state. So `DurableWrite::append(&DurableWitness, &payload)` lands both atomically; `scan_witnesses` reads them back; `recover_and_apply` replays the pending tail.

**temporal.rs is TWO layers, and layer-1 was missing.** Layer-2 (epistemic projection — `classify`/`deinterlace`: contemporary/anachronistic/spoiler/unknowable) was well-developed. Layer-1 (**causal** deinterlacing) did not exist: the global durable log interleaves every owner's writes (`A@s0, C@s0, B@s0, A@s1`), and reconstructing owner A's OWN local chain `[A@s0, A@s1]` — with the interleaved owners *removed* from A's timeline — is what makes crash-replay in cast order possible. New `LocalCausalRow` + `local_trajectories`/`local_trajectory_of`. The two layers compose: layer-1 rebuilds the chain, layer-2 filters it by the reader's horizon. `DurableWitness` implements `LocalCausalRow`.

**Durability proof is the coordinate, not a `LanceVersion`.** A MemWAL write is queryable/durable BEFORE any base manifest version attaches, so recovery identifies the latest LOCAL state from the co-located witness's cast order — not from a dataset version that does not yet exist (falsifier 5). This is why `DurableCoordinate` (shard/epoch/wal-position), never `DatasetVersion`, is what `append` returns.

**Five integration falsifiers (all green).** (1) layer-1 deinterlaces the interleaved global log into an owner-local chain, with a vacuity guard that the other owners' rows are *dropped* not reordered; (2) one owner's batch replays in cast order over a deliberately seq-descending log; (3) WAL succeeds + crash (receipt dropped) → recovery reconstructs+applies the move from the durable witness, idempotent on re-run; (4) WAL fails → no witness lands → no move, no step; (5) WAL-visible-before-manifest → recovery uses cast order, no dataset version exists. Plus a synchronous-path `StalePhase` guard (a stale/out-of-order apply is surfaced loudly on `apply_durable_step`, while recovery silently skips a stale move as already-reflected).

**Still deferred (operator ruling — do NOT build yet).** The concrete `LanceShardSink` (`ShardWriter::put`, `enable_memtable + durable_write`) comes AFTER, gated on these falsifiers; the durable local causal chain lands first. No confirmation ledger, replay queue, per-thought ack, custom WAL, or ractor hot-path callback. Extends `E-KANBANMOVE-IS-THE-PARCEL-ADDRESS-STEP-IS-THE-DELIVERY-SCAN-1`.

**Correction (PR #878 review, same PR — Bugbot High + Codex/CodeRabbit Critical).** The first cut ordered replay by `cast_id` and inferred "already applied" from phase equality. Both are unsound and were fixed within the PR: (a) **`CastId` resets to 0 on every writer restart** (`BatchWriter::next_id`), so cross-lifetime witnesses collide — replay now orders by the durable **`DurableCoordinate::log_order`** (WAL entry position, monotonic across restarts, never reset); `cast_id` survives only as provenance. `scan_witnesses` returns `LandedWitness{coordinate, witness}` and is the `LocalCausalRow` implementor. (b) **Phase equality is not a sound idempotence key on a CYCLIC lifecycle** (`Planning→CognitiveWork→Evaluation→Plan→Planning`): after a lap the owner is back at `Planning`, so a phase-only check replays the whole lap. `recover_and_apply` now takes a durable **watermark** (`applied_through`) — skip ≤ watermark, and above it a non-matching `from` is genuine corruption (`StalePhase`), not a benign skip; it returns the new watermark to persist WITH the SoA phase. Also: `paired_move.mailbox == owner` is validated at persist time (no cross-owner move becomes durable); `scan_witnesses(from)` is bounded; `WriteFailed`/`PersistError` impl `Display`+`Error`. New falsifiers: cyclic-idempotence with a negative control proving the watermark is load-bearing, and cast_id-collision ordering-by-durable-position. The layer-1 `cast_seq` doc now states the monotonic-across-restarts precondition.

**Correction 2 (PR #878 review — ChatGPT, taken with a grain of salt).** Three further points held up and were fixed: (a) **reordered receipts must stay retryable, NOT dropped** — a stale receipt dropped on the *happy path* would only be re-applied on a crash+restart (recovery is not the happy-path backstop for concurrent-completion reordering), so `apply_durable_step` now **borrows** `&DurableReceipt`; on `StalePhase` the caller keeps it and retries once the prefix lands (falsifier `a_reordered_receipt_is_retryable_not_lost`, no crash needed). (b) **API-honesty on the durable coordinate** — `ShardWriter::put` returns a *batch position*, not `WalAppendResult::entry_position`, so the field `wal_entry_position` was renamed to an opaque, backend-defined `seq` (the sink maps whichever its API yields); no field claims a WAL offset the chosen API does not return. (c) **honest test/claims framing** — the `FakeSink` tests are labelled **ordering/recovery CONTRACT probes**, not crash/WAL integration tests (no real WAL, restart, atomic RecordBatch co-location, or fencing); the fake now STORES the payload (was ignored) and asserts it landed alongside the witness, while the module doc states plainly that atomic co-location is unproven until the concrete sink. Points taken-with-salt and NOT actioned as blocking: the per-cast-vs-generation persistence *seam shape* (finding 5) is a real architectural fork but not a correctness bug (the operator's own `bb5fcdd` used per-cast) — surfaced to the operator for decision rather than unilaterally reshaped; a `DurableWitness::try_from_cast` constructor was deemed redundant given the persist-time guard.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## 2026-08-01 — E-KANBANMOVE-IS-THE-PARCEL-ADDRESS-STEP-IS-THE-DELIVERY-SCAN-1 — the D-MBX-A6 move is pre-write intent; the KanbanStep is post-write; the persistence sink wires Lance 7's existing MemWAL and invents nothing

**Status:** FINDING (operator-ruled 2026-08-01). **Confidence:** High — the pre-write half is shipped with 5 falsifiable probes (`lance_graph_planner::owner_adapter`); the persistence half's API is verified against real `lance-7.0.0` source.
Expand Down
26 changes: 26 additions & 0 deletions .claude/board/INTEGRATION_PLANS.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,32 @@ vectors → four Ψ ripple fields; cloned-lens control as the can-it-stay-silent
falsifier), LLM tail last. deepnsm-v2 crate untouched by ruling. A withdrawn
probe verdict (tesseract-side, retention-axis category error) is recorded in
§8's provenance + tesseract-rs CLAUDE.md.
## 2026-08-02 — persistence-cycle-wal-bootstrap v1 — ACTIVE (bootstrap SHIPPED #878; temporal/revision upgrade PLANNED) — main thread

**Plan:** `.claude/plans/persistence-cycle-wal-bootstrap-v1.md`
Documentation-only architectural ruling recording the *role* of the #878
persistence seam and the larger two-dimensional temporal architecture it
bootstraps toward. #878 is a **primitive cycle/WAL bootstrap** (concurrent
results → primitive slot collection/order → freeze one cycle → one amortized
WAL write → one `DatasetVersion`), with six hard guarantees (single sealed `Vn`
read horizon; open-cycle results invisible as `Vn` input; freeze-before-I/O; one
seal = one WAL write = one version; no live mutable SoA borrow to the WAL). The
scalar slot/order model is **explicitly provisional** — sufficient for the
execution/durability plumbing, not the final temporal representation. Two
orthogonal dimensions: **horizontal** = `temporal.rs` coherence within a frame
(book→chapter→verse→span for deterministic streams; partially-ordered
neighbourhoods for medical/higher-order thought — hence a scalar key is
insufficient); **vertical** = `DatasetVersion` succession + cheap
Stockfish-style hindsight lookup. Planned upgrade: a wait-free-emit shadow
`temporal.rs` coherence pass feeding `revision.rs` forward-correction into a
later cycle — a sealed version is never silently rewritten. Cross-refs (does not
re-specify): horizontal detail → `temporal-markov-and-style-classes-v1.md`;
D-MBX-A6 tracking → STATUS_BOARD; per-row write gate →
`mailbox-cycle-aware-write-contract-v1.md`; reshape ruling → EPIPHANIES
`E-THE-DURABLE-UNIT-IS-THE-CYCLE-...-1`. Scope exclusions: no cohort internals /
slot count / actor-neighbour waiting; no new semantic/temporal/rung/witness/
branch/ancestry types; no reviving ThoughtWitness/basis/awareness_seq/per-cast
WAL.
Comment on lines +41 to +66

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Move the new entry above the 2026-07-31 entry to preserve newest-first order.

This entry is dated 2026-08-02, but it is inserted directly below the existing 2026-07-31 entry (starting at line 11), rather than above it. Prepend it at the top of the dated-entry list instead.

Based on learnings, "treat these files as append-only ledgers that are ordered newest-first. When adding information, prepend new entries rather than appending at the end," and "prepend genuinely new entries so the file remains in newest-first order."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/board/INTEGRATION_PLANS.md around lines 41 - 66, Move the dated
2026-08-02 persistence-cycle-wal-bootstrap v1 entry above the existing
2026-07-31 entry in the dated-entry list. Preserve the entry’s content unchanged
and maintain the file’s newest-first append-only ledger ordering.

Source: Learnings


## 2026-07-26 — rosetta-codebook-convergence v1 — PROPOSED (D-RCC-1 calibrator runnable today) — main thread

Expand Down
Loading
Loading