soa_verbatim: pin the physical layout, correct the compression claim, prove it over S3

.claude/agents/lance-cache-cartographer.md

@@ -0,0 +1,61 @@
+# lance-cache-cartographer — who can size the cache from outside?
+
+**Tier: Opus (filigran).** Plumbing traces are accumulation: the answer is a
+CHAIN (public API → builder → session → backend), and any single hop read in
+isolation gives a false verdict in either direction.
+
+## RULE
+
+**A knob exists only if a consumer can turn it without forking.** Finding
+`LanceCache::with_capacity` in lance-core proves nothing about reachability;
+the verdict is the exact public call chain from `lancedb::connect(...)` or
+`lance::dataset::DatasetBuilder` (or a documented env var) down to the
+`MokaCacheBackend` constructor — or a definitive UNREACHABLE with the point
+where the chain breaks, cited as `file:line`.
+
+## Incident grounding
+
+The same day this card was written, the workspace hit the sibling trap on the
+S3 provider: the capability existed (`lancedb/aws`), was off by default, and
+its absence produced an error that pointed at credentials instead of at the
+feature. A capability that exists but is not wired to a public surface
+produces exactly this class of lost afternoon. Map the wiring, not the
+capability.
+
+Fork pressure is the second reason this card exists: lance is
+UPSTREAM-AUTHORITATIVE (`E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1`) — if the knob
+is unreachable, the output must say "small upstream ask" with the exact
+missing hop, never "patch it in our tree".
+
+## Mandatory reads before output
+
+1. `.claude/knowledge/lance-cache-surface.md` (P-CACHE-2 + the decision table)
+2. The sweep inventories handed to you (capacity plumbing, lancedb surface,
+   env vars)
+
+## Method
+
+1. Start from the BACKEND and walk outward: who constructs
+   `MokaCacheBackend::with_capacity` / `no_cache` / `LanceCache::with_capacity`?
+   Who owns that object (Session? Dataset? Connection?)?
+2. Then from the PUBLIC surface inward: `lancedb::connect` builder methods;
+   `DatasetBuilder` options (`index_cache_size`, `metadata_cache_size`,
+   session injection, `ReadParams`); any `LANCE_*`/env lookup.
+3. The verdict is per-knob: capacity-in-bytes, entry-count variants,
+   `no_cache`, custom `CacheBackend` injection — each REACHABLE (with the
+   chain) or UNREACHABLE (with the break point).
+4. Note defaults with file:line: what capacity does a consumer get who sets
+   nothing?
+
+## Output shape
+
+Per knob: chain or break point, each hop `file:line`. Then the defaults
+table. Then one paragraph: what the lance-graph consumer should call today,
+and what (if anything) is the minimal upstream ask.
+
+## Hard rules
+
+- Registry sources are read-only; cite exact paths.
+- Do not run cargo. Do not write any file. Return findings as output; the
+  orchestrating main thread is the sole writer of board files.
+- Read `.claude/board/AGENT_LOG.md` before starting. Do NOT write it.

.claude/agents/lance-cache-contents-auditor.md

@@ -0,0 +1,55 @@
+# lance-cache-contents-auditor — what does `LanceCache` actually hold?
+
+**Tier: Opus (filigran).** This is accumulation: the verdict only makes sense
+after holding every insert site in mind at once, and a single missed site
+inverts the conclusion.
+
+## RULE
+
+**Classify the cache by its INSERTS, never by its name or docs.** A cache
+called `metadata_cache` may hold decoded data; a doc saying "file metadata"
+may be stale. The only admissible evidence is an enumerated insert site with
+the inserted TYPE and that type's size class, cited as `file:line`.
+
+## Incident grounding (why this card exists)
+
+On 2026-08-07 this workspace nearly reasoned itself into "cap the moka cache
+and the RAM bill is bounded" on the strength of `with_capacity` existing. If
+the data path never enters the cache, that lever is inert and the RAM lives in
+the consumer's own collect/concat habits — a failure mode already caught once
+on this branch (commit `a27b06a`: a whole-table `concat_batches` inside a
+module whose premise was zero-copy). The lever's existence says nothing about
+what it levers.
+
+## Mandatory reads before output
+
+1. `.claude/knowledge/lance-cache-surface.md` (the evidence table + P-CACHE-1)
+2. The sweep inventory handed to you (every `CacheKey` implementor + insert site)
+
+## Method
+
+1. For every `CacheKey`/`UnsizedCacheKey` implementor: what is `ValueType`?
+   Where is it inserted? What is its size class — O(bytes-of-dataset) or
+   O(metadata)?
+2. Follow the DATA read path specifically: `FileReader` → decoded pages →
+   `RecordBatch`. Does ANY step insert into a `LanceCache`? Name the function
+   that would have done it and show it absent, not just "no grep hit".
+3. Distinguish the three caches if they exist separately (session/index/
+   metadata) — a claim about "the cache" that conflates them is unusable.
+4. Verdict vocabulary: **DATA-CACHED** / **METADATA-ONLY** /
+   **MIXED (list which)** — each row with file:line.
+
+## Output shape
+
+A table (implementor, ValueType, insert site file:line, size class), then the
+verdict, then the single strongest piece of contrary evidence you found and
+why it does not change the verdict. If you cannot rule a path in or out, say
+UNRESOLVED for that path — an honest gap beats a smooth story.
+
+## Hard rules
+
+- Registry sources are read-only; cite exact paths under
+  `~/.cargo/registry/src/index.crates.io-*/lance*-9.0.0/`.
+- Do not run cargo. Do not write any file. Return your findings as output;
+  the orchestrating main thread is the sole writer of board files.
+- Read `.claude/board/AGENT_LOG.md` before starting. Do NOT write it.

.claude/board/EPIPHANIES.md

@@ -1,3 +1,63 @@
+## 2026-08-07
+
+### E-COMPRESSION-META-INERT-AT-512-STRIDE-1
+
+`soa_to_lance.rs`'s `lance-encoding:compression = "none"` field metadata was
+documented as load-bearing for the verbatim-write deployment pattern. **Measured
+false**: removing the key, or setting it to `"zstd"`, leaves the file
+byte-identical. The key is spelled correctly and IS parsed
+(`lance-encoding-9.0.0` `compression.rs:576`) — it simply never reaches a
+512-byte column.
+
+Root cause, read from lance 9 source: `is_narrow`
+(`encodings/logical/primitive.rs:3861`) calls a value narrow below
+`MINIBLOCK_MAX_BYTE_LENGTH_PER_VALUE = 256`. `NODE_ROW_STRIDE = 512` is not
+narrow, so the column takes **full-zip**, whose `create_per_value` returns
+`ValueEncoder::default()` unconditionally for `FixedWidth` data
+(`compression.rs:753`) — the merged field params are computed one line earlier
+and then ignored. Only the **mini-block** path
+(`build_fixed_width_compressor`, `compression.rs:624`) honours the metadata,
+and a 512-byte value never reaches it.
+
+So the canonical stride, not the metadata, is what buys the verbatim mmap
+premise. `crates/lance-graph/tests/soa_verbatim.rs::the_narrow_column_falsifier`
+proves the byte search can actually detect compression (at a 64-byte control
+stride, `"none"` keeps rows verbatim and `"zstd"` makes them vanish — both
+measured on the same shape). Both docs (`soa_to_lance.rs`, `soa_verbatim.rs`)
+corrected in place, crediting the stride and demoting the metadata line to a
+documented backstop.
+
+`crates/lance-graph/tests/soa_verbatim.rs` also gained
+`a_slab_is_written_verbatim_to_s3_too` — the same physical-layout assertion run
+against the real S3-compatible object store this session has credentials for
+(`AWS_S3_BUCKET_NAME` + the standard `AWS_*` vars — the same variable names
+Railway deployments already set, no new key invented), proving the local
+finding also holds through the object-store write/read path, not only on a
+local filesystem.
+
+### E-HYDRATION-IS-FIXED-COST-NOT-SIZE-COST-1 — the idle-flush plan's §4 blocker closes, and the argument it used against size-weighted eviction is refuted by the same probe
+
+**Status:** FINDING (measured, `crates/lance-graph/examples/hydration_probe.rs`, lance 9.0.0, one endpoint, one day, 0.3–33.5 MB single-fragment datasets). **Confidence:** High for the §4 gate (flat in size, 25–30×, and the fallback path is independently cheap); High for the fixed/variable decomposition within the probed range; **Low for the absolute constants** — endpoint-, region- and day-specific, explicitly not re-run elsewhere.
+
+> **Correction (PR #907 review):** the phrase "the version read needs no open"
+> below overstates what was measured. `latest_version_id()` was timed on
+> `warm`, a `Dataset` handle already produced by one `Dataset::open` — the
+> probe measures the AMORTIZED per-candidate cost of a version check once a
+> handle is held, not a version read with no dataset-open lifecycle anywhere.
+> The gate conclusion is corrected to that lifecycle, below.
+
+**The blocker that closed.** `.claude/plans/idle-flush-dataset-eviction-v1.md` §9.1 named its own first task: *"cheap local version read — assumed, unchecked … if it fails, the plan needs a different dirty-detector and this document is wrong rather than incomplete."* Measured: on an already-open `Dataset` handle, `Dataset::latest_version_id()` resolves a manifest **location** (no manifest read, no data read) in **8–11 µs**, versus **0.23–0.29 ms** for a full `Dataset::open`. Both are **flat in dataset size** across a 100× span. The gate offered two ways to pass and **both hold** — one open amortized across many cheap version reads is far cheaper than re-opening per candidate, *and* even re-opening per candidate is cheap enough (1,000 candidates in 0.27 s) that the fallback would have sufficed anyway.
+
+**The correction nobody asked for, from the same run.** Hydration decomposes as **≈ 2.63 s fixed + ≈ 0.021 s/MB** (0.3 MB → 2.64 s; 33.5 MB → 3.33 s — a 100× size increase costs **1.26×** the time). So §2's deferred size-weighted ranking was declined for a reason that does not survive measurement: the plan argued *"rehydration cost is also proportional to size, so a size-weighted key preferentially evicts what is most expensive to get back."* In the probed range rehydration cost is **dominated by a size-independent constant**. Evicting the large dataset frees ~100× the bytes for ~1.26× the restore cost — the opposite of the stated objection.
+
+**What that does NOT license.** It removes one argument, not the decision. Age-ordering may still be the right default; the *reason recorded for it* is now known to be wrong, which is a different and smaller claim. The plan's grading was updated in both directions rather than promoted: §0's row now reads "CONJECTURE, and its stated ARGUMENT is refuted".
+
+**The transferable half — a cost model that prices the wrong term is worse than one that admits it is incomplete.** §1 of the plan was already honest that its economics omitted request count, and reviewers on PR #901 had sharpened it. But the omission that mattered was not a missing *term*; it was an assumed *shape*. Both sides of the plan's ledger were reasoned about as if proportional to bytes, so the whole argument silently tracked the wrong variable. With a fixed per-hydration cost, the quantity that decides the policy is **how many** hydrations it causes and never how large they are — which is exactly what §7's thrash criterion already counts. The plan gated on the right metric while justifying it with the wrong model, and only a measurement could tell those apart.
+
+**Also settled, cheaply:** T10 (flush → rehydrate → read equality) is **green** at all three sizes by full-scan `id` checksum — a row-count check would have passed a truncated hydration. And §1's unmeasured request count is **3 remote objects per dataset** here (`.txn` + `.manifest` + one data file), which bounds the small case; it grows with fragment count, which single-fragment writes do not exercise.
+
+**Scope, so the numbers are not over-read:** nothing was evicted, no policy was implemented, and none of the plan's other acceptance criteria ran. Companion tool: `lab/s3rm.py` — written because `s3put.py` could only write, so the probe had no way to remove its own scratch and would have left debris in a curated prefix.
+
 ## 2026-08-06 — E-D-IGN-B-CORPUS-PRODUCED-NOTHING-TO-READ-1 — arming a CI gate that ran zero tests exposed an empty-vs-empty digest collision; the corpus was the defect, the untagged digest was correct
 
 **Status:** FINDING (reproduced at `f9206fc`, fixed, both mutation directions verified). **Confidence:** High for the mechanism (the empty-hash constant was computed and matched; the pre-fix corpus was traced token-by-token through the clause machine and emits nothing); High for the fix (all four lens arms now measured non-empty, 4/4 distinct digests on one owner). Test-fixture only — `stance.rs` is untouched.

.claude/knowledge/lance-cache-surface.md

@@ -0,0 +1,89 @@
+# The lance 9 cache surface — what moka holds, who can size it, and what that means for RAM-billed deployments
+
+> **READ BY:** `lance-cache-cartographer`, `lance-cache-contents-auditor`,
+> `integration-lead`, `truth-architect`, and any session that reasons about
+> RAM footprint of a lance-graph consumer, proposes a disk cache backend,
+> touches `.claude/plans/idle-flush-dataset-eviction-v1.md`, or debugs a
+> `"No object store provider found for scheme"` error.
+>
+> **Companions:** `.claude/knowledge/s3-hydration-lifecycle.md` (the
+> three-layer model this doc's question sits under) ·
+> `.claude/plans/idle-flush-dataset-eviction-v1.md` §8a (the measured
+> hydration cost) · `zero-copy-lens-law.md` (why a materializing read path is
+> a violation, not a convenience).
+
+## The operator's question, stated before any answer
+
+> *"Der Gedanke warum ich S3 sink-in to Harddisk möchte ist, daß lance nur
+> die aktiven Bestandteile in den Speicher zieht und die Railway-Rechnung für
+> RAM usage kleiner wird."* — and, one turn later: *"man müsste dann halt nur
+> radikal moka flushen."*
+
+Two candidate mechanisms for a small RAM bill over a large dataset:
+
+- **A. Disk sink-in:** hydrate S3 → local disk, rely on demand paging, evict
+  after idle (the idle-flush plan).
+- **B. Capped moka:** read S3 (or disk) directly and bound the in-memory
+  cache — `radikal flushen` as configuration.
+
+Which one is real depends on two facts about lance 9 that this doc exists to
+pin: **what the cache actually holds** and **who can size it from outside**.
+
+## Evidence status (workspace rule: label everything)
+
+| claim | status | evidence |
+|---|---|---|
+| `moka` is an UNCONDITIONAL dependency of `lance-core` — no feature gate, present in local-only builds | **FINDING** (source-read 2026-08-07, lance-core 9.0.0) | `Cargo.toml` `[dependencies.moka] version = "0.12"` with no `optional`; `cache/mod.rs:51` `mod moka;` ungated |
+| The `aws` feature gates only the object-store PROVIDER, not any cache | **FINDING** (measured) | with every `AWS_*` var set correctly, `connect("s3://…")` without `lancedb/aws` fails `No object store provider found for scheme: 's3'` — the error names the scheme, not a credential. With the feature: first-try success |
+| `MokaCacheBackend::with_capacity(bytes)` is byte-weighed, not entry-counted | **FINDING** (source-read) | `moka.rs`: `.max_capacity(capacity)` + `.weigher(\|key, entry\| key_footprint(key) + entry.size_bytes)` |
+| `MokaCacheBackend::no_cache()` exists (`Cache::new(0)`) — "radikal flushen" as a constructor | **FINDING** (source-read) | `moka.rs` |
+| `CacheBackend` is a pluggable trait; the docs name "persistent backends" as intended; **only moka ships** | **FINDING** (source-read) | `cache/{mod,backend,codec}.rs`; `codec.rs` describes "scanning a persistent store at startup"; the only implementor in-tree is `MokaCacheBackend` |
+| lance mmaps nothing — `memmap` absent from lance / lance-io / lance-file / lance-core; local reads are seek+read into heap `Bytes` | **FINDING** (source-read) | zero grep hits across the four manifests and sources; `object_store` `local.rs` seeks and reads |
+| Lance's read path gives NO 64-byte alignment guarantee (varies with allocator state) | **FINDING** (measured, this branch) | commit `a27b06a` message + `osm-soa-bake` `slab.rs` — the same read passed alone and failed in-suite |
+| `FixedSizeBinaryArray::try_from_iter` copies chunk-by-chunk; `Buffer::from_vec` adopts the allocation | **FINDING** (source-read + pointer-identity test before removal) | arrow-array 58 `fixed_size_binary_array.rs:553` (`MutableBuffer`), arrow-buffer `immutable.rs:141` |
+| **What `LanceCache` actually holds on the DATA path** — decoded pages / batches vs only manifests, schemas, index metadata | **OPEN — P-CACHE-1** | nothing read yet; this decides whether a capacity cap bounds the RAM that matters |
+| **Whether cache capacity / `no_cache` is reachable from `lancedb::connect` or `DatasetBuilder`** (public API, env, session object) | **OPEN — P-CACHE-2** | `LanceCache::with_capacity` found only in lance-core; plumbing untraced |
+| **Empirical RAM shape**: RSS across scans at different capacities, re-fetch behaviour over S3 on a second scan | **OPEN — P-CACHE-3** | needs the probe; timing + `/proc/self/status` VmRSS, honest about network variance |
+
+**Nothing below is promoted past its row above.**
+
+## Why P-CACHE-1 is the load-bearing question
+
+If the data path (decoded column pages, record batches) does NOT go through
+`LanceCache`, then capping it bounds only metadata — and the RAM that shows up
+on a Railway bill lives in whatever the *caller* holds (the failure mode
+already caught once on this branch: a `read_batch` that collected the whole
+table and concat-copied it, commit `a27b06a`). In that world, mechanism B is
+an illusion and the RAM answer is *streaming discipline in the consumer*, not
+cache configuration.
+
+If the data path DOES go through the cache, `with_capacity(n)` is a hard
+byte ceiling on exactly the memory the operator is billed for, and mechanism
+B beats mechanism A on every axis except S3 request count.
+
+## Decision table this doc must end up supporting
+
+| finding | consequence |
+|---|---|
+| data cached + capacity reachable | **B wins for RAM**; idle-flush plan remains a *request-cost* optimisation only |
+| data cached + capacity NOT reachable | small upstream ask (expose the knob), NOT a fork — lance is upstream-authoritative (`E-LANCE-IS-UPSTREAM-AUTHORITATIVE-1`) |
+| data not cached | RAM bill is the consumer's streaming discipline; B is inert for data; re-reads hit S3 every time → A (disk sink-in) regains its case as *request* mitigation |
+
+## Probe queue
+
+| probe | question | pass/fail shape | status |
+|---|---|---|---|
+| P-CACHE-1 | do decoded data bytes enter `LanceCache`? | enumerate every `CacheKey` implementor + every insert site; classify metadata vs data with file:line | NOT RUN |
+| P-CACHE-2 | is capacity / `no_cache` settable via lancedb / DatasetBuilder / env / Session? | the exact public call chain, or a definitive "unreachable" | NOT RUN |
+| P-CACHE-3 | does RSS track capacity? does a second S3 scan re-fetch? | probe binary, RSS + wall time, both capacities; network variance stated | NOT RUN |
+
+## Traps already paid for (do not re-pay)
+
+1. **The scheme error is not a credential error.** Provider absent ⇒ fail at
+   registration; every credential correct and irrelevant.
+2. **"Usually aligned" passes CI.** Never assert an alignment VALUE; assert
+   the invariant (`rows().is_some() == (ptr % 64 == 0)`).
+3. **A pointer-identity test proves nothing if its reference is the copy.**
+   The removed module's test compared the slab against the *concatenated*
+   buffer — the copy it should have ruled out was one call earlier.
+4. **`try_from_iter` is a copy wearing a constructor's name.**