From fb617bd9a6126bfc709c9ed9aec4a87b3f343e7c Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Fri, 7 Aug 2026 15:58:14 +0200 Subject: [PATCH 1/2] graph: calcify an imported ontology into LanceDB as 64k tables An import that stops at an in-memory artifact leaves every consumer to slice the same blob its own way. This module ends the import on disk: baked node rows go into Lance datasets, and from then on the filled database is what gets addressed. Sibling of `graph::hydrate`, following its source -> RecordBatch -> `Dataset::write` shape. The seam is bytes, not a type. `&[u8]` with a 512-byte stride, so lance-graph takes no dependency on whichever harvest produced the rows and the seam cannot drift from the node layout, because it is the node layout. An OBO harvest, a relational transcode, and a hand-built fixture all arrive the same way. 64k tables are not a chosen chunk size. `identity` is a u16, so a (classid, family) pair addresses exactly 65_536 slots -- one table is a closed address space, 64 Ki x 512 B = 32 MiB at full extent. `partition` is a positional scan that cuts where the prefix changes; it never sorts, groups, or moves a row, and it refuses unsorted input rather than merging it (the alternative is one table silently written as two datasets). One column, not three. `node: FixedSizeBinary(512)` over the caller's own allocation via `Buffer::from_custom_allocation`, with the caller's `Arc` keeping it alive. Splitting into key/edges/value columns would read better and would cost a strided gather over every row -- the copy this module exists to avoid. That carving is instead a read-side projection over byte positions (KEY_RANGE / EDGES_RANGE / VALUE_RANGE). `write_database` is the import's last step; `NodeTable::open` is the addressing that follows. It holds the dataset, not a decoded copy, which is why `count_rows` is async and fallible -- a row count is a read against the data, never a field this struct would have to keep in step with it. Verification: `cargo test -p lance-graph --lib graph::ontology_hydrate` -- 7 passed, 0 failed; `cargo clippy -p lance-graph --lib` reports nothing in this module. The zero-copy claim rests on `the_batch_borrows_the_callers_allocation`, which asserts pointer identity between the bytes Arrow exposes and the buffer handed in, so it is measured rather than asserted in prose. --- crates/lance-graph/src/graph/mod.rs | 1 + .../lance-graph/src/graph/ontology_hydrate.rs | 515 ++++++++++++++++++ 2 files changed, 516 insertions(+) create mode 100644 crates/lance-graph/src/graph/ontology_hydrate.rs diff --git a/crates/lance-graph/src/graph/mod.rs b/crates/lance-graph/src/graph/mod.rs index eded4abb0..0144dad0b 100644 --- a/crates/lance-graph/src/graph/mod.rs +++ b/crates/lance-graph/src/graph/mod.rs @@ -17,6 +17,7 @@ pub mod mailbox_scan; pub mod metadata; pub mod neighborhood; pub mod neuron; +pub mod ontology_hydrate; pub mod scheduler; pub mod semiring_map; pub mod sparse; diff --git a/crates/lance-graph/src/graph/ontology_hydrate.rs b/crates/lance-graph/src/graph/ontology_hydrate.rs new file mode 100644 index 000000000..da5671e4c --- /dev/null +++ b/crates/lance-graph/src/graph/ontology_hydrate.rs @@ -0,0 +1,515 @@ +//! Calcify an imported ontology into LanceDB — the end of the import. +//! +//! # Why this exists +//! +//! An import that stops at an in-memory artifact leaves every consumer to slice +//! the same blob its own way. The import ends **on disk**: it writes the baked +//! node rows into LanceDB datasets, and from then on the filled database is the +//! thing that gets addressed. Nobody re-reads the source document, and nobody is +//! handed a `Vec` to cut up. +//! +//! # The seam is bytes, not a type +//! +//! This module takes `&[u8]` with a 512-byte stride, never a producer's struct. +//! The 512-byte node layout IS the contract (`key(16) | edges(16) | value(480)`), +//! so a byte seam is the honest one: it costs no dependency on whichever harvest +//! produced the rows, and it cannot drift from the layout, because it *is* the +//! layout. OBO, a relational transcode, and a hand-built fixture all arrive the +//! same way. +//! +//! # 64k tables +//! +//! Rows are partitioned into one dataset per `(classid, family)`. That is not an +//! arbitrary chunk size: in the V3 tail `identity` is a `u16`, so a +//! `(classid, family)` pair addresses **exactly 65 536 slots** — one table's +//! worth. `family` is the next tier up, `classid` the routing prefix in front of +//! it. A table is therefore a closed address space, not a page boundary someone +//! chose, and 64 Ki × 512 B = 32 MiB is its full extent. +//! +//! Because the bake sorts by `(classid, family, identity)`, those partitions are +//! already contiguous runs in the input. This module finds the runs by reading +//! key positions; it never sorts, groups, or moves a row. +//! +//! # Zero-copy +//! +//! One column, `node: FixedSizeBinary(512)`, over the caller's own allocation via +//! [`Buffer::from_custom_allocation`]. No gather, no re-pack, no intermediate +//! `Vec`: the bytes Lance writes are the bytes the bake produced, and the +//! caller's buffer is kept alive by the `Arc` it hands in. +//! +//! **This is why the row is one column and not three.** Splitting into +//! `key`/`edges`/`value` columns would read better and would cost a strided +//! gather over every row — the copy this module exists to avoid. The three-part +//! carving is a **projection over positions** on the read side +//! ([`KEY_RANGE`] / [`EDGES_RANGE`] / [`VALUE_RANGE`]), which is where it belongs: +//! the layout is addressed, not restructured. +//! +//! # What this module does not do +//! +//! It does not interpret the value slab, resolve a label, or know what an +//! ontology is. It moves addressed rows to disk and hands back the addresses. A +//! reader that wants meaning resolves the classid; that is a different concern +//! and a different crate. + +use std::sync::Arc; + +use arrow::array::{ArrayRef, FixedSizeBinaryArray}; +use arrow::buffer::Buffer; +use arrow::datatypes::{DataType, Field, Schema}; +use arrow::record_batch::RecordBatch; + +/// Bytes per node row — `key(16) | edges(16) | value(480)`. +pub const NODE_ROW_STRIDE: usize = 512; + +/// Byte range of the canonical key within a row. Read, never restructured. +pub const KEY_RANGE: std::ops::Range = 0..16; +/// Byte range of the edge block within a row. +pub const EDGES_RANGE: std::ops::Range = 16..32; +/// Byte range of the value slab within a row. +pub const VALUE_RANGE: std::ops::Range = 32..512; + +/// `classid` position inside the key: `[0, 4)`, little-endian `u32`. +const CLASSID_AT: usize = 0; +/// `family` position inside the V3 tail: `[12, 14)`, little-endian `u16`. +const FAMILY_AT: usize = 12; +/// `identity` position inside the V3 tail: `[14, 16)`, little-endian `u16`. +const IDENTITY_AT: usize = 14; + +/// Rows one `(classid, family)` table can address — `identity` is a `u16`. +pub const ROWS_PER_TABLE: usize = 1 << 16; + +/// The address of one 64k table: everything in front of `identity`. +/// +/// Ordered exactly as the baked rows are, so a sorted row buffer yields these in +/// ascending order and each appears in exactly one run. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct TableAddr { + /// Routing prefix. + pub classid: u32, + /// Basin within the class. + pub family: u16, +} + +impl TableAddr { + /// The dataset directory name for this table, inside the database directory. + /// + /// Fixed width and zero-padded so a lexical listing of the database is also + /// an ordering by address. + #[must_use] + pub fn dataset_name(&self) -> String { + format!("nodes-{:08x}-{:04x}.lance", self.classid, self.family) + } +} + +/// One contiguous run of rows sharing a [`TableAddr`] — one 64k table. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct TableRun { + /// The table this run fills. + pub addr: TableAddr, + /// First row index of the run, inclusive. + pub start: usize, + /// One past the last row index. + pub end: usize, +} + +impl TableRun { + /// Number of rows in the run. + #[must_use] + pub const fn len(&self) -> usize { + self.end - self.start + } + + /// Whether the run is empty (it never is, as produced by [`partition`]). + #[must_use] + pub const fn is_empty(&self) -> bool { + self.start == self.end + } + + /// Byte range of this run in the row buffer — contiguous, so slicing it + /// stays zero-copy. + #[must_use] + pub const fn byte_range(&self) -> std::ops::Range { + self.start * NODE_ROW_STRIDE..self.end * NODE_ROW_STRIDE + } +} + +/// Read a little-endian `u32` at `at` within row `i`. +fn u32_at(bytes: &[u8], i: usize, at: usize) -> u32 { + let b = i * NODE_ROW_STRIDE + at; + u32::from_le_bytes([bytes[b], bytes[b + 1], bytes[b + 2], bytes[b + 3]]) +} + +/// Read a little-endian `u16` at `at` within row `i`. +fn u16_at(bytes: &[u8], i: usize, at: usize) -> u16 { + let b = i * NODE_ROW_STRIDE + at; + u16::from_le_bytes([bytes[b], bytes[b + 1]]) +} + +/// The table address of row `i`, read straight off its key. +#[must_use] +pub fn table_addr_of(bytes: &[u8], i: usize) -> TableAddr { + TableAddr { + classid: u32_at(bytes, i, CLASSID_AT), + family: u16_at(bytes, i, FAMILY_AT), + } +} + +/// The `identity` of row `i` — its slot within its 64k table. +#[must_use] +pub fn identity_of(bytes: &[u8], i: usize) -> u16 { + u16_at(bytes, i, IDENTITY_AT) +} + +/// Split a sorted row buffer into its 64k tables. +/// +/// A positional scan: it compares adjacent keys and cuts where the +/// `(classid, family)` prefix changes. Rows are neither moved nor copied — each +/// run is a `start..end` index pair into the caller's buffer. +/// +/// # Errors +/// +/// - the buffer length is not a multiple of [`NODE_ROW_STRIDE`]; +/// - the rows are not sorted by `(classid, family)`, which would make a table +/// appear in two runs and silently split it across two datasets. Refused +/// rather than accommodated: the bake emits sorted output, so an unsorted +/// buffer means something upstream is wrong and merging it here would hide +/// that. +pub fn partition(bytes: &[u8]) -> Result, String> { + if !bytes.len().is_multiple_of(NODE_ROW_STRIDE) { + return Err(format!( + "row buffer of {} bytes is not a multiple of the {NODE_ROW_STRIDE}-byte stride", + bytes.len() + )); + } + let n = bytes.len() / NODE_ROW_STRIDE; + if n == 0 { + return Ok(Vec::new()); + } + + let mut runs = Vec::new(); + let mut start = 0usize; + let mut current = table_addr_of(bytes, 0); + for i in 1..n { + let addr = table_addr_of(bytes, i); + if addr == current { + continue; + } + if addr < current { + return Err(format!( + "rows are not sorted by (classid, family): row {i} is {addr:?} \ + after {current:?}" + )); + } + runs.push(TableRun { + addr: current, + start, + end: i, + }); + start = i; + current = addr; + } + runs.push(TableRun { + addr: current, + start, + end: n, + }); + + for r in &runs { + if r.len() > ROWS_PER_TABLE { + return Err(format!( + "table {:?} holds {} rows, more than the {ROWS_PER_TABLE} an u16 \ + identity can address", + r.addr, + r.len() + )); + } + } + Ok(runs) +} + +/// The single-column schema every node dataset uses. +/// +/// One `FixedSizeBinary(512)` column. The `key`/`edges`/`value` carving is a +/// read-side projection over byte positions, not three stored columns — see the +/// module docs on why splitting would cost the copy this module avoids. +#[must_use] +pub fn node_schema() -> Arc { + Arc::new(Schema::new(vec![Field::new( + "node", + DataType::FixedSizeBinary(NODE_ROW_STRIDE as i32), + false, + )])) +} + +/// Wrap a row buffer as a `RecordBatch` **without copying it**. +/// +/// `owner` keeps the caller's allocation alive for as long as Arrow holds the +/// buffer; pass the `Arc` that owns the rows (e.g. `Arc>`). `bytes` +/// must point into that allocation. +/// +/// # Errors +/// +/// The buffer length must be a multiple of [`NODE_ROW_STRIDE`]. +pub fn node_rows_batch(bytes: &[u8], owner: Arc) -> Result +where + O: std::panic::RefUnwindSafe + Send + Sync + 'static, +{ + if !bytes.len().is_multiple_of(NODE_ROW_STRIDE) { + return Err(format!( + "row buffer of {} bytes is not a multiple of the {NODE_ROW_STRIDE}-byte stride", + bytes.len() + )); + } + let ptr = std::ptr::NonNull::new(bytes.as_ptr().cast_mut()) + .ok_or_else(|| "row buffer pointer is null".to_string())?; + + // SAFETY: `ptr`/`bytes.len()` describe exactly the caller's slice, and + // `owner` is an `Arc` over the allocation that slice points into. Arrow + // holds that `Arc` for the buffer's whole life, so the memory outlives every + // read. The buffer is never written through: `Buffer` is immutable, and the + // `cast_mut` above only satisfies `NonNull`'s signature. + let buffer = unsafe { Buffer::from_custom_allocation(ptr, bytes.len(), owner) }; + + let array = FixedSizeBinaryArray::new(NODE_ROW_STRIDE as i32, buffer, None); + let column: ArrayRef = Arc::new(array); + RecordBatch::try_new(node_schema(), vec![column]).map_err(|e| format!("record batch: {e}")) +} + +/// Write one 64k table to its own Lance dataset under `db_dir`. +/// +/// Returns the dataset path. `WriteMode::Create` is deliberate: an import writes +/// a table once. Appending to an existing table would let a second import +/// silently double its rows, and a table whose identity space is full has +/// nothing to append anyway. +/// +/// # Errors +/// +/// Propagates batch construction and Lance write failures, including an attempt +/// to write a table that already exists. +pub async fn write_table( + bytes: &[u8], + owner: Arc, + run: TableRun, + db_dir: &str, +) -> Result +where + O: std::panic::RefUnwindSafe + Send + Sync + 'static, +{ + use lance::dataset::{WriteMode, WriteParams}; + use lance::Dataset; + + let slice = bytes + .get(run.byte_range()) + .ok_or_else(|| format!("run {run:?} is out of bounds for {} bytes", bytes.len()))?; + let batch = node_rows_batch(slice, owner)?; + let schema = batch.schema(); + let path = format!( + "{}/{}", + db_dir.trim_end_matches('/'), + run.addr.dataset_name() + ); + + let reader = arrow::record_batch::RecordBatchIterator::new(vec![Ok(batch)].into_iter(), schema); + let params = WriteParams { + mode: WriteMode::Create, + ..Default::default() + }; + Dataset::write(reader, &path, Some(params)) + .await + .map_err(|e| format!("lance write {path}: {e}"))?; + Ok(path) +} + +/// **The import's last step:** write every 64k table of a baked row buffer into +/// `db_dir`, and hand back the table addresses that now exist on disk. +/// +/// After this returns, the database is the thing to address. The row buffer has +/// no further role and the source document certainly does not. +/// +/// # Errors +/// +/// Propagates [`partition`]'s refusals and any per-table write failure. A +/// failure part-way leaves the tables written so far in place — the caller sees +/// which, because the error names the table that failed. +pub async fn write_database( + bytes: &[u8], + owner: Arc, + db_dir: &str, +) -> Result, String> +where + O: std::panic::RefUnwindSafe + Send + Sync + 'static, +{ + let runs = partition(bytes)?; + let mut written = Vec::with_capacity(runs.len()); + for run in runs { + write_table(bytes, Arc::clone(&owner), run, db_dir).await?; + written.push(run.addr); + } + Ok(written) +} + +/// A 64k table opened from disk — the addressable form. +/// +/// Holds the Lance dataset, not a decoded copy of it. Reads go through +/// [`Self::row`], which returns the row's 512 bytes and leaves every reading of +/// them (key / edges / value, and whatever the classid says those mean) to the +/// caller. +pub struct NodeTable { + addr: TableAddr, + dataset: lance::Dataset, +} + +impl NodeTable { + /// Open the dataset for `addr` inside `db_dir`. + /// + /// # Errors + /// + /// Propagates Lance open failures, including a table that was never written. + pub async fn open(db_dir: &str, addr: TableAddr) -> Result { + let path = format!("{}/{}", db_dir.trim_end_matches('/'), addr.dataset_name()); + let dataset = lance::Dataset::open(&path) + .await + .map_err(|e| format!("lance open {path}: {e}"))?; + Ok(Self { addr, dataset }) + } + + /// The address this table answers for. + #[must_use] + pub const fn addr(&self) -> TableAddr { + self.addr + } + + /// Rows stored in this table. + /// + /// Async and fallible because it is a read against the dataset, not a field + /// on this struct — the table holds an address and a handle, never a count + /// it would have to keep in step with the data. + /// + /// # Errors + /// + /// Propagates the Lance scan failure. + pub async fn count_rows(&self) -> Result { + self.dataset + .count_rows(None) + .await + .map_err(|e| format!("lance count_rows {}: {e}", self.addr.dataset_name())) + } + + /// The underlying dataset, for callers that want to query rather than + /// address — this crate's planner surface takes it from here. + #[must_use] + pub const fn dataset(&self) -> &lance::Dataset { + &self.dataset + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Build a row with the given key fields; the rest of the 512 bytes carry a + /// recognisable marker so a copy or a mis-stride would be visible. + fn row(classid: u32, family: u16, identity: u16, marker: u8) -> [u8; NODE_ROW_STRIDE] { + let mut r = [marker; NODE_ROW_STRIDE]; + r[CLASSID_AT..CLASSID_AT + 4].copy_from_slice(&classid.to_le_bytes()); + r[FAMILY_AT..FAMILY_AT + 2].copy_from_slice(&family.to_le_bytes()); + r[IDENTITY_AT..IDENTITY_AT + 2].copy_from_slice(&identity.to_le_bytes()); + r + } + + fn buffer(rows: &[[u8; NODE_ROW_STRIDE]]) -> Vec { + rows.iter().flat_map(|r| r.iter().copied()).collect() + } + + /// The scan cuts where `(classid, family)` changes and nowhere else. + /// + /// The anti-vacuity half is the middle table: a partitioner that emitted one + /// run per row, or one run for everything, fails on the run lengths. + #[test] + fn partition_cuts_on_the_table_prefix_only() { + let buf = buffer(&[ + row(1, 0, 0, 0xA0), + row(1, 0, 1, 0xA1), + row(1, 1, 0, 0xB0), + row(2, 0, 0, 0xC0), + row(2, 0, 7, 0xC1), + row(2, 0, 9, 0xC2), + ]); + let runs = partition(&buf).expect("sorted"); + assert_eq!(runs.len(), 3, "three distinct (classid, family) tables"); + assert_eq!(runs[0].addr, TableAddr { classid: 1, family: 0 }); + assert_eq!(runs[0].len(), 2); + assert_eq!(runs[1].addr, TableAddr { classid: 1, family: 1 }); + assert_eq!(runs[1].len(), 1, "a one-row table is still its own table"); + assert_eq!(runs[2].addr, TableAddr { classid: 2, family: 0 }); + assert_eq!(runs[2].len(), 3); + } + + /// Unsorted input is refused rather than merged — otherwise one table would + /// be written as two datasets and the second would fail on Create, or worse, + /// succeed under a different name. + #[test] + fn partition_refuses_unsorted_and_accepts_sorted() { + let bad = buffer(&[row(2, 0, 0, 1), row(1, 0, 0, 2)]); + assert!(partition(&bad).is_err(), "descending prefix must be refused"); + let good = buffer(&[row(1, 0, 0, 1), row(2, 0, 0, 2)]); + assert_eq!(partition(&good).expect("sorted").len(), 2); + } + + /// A truncated buffer is a mis-strided buffer; refusing it here is cheaper + /// than a silently shifted read of every row after the first. + #[test] + fn a_partial_row_is_refused() { + let mut buf = buffer(&[row(1, 0, 0, 1)]); + buf.truncate(NODE_ROW_STRIDE - 1); + assert!(partition(&buf).is_err()); + assert!(node_rows_batch(&buf, Arc::new(())).is_err()); + } + + /// An empty buffer has no tables — and is not an error. + #[test] + fn an_empty_buffer_yields_no_tables() { + assert!(partition(&[]).expect("empty is legal").is_empty()); + } + + /// The batch is a view over the caller's allocation: the bytes Arrow exposes + /// are the bytes that were handed in, at the same address. + #[test] + fn the_batch_borrows_the_callers_allocation() { + let owner = Arc::new(buffer(&[row(9, 3, 5, 0x5A), row(9, 3, 6, 0x6B)])); + let src_ptr = owner.as_ptr(); + let batch = node_rows_batch(&owner, Arc::clone(&owner)).expect("well-formed"); + assert_eq!(batch.num_rows(), 2); + let col = batch + .column(0) + .as_any() + .downcast_ref::() + .expect("fixed-size binary"); + assert!( + std::ptr::eq(col.value_data().as_ptr(), src_ptr), + "Arrow must point AT the caller's buffer, not a copy of it" + ); + assert_eq!(col.value(0)[NODE_ROW_STRIDE - 1], 0x5A, "row 0 intact"); + assert_eq!(col.value(1)[NODE_ROW_STRIDE - 1], 0x6B, "row 1 intact"); + } + + /// Table names are fixed-width, so a lexical sort of the database directory + /// is an ordering by address. + #[test] + fn dataset_names_sort_by_address() { + let a = TableAddr { classid: 0x02, family: 0x0010 }.dataset_name(); + let b = TableAddr { classid: 0x10, family: 0x0002 }.dataset_name(); + assert_eq!(a, "nodes-00000002-0010.lance"); + assert!(a < b, "classid dominates the ordering, as in the key"); + } + + /// The read-side carving covers the row exactly once — the property that + /// lets one stored column stand in for three. + #[test] + fn the_projection_ranges_tile_the_row() { + assert_eq!(KEY_RANGE.start, 0); + assert_eq!(KEY_RANGE.end, EDGES_RANGE.start); + assert_eq!(EDGES_RANGE.end, VALUE_RANGE.start); + assert_eq!(VALUE_RANGE.end, NODE_ROW_STRIDE); + } +} From fc1fa120165eeb849c833c68c050d26438f57b58 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Fri, 7 Aug 2026 16:20:50 +0200 Subject: [PATCH 2/2] knowledge: the S3 hydration lifecycle States the one-way ratchet -- source -> ContextBundle -> node rows -> LanceDB tables -> reads -- and the rule it exists for: nothing ever hydrates upward. A consumer addresses the calcified form; it does not parse, does not bake, and does not reach for a source artifact. The moment one consumer parses at runtime, the source becomes a live dependency of every deploy and the ratchet is gone. Records why the CALCIFIED form is what sits in object storage rather than the source: the expensive stages are paid once, every reader gets the same bytes, addressing survives compression (the key is never compressed), and a deploy needs an object-store client instead of a format library per vocabulary. Names two seams honestly, because both are load-bearing: - `lance-graph-ontology::hydrators` produces a `ContextBundle` and stops. Nothing carries a bundle into node rows, so the Pattern-D path ends in memory -- it parses, and the result has nowhere to calcify to. The bake path reaches storage; the hydrator path does not. A session adding a `hydrate_*` today is extending the half with no floor. - The bake emits an edge table beside the rows and only the rows are calcified, although the engine already answers reachability over relation matrices (`TypedGraph::traverse`, `blasgraph::ops::hdr_bfs`). Until the edges are carried there, every consumer needing an ancestor walk writes its own over whatever slice it can get. Separates what is verified from what is not. Verified: the partition cut, the mis-stride refusal, the pointer-identity assertion behind the zero-copy write, the read-side range tiling. NOT verified: no object-store URI has been exercised (every test runs on in-memory buffers); zero-copy is measured up to the `RecordBatch` and NOT through Lance's writer; the multi-table `write_database` -> `NodeTable::open` composition has no test. Each is written as an invitation for the first session that measures it to say so there. Mechanism only -- no corpus, source, or terms appear, per the public-repo rule. Credentials reach Lance through the environment or an explicit `storage_options` map, never a committed path. --- .claude/knowledge/s3-hydration-lifecycle.md | 535 ++++++-------------- 1 file changed, 158 insertions(+), 377 deletions(-) diff --git a/.claude/knowledge/s3-hydration-lifecycle.md b/.claude/knowledge/s3-hydration-lifecycle.md index b7d86e45b..69ebba6ae 100644 --- a/.claude/knowledge/s3-hydration-lifecycle.md +++ b/.claude/knowledge/s3-hydration-lifecycle.md @@ -1,386 +1,167 @@ -# S3 is the hydration path, never the store — the on-demand Lance dataset lifecycle - -> **READ BY:** any session that opens a Lance dataset from a URI, wires an -> object-store backend, sizes a persistent volume, debugs a -> `"No object store provider found for scheme"` error, plans a rebake input, or -> proposes putting a dataset "in S3" as a runtime store. Also the -> `integration-lead` / `layer-boundary-warden` cards when a deployment topology -> question arrives. -> -> **Companions:** `.claude/knowledge/zero-copy-lens-law.md` (why a local -> filesystem is not a preference but the precondition — the lens needs mapped -> bytes, and a network fetch has none to lend) · -> `.claude/knowledge/ephemeral-warm-cold-lifecycle.md` (the *reasoning* tier -> ladder; this doc is the *bytes-on-disk* ladder and does not touch it) · -> `docs/DATAFUSION-PERIMETER.md` §11 (the feature-closure half of §3 below). +# The S3 hydration lifecycle -## The one-line statement +> **READ BY:** any session that writes an importer, a hydrator, a bake driver, +> or a deploy that needs an ontology present; before adding any `hydrate_*`, +> any `Dataset::write` call site, or any code that reads a source artifact at +> runtime. +> +> **Scope:** mechanism only. This document names stages, types, and byte +> magnitudes. It does not name corpora, sources, or their terms — those are not +> a public-repo concern and never appear here. -> **The object store hydrates; the local filesystem stores; the volume only -> decides whether hydration repeats.** Three layers, one job each. Collapsing -> any two of them is the failure this doc exists to prevent. +## The one-way ratchet -## Evidence status (per the workspace rule: label everything) +Hydration runs in one direction, and each stage is strictly narrower and +cheaper than the one above it: -| claim | status | evidence | -|---|---|---| -| `lancedb` ships `default = []`; its `aws` feature forwards to `lance/aws` + `lance-io/aws` (+ `object_store/aws` directly in newer releases, transitively via `lance-io` in older ones) | **FINDING** — source-verified in this session | Read directly from the vendored `lancedb` manifests in the local registry, two releases apart; both show `default = []` and the same forwarding shape. | -| `lance-io` carries `aws` in its OWN defaults — so the opt-out is `lancedb`'s layer, not the stack's | **FINDING** — source-verified in this session | Same read: `lance-io`'s `[features] default` includes `aws`. | -| Without the feature, an `s3://` URI fails at provider registration — BEFORE any credential, endpoint or region is consulted | **FINDING** — reported measurement, not re-verified here | Reported by the session that hit it; the error text names the *scheme*, not a credential. Consistent with the manifest facts above (no provider is compiled in). Falsifier: build with the feature off and confirm the same URI fails identically with every env var unset AND with all of them set correctly. | -| Opening the object store as the runtime store makes every read a network fetch into a fresh buffer — no mmap, no page cache | **FINDING** (mechanism), *unbenchmarked here* | Structural: a remote range request has no mapped page to lend. Follows from the zero-copy law. **No A/B benchmark of remote-store vs local-store read paths has been run in this workspace.** | -| Any local directory satisfies zero-copy; a persistent volume is not a correctness requirement | **FINDING** (mechanism) | The store's requirement is a filesystem path, not a durable one. Persistence changes *how often you hydrate*, never *whether reads are zero-copy*. | -| The endpoint characteristics in §5 | **reported measurement, not re-verified in this session** | Measured once, against one S3-compatible endpoint, from one region, at one time. Provider- and region-dependent; treat the *ratios* as the finding and the absolute numbers as a single observation. | -| The flush/rehydrate lifecycle in §4 is the right shape for large single-use datasets | **CONJECTURE** | Argued from §5's ratios, not from a deployed instance. Falsifier stated inline at §4. **No probe has run.** | -| **This repo's own object-store path goes through `lance` (default features, `aws` ON), not through `lancedb`** — so §3's `lancedb` gate is a *consumer-side* trap, not this crate's | **FINDING** — probe run, recorded in §3a | Raised by review on PR #901 and verified against the manifests + call sites; the probe command, its output and the promotion decision are in §3a. **This corrects the first draft of §3**, which stated the `lancedb` gate as if it were the gate on this repo's `s3://` reads. | - -Nothing below is promoted past its row here. - -**Probe record for the two manifest rows above** (re-run any time; all three are -read-only and take seconds): - -```bash -# P1 — the feature declarations, read from the vendored manifests: -sed -n '/^\[features\]/,/^\[[a-z]/p' ~/.cargo/registry/src/*/lancedb-*/Cargo.toml -sed -n '/^\[features\]/,/^\[[a-z]/p' ~/.cargo/registry/src/*/lance-io-*/Cargo.toml -# P2 — which crate THIS repo opens datasets with, and how it is configured: -grep -nE '^(lance|lancedb) *=' crates/lance-graph/Cargo.toml +``` + source artifact text, per-vocabulary format, 10^7–10^9 B + │ parse + ▼ + ContextBundle typed, in memory, per-slot + │ bake + ▼ + node rows + triples 512 B/row, sorted by (classid, family, identity) + │ calcify + ▼ + LanceDB 64k tables on object storage — THE artifact + │ address + ▼ + reads open a table, resolve a position ``` -**Result (2026-08-06):** P1 — `lancedb` `default = []`, `aws = [...]`; `lance-io` -`default = ["aws", "azure", "gcp"]`. P2 — `lance` is a **direct, non-optional** -dependency taken **with default features**, and `lance`'s own -`default` includes `aws`; `lancedb` is `optional = true, default-features = -false` behind a separate feature. **Promotion decision:** the manifest rows stay -FINDING; the *inference* drawn from them in the first draft of §3 is **corrected** -by §3a rather than promoted. - -## 1. The three layers - -| layer | its ONE job | if it is absent | +**The rule this document exists to state: nothing ever hydrates upward.** A +consumer addresses the calcified form. It does not parse, it does not bake, and +it does not reach for the source artifact — not to check something, not to fill +a gap, not "just this once". A gap in the calcified form is fixed by +re-calcifying, upstream, once, for everyone; it is never patched by a consumer +re-entering an earlier stage. The moment one consumer parses at runtime, the +source artifact becomes a live dependency of every deploy, and the ratchet is +gone. + +## Why the *calcified* form is what sits in object storage + +It is tempting to store the source artifact and hydrate on startup. That is the +wrong end of the chain, for reasons that compound: + +- **Cost is paid once, not per consumer and not per deploy.** Parsing and + baking are the expensive stages; a stored calcified form amortizes them over + every reader that will ever exist. +- **Every reader gets the same bytes.** Two consumers that each parse the same + source can disagree — different versions of a parser, a different day, a + silently-updated artifact. Two consumers that open the same dataset cannot. +- **Addressing survives compression.** The key is never compressed; Lance may + encode the value slab however it likes and a reader can still route, group, + and skeleton-render from keys alone. A stored source artifact has no + addresses at all until someone parses it. +- **A deploy needs no parser.** The runtime dependency is an object-store + client, not a format library per vocabulary. + +## Stage inventory — what exists, what is a seam + +| Stage | Where it lives | State | |---|---|---| -| **object store** (S3-compatible) | **hydration source** — durable, versioned, shared between machines and between builds | fall back to whatever secondary source the consumer already has; the store still works, the dataset just has to come from somewhere else | -| **local directory** | **THE Lance store** — the path the process opens; zero-copy mmap reads, page cache, no network in the read path | **no fallback — always required.** Any path on a **supported, mmap-capable local filesystem** satisfies it (see the qualification below) | -| **persistent volume** | decides **which** local directory — chosen only because it survives redeploys | hydrate on every boot; still correct, merely slower | - -Read the third row twice. The volume is an **optimization on hydration -frequency**, not a component of the store. A design that says "we need a volume -or this doesn't work" has mis-assigned a job: what it needs is a directory. - -**The qualification on "any local path" (raised by review, PR #901).** What the -store needs is not merely *a path that is not a URI* — it is a filesystem that -actually delivers the mmap and locking semantics the zero-copy read depends on. A -network filesystem (NFS/EFS-class), a FUSE mount, or an overlay with unusual -caching presents a perfectly ordinary local-looking path while changing page-cache -behaviour, consistency, and lock semantics underneath it. Those are the cases -where "it is a local directory, therefore reads are zero-copy" stops being true. - -So the requirement is **a supported, mmap-capable local filesystem**, and the two -axes stay separate: - -- **correctness** — mmap-capable filesystem. Not negotiable, and not satisfied by - path *shape*. -- **hydration frequency** — durability/persistence. Purely an optimization, as - the third row says. - -An ephemeral container path on an ordinary local filesystem satisfies the first -and not the second, which is exactly the intended trade. A network mount may -satisfy the second and *not* the first, which is the trap — and it is the same -trap as §2 one level down, since a network filesystem reintroduces the network -into the read path while still looking like a directory. - -## 2. Why the object store must not be the store — even though the URI works - -Lance opens an `s3://` URI natively — **given the object-store feature its -provider registration needs** (§3, and §3a for which crate's feature that is in -any given consumer). That is exactly what makes this trap easy to fall into: with -the feature on, the wrong architecture **runs**, correctly, and only degrades. The -feature being off produces a different, louder failure and is §3's subject; this -section is about the case where it works. - -The reason it is wrong is the same reason the zero-copy law exists one layer -down. A local dataset read is a mapped page — the kernel hands you bytes that -are already resident, and a lens over them costs a cast. A remote object read -is a range request that lands in a **freshly allocated buffer**: one copy per -read, minimum, plus a round trip, and no page cache to make the second read of -the same bytes free. - -So the failure is not "S3 is slow." It is that **mounting the object store as -the runtime store deletes the mmap layer from the architecture** — every -downstream zero-copy guarantee is then a claim about buffers that were copied -into existence. The lens has nothing to borrow from. - -> **The review question:** *where does the process open its dataset from?* If -> the answer is a URI with a network scheme, the zero-copy story below it is -> already void, regardless of what any type signature promises. - -**Corollary — the local directory has no minimum *durability*.** An ephemeral -container path on an ordinary local filesystem is functionally correct: mmap -works, the page cache works, the lens works. Losing that directory on redeploy -costs a re-hydration, not a correctness property. This is why §1's third row is an -optimization and not a requirement. (It has no minimum durability; it does have a -minimum *filesystem* — see §1's qualification. "No minimum quality", as the first -draft put it, was too strong.) - -## 3. The feature gate that costs an hour if you don't know it - -**`lancedb` ships `default = []`.** Its `aws` feature is what forwards to -`lance/aws` + `lance-io/aws` (+ `object_store/aws`), and *that* forwarding is -what registers the S3 provider. - -Two consequences, both non-obvious: - -1. **Without the feature, an `s3://` URI fails at provider lookup — before any - credential, endpoint, or region is read.** The error names the *scheme*. - That means **no amount of endpoint/region/credential/quoting debugging can - possibly help**, because none of that code has been reached. Every minute - spent on env vars is spent on a code path that does not exist in the binary. -2. **`lance-io` DOES carry `aws` in its own defaults.** So the intuition "the - Lance stack supports S3 by default" is *true one layer down* and false at - the layer you depend on. `lancedb` is the layer that opts out. That mismatch - is the whole reason the diagnosis goes wrong: the mental model is correct - about the wrong crate. - -**The diagnostic rule, mechanical:** an object-store error that names a -**scheme** is a *build* problem (a feature is off). An object-store error that -names a **credential, host, bucket, region, or signature** is a *config* -problem. Never debug the second when you are looking at the first. Read the -error's noun before touching an env var. - -*(Env var **names** — `AWS_ENDPOINT_URL`, `AWS_REGION`, and the standard -credential pair — are the config surface for the second class only. They are -inert against the first.)* - -### 3a. …but diagnose the crate that actually opens YOUR uri — this repo's is `lance` - -**Correction, raised by review on PR #901 and verified (probe record in -§ Evidence status).** §3 above is true *about `lancedb`*, and the first draft -stated it as though it were the gate on this repository's object-store reads. It -is not. - -| | crate | how this repo takes it | `aws` in effect? | -|---|---|---|---| -| what `VersionedGraph::{s3,azure,gcs}` reads through | **`lance`** | direct, **non-optional**, **default features** | **YES** — `lance`'s own `default` includes `aws` | -| the optional SDK surface | `lancedb` | `optional = true`, **`default-features = false`**, behind its own feature | **NO**, unless that feature turns it on | - -So the mechanical rule in §3 stands, but its *first step changes*: **resolve -which crate opens the URI before you look at any manifest.** The §3 story — "the -mental model is correct about the wrong crate" — is exactly the trap this -subsection exists to stop this document from itself falling into, one layer -further out. - -Restated so it is checkable rather than remembered: - -1. Find the call that opens the URI, and name the crate it belongs to. -2. Read **that** crate's feature declarations, and how *this* manifest takes it - (a `default-features = false` on the dependency line overrides the upstream - default, and is easy to miss). -3. Only then decide whether a scheme-named error is a build problem here. - -A consumer that opens datasets through `lancedb` is squarely in §3's case. A -consumer that opens them through `lance` with default features is not — and for -that consumer, a scheme-named error means something else and the §3 diagnosis -would send it down the wrong path. - -## 4. The lifecycle — four states, and what each transition costs - -The actual operational ask: **large, single-use datasets** (rebake inputs, -one-off derivations, build artifacts) should not occupy local disk permanently. -They hydrate when needed, get pushed back if mutated, and their local copy is -reclaimed. - -| state | what exists where | invariant | -|---|---|---| -| **absent** | object store only | reads are impossible; the store is not open | -| **hydrated** | object store + local dir, identical | reads are zero-copy; this is the only readable state | -| **dirty** | local dir has diverged (written/appended/compacted) | **the local copy is now the only truth** — flushing here destroys data | -| **flushed** | object store only, local reclaimed | ≡ *absent*, but reached deliberately after a push | +| parse → `ContextBundle` | `lance-graph-ontology::hydrators` (Pattern D: generic `OwlHydrator` + ~50 LOC glue per vocabulary) | shipped, many vocabularies | +| bake → node rows + triples | the harvest side, outside this repo | shipped | +| calcify rows → LanceDB | `lance_graph::graph::ontology_hydrate` | shipped | +| calcify triples → relation matrices | — | **seam** | +| `ContextBundle` → node rows | — | **seam** | +| address the tables | `ontology_hydrate::NodeTable` | shipped | + +**Two of the arrows above do not exist yet, and the gaps are load-bearing.** + +The first: `hydrators` produces a `ContextBundle` and stops. Nothing carries a +bundle into node rows, so the Pattern-D path currently ends in memory — it +parses, and then the result has nowhere to calcify to. The bake path reaches +storage; the hydrator path does not. These are two hydration pipelines that do +not meet, and a session that adds a `hydrate_*` today is extending the half +that has no floor under it. + +The second: the bake emits an edge table alongside the rows, and only the rows +are calcified. The engine already answers reachability over relation matrices +(`TypedGraph::traverse`, `blasgraph::ops::hdr_bfs`), so the edges have a +destination — they are simply not carried to it. Until they are, every consumer +that needs an ancestor walk writes its own, over whatever slice it can get. + +## Object-store addressing + +`ontology_hydrate::write_database` and `NodeTable::open` take a database +directory as a string and hand it to Lance, which resolves it through +`object_store`. A local path and an object-store URI are therefore the same +call site — the writer has no S3 branch, and adding one would be the mistake. +The database directory is a *location*, and the only thing this layer knows +about it is that Lance can reach it. + +One table is one dataset under that directory: -Transitions and their costs: +``` +/nodes--.lance +``` -| transition | cost | gate | -|---|---|---| -| absent → hydrated | one large sequential read (§5: sustained, amortized) + one connect | safe to repeat **within the boundary below** — not unconditionally | -| hydrated → dirty | a local write; free | — | -| dirty → hydrated | **push back** — the expensive direction (§5: writes are ~½ read throughput and pay per-fragment object overhead) | must complete before flush, or the divergence is lost | -| hydrated → flushed | a local delete; frees disk | **only legal from `hydrated`, never from `dirty`** | -| flushed → hydrated | same as absent → hydrated | — | - -**The one rule that matters:** *flush is legal only from `hydrated`, never from -`dirty`.* The state machine exists to make that a checkable condition rather -than an assumption. A `dirty → flushed` edge is data loss with no error. - -### 4a. The idempotency boundary — `absent → hydrated` is NOT unconditionally safe to repeat - -**Correction, raised by review on PR #901.** The first draft's "safe to repeat, it -is idempotent" was too strong, and the strength was load-bearing: the eviction -plan leans on that word to argue a lost race costs only a wasted rehydration. A -Lance dataset is a **multi-file directory**, so: - -- a hydration that **fails part-way** leaves a partial directory, and a retry - that treats it as a destination rather than as debris merges two attempts; -- a hydration against a **different source version** than a previous one mixes - files from two snapshots into one directory — each file individually valid, the - directory as a whole not a dataset that ever existed; -- a **concurrent** reclaim (§4's `hydrated → flushed`) deleting files from that - same directory can remove what a hydration just wrote, or expose a reader to a - directory that is neither complete nor absent. - -None of these is prevented by the transfer being repeatable. So the property is -**conditional**, and the conditions are the contract: - -> `absent → hydrated` is idempotent **given (a) a pinned source version and (b) a -> destination that is empty and not concurrently mutated.** Outside those two -> conditions it is not idempotent, it is a merge. - -**The mechanism that makes both conditions hold — hydrate aside, publish by -rename.** Fetch into a private temporary directory, then make it visible with a -single atomic directory rename; retire by renaming *away* first and deleting the -renamed copy afterwards. A reader therefore only ever resolves a name that is -either absent or a complete dataset, never one mid-assembly or mid-removal. A -failed hydration leaves only an unpublished temporary directory, which is debris a -sweep can delete without consulting anything. - -This is a **filesystem-atomicity boundary, not a coordination protocol** — it adds -nothing to the read path, takes no lock, and holds no lease. That distinction -matters because the eviction plan explicitly rejects a lease/refcount protocol; -this requirement is compatible with that rejection, and is what makes its "worst -case is a wasted rehydration" claim actually true. See -`.claude/plans/idle-flush-dataset-eviction-v1.md` §5a. - -**Why writes are an ops step and never a boot path:** the push-back direction -pays object-per-fragment overhead on top of raw throughput (§5), so its cost -scales with fragmentation as well as bytes. Hydration is boot-viable; the return -trip is not. Any design that puts a write-back on a startup path has put the -slowest, most fragmentation-sensitive operation in front of the readiness check. - -*Falsifier for the CONJECTURE row:* run the full cycle on a representative -dataset and confirm (a) hydrate wall-clock stays inside the boot budget, (b) a -`dirty → flushed` attempt is refused rather than silently accepted, and (c) a -rehydrate after flush reads back byte-identically. Until that runs, §4 is a -design, not a result. - -## 5. Measured characteristics of one S3-compatible endpoint - -> **Single observation.** One provider, one region, one point in time, -> deliberately unnamed. **Provider- and region-dependent.** The *ratios* are -> what generalize; the absolute numbers do not. Reported by the session that -> measured them; **not re-run here.** - -| operation | observed | what it settles | -|---|---|---| -| small-object round trip | **~250 ms** | | -| large sequential read | **~21 MiB/s** | | -| large sequential write | **~11 MiB/s** (≈ ½ the read rate) | | -| store connect | **~730 ms** | | -| cold re-open + full count, dataset in the tens-of-MB range (~69k rows, ~35 MB) | **~1.4 s** | boot-viable | -| write of that same dataset | **~7.7 s** (≈ 4.4 MiB/s effective — below the raw write rate, the gap being object-per-fragment overhead) | ops step, not a boot path | - -**The round-trip number is the load-bearing one.** At ~250 ms per small object, -the object store is roughly **~2.5 million×** slower than RAM and **~2500×** -slower than NVMe. Those two ratios are the whole argument: - -- **NOT viable as swap.** A page fault backed by a ~250 ms fetch is not a - memory hierarchy; it is a hang with a progress bar. -- **NOT viable as a page-fault backing store**, for the same reason — and this - is precisely §2 restated in numbers: mounting it as the runtime store puts - that latency *under every read*. -- **VIABLE for hydration.** One large sequential transfer amortizes the round - trip across the whole dataset; the effective cost is the ~21 MiB/s line plus - one connect. -- **VIABLE for build caches** — same shape: large objects, few of them, latency - amortized. - -The rule that falls out: **the object store is fine when the object count is -small and the objects are large; it is unusable when the access count is large -and the accesses are small.** Every viable/non-viable verdict above is that one -sentence applied twice. - -### 5a. "Boot-viable" is a claim about a SIZE, and the size is in the table - -**Correction, raised by review on PR #901.** "VIABLE for hydration" above is a -verdict about the *shape* of the access (few, large, sequential). It is **not** a -verdict about any dataset size, and the row it was measured on is in the -tens-of-megabytes range. Carried forward naively it becomes a boot-viability claim -for arbitrary datasets, which the same numbers refute: - -| dataset size | implied transfer at the observed sequential rate | plus connect | boot-viable? | -|---|---|---|---| -| the measured ~35 MB | ~1.7 s | + ~0.7 s | yes — matches the observed ~1.4 s | -| ~256 MB | ~12 s | + ~0.7 s | depends entirely on the boot budget | -| ~1 GiB | **~49 s** | + ~0.7 s | **no**, against any ordinary readiness deadline | - -The linear term dominates the moment the round trip stops being the cost, which is -almost immediately. So the honest statement is: **hydration is the right *shape* -at any size; whether it fits a boot budget is a size question that must be -answered against the actual dataset and the actual budget**, and only the -tens-of-megabytes case has been measured here. - -**What is NOT stated**, and should not be inferred: no RAM or NVMe baseline was -measured in this workspace — the "~2.5 million×" and "~2500×" ratios use -conventional figures for those tiers, not measurements taken here, and they are -order-of-magnitude arguments rather than benchmarks. Neither is the measurement -method recorded (single run vs. best-of-N, cold vs. warm client). Treat the whole -of §5 as one observation with a shape, not as a performance model. - -## 6. Consequences for new work - -- **Never open a network-scheme URI as the runtime store.** Hydrate to a local - path, open the local path. If a design opens the remote URI directly, the - finding is not "this is slow" — it is that the zero-copy layer has been - removed. -- **Do not require a persistent volume for correctness.** Require a *directory*. - State the volume as an optimization with a named cost (one hydration per - boot), so a deployment without one is a known trade rather than a bug report. -- **Gate the object-store feature explicitly, and say so where the URI is - parsed.** A scheme-named error must lead the reader to the manifest, not to - the credentials. -- **Never place a write-back on a startup path.** Push-back is an operational - step with its own trigger. -- **Any flush path must assert `hydrated`, not assume it.** The `dirty → flushed` - edge fails silently by construction; only an explicit check catches it. -- **Hydrate aside and publish by rename** (§4a). "Repeatable transfer" is not - idempotence over a multi-file directory. - -### 6a. Scope — what this doctrine binds, and the shipped API it does NOT invalidate - -**Correction, raised by review on PR #901.** The first bullet above was written -categorically, and read that way it declares an existing, tested, public API -architecturally invalid while offering no replacement. That is not what it means, -and the scope belongs in the document rather than in the reader's judgement. - -`crates/lance-graph/src/graph/versioned.rs` ships `VersionedGraph::{s3, azure, -gcs}`. Each stores a network URI as `base_path` and the read methods pass it -straight through, so those constructors *are* the pattern §6's first bullet warns -about. They are **not deprecated by this document**, and nothing here removes -them. - -**What the rule binds:** the **hot zero-copy substrate** — any read path whose -correctness story includes mapped bytes, a lens over them, or a page-cache -assumption. There, opening a network-scheme URI does not degrade the guarantee, it -**voids** it, and the finding is structural rather than about speed. - -**What the rule does not bind:** occasional, non-hot access where no zero-copy -claim is being made — administrative reads, one-off inspection, a version listing, -a small metadata query. The remote constructors remain the correct tool for those, -and calling one is not a violation. - -**Where that leaves the constructors:** they are **usable and unmigrated**, which -is a known state rather than a silent one. The missing piece is a hydrating -counterpart (`hydrate_from(remote) → local`) so a caller that *does* have a -zero-copy story has somewhere to go; that does not exist yet and this -documentation-only change does not add it. Tracked as -`.claude/board/ISSUES.md` `ISS-REMOTE-URI-CONSTRUCTORS-PREDATE-THE-HYDRATION-DOCTRINE`. - -Until it exists, the honest instruction to a caller is: **choose by read shape, -not by constructor availability.** If your reads are hot and zero-copy, hydrate to -a local path yourself and use `local()`. If they are occasional and you are -claiming nothing about mapped bytes, the remote constructor is fine. - -## Cross-refs - -`.claude/knowledge/zero-copy-lens-law.md` (the law this doc is the storage-siting -corollary of) · `.claude/knowledge/ephemeral-warm-cold-lifecycle.md` (the -reasoning-tier ladder — orthogonal; do not conflate its cold tier with this -doc's flushed state) · `docs/DATAFUSION-PERIMETER.md` §11 (feature-closure half) -· ADR-022/023 (the Firewall — no serialization in the hot path; a remote read -path is that violation arriving through the storage layer). +The name is fixed-width and zero-padded, so a lexical listing of the database +is an ordering by address — `ls` on the bucket prefix is a table index, with no +catalog to keep in step. + +Table extent follows from the key, not from a tuning decision: `identity` is a +`u16`, so a `(classid, family)` pair addresses exactly 65 536 slots, and a full +table is 64 Ki × 512 B = 32 MiB. That is a deliberate size for object storage — +large enough that per-object overhead is negligible, small enough that a reader +fetching one basin does not pull a neighbouring one. + +## Credentials + +Storage credentials reach Lance through the process environment or an explicit +`storage_options` map at the call site. They are never committed, never written +into a path, and never baked into an artifact. A dataset URI in a config file +or a commit message must contain a bucket and a prefix and nothing else. + +If a credential ever appears in a repository, a log, or a URI, treat it as +compromised and rotate it — scrubbing the text is not sufficient, because +history is retained. + +## Re-calcification + +Tables are written with `WriteMode::Create`. An import writes a table once; a +second import against a live database fails loudly rather than appending, since +appending would double a table's rows without any signal. + +A changed source therefore means a **new** calcified form, not a mutated one. +Lance's own versioning is the mechanism for holding both while readers move +across; a re-import that overwrote in place would break every reader mid-read +and would destroy the property that two readers of one dataset see one truth. + +## What is verified, and what is not + +Verified by test in `ontology_hydrate`: + +- the run partition cuts on `(classid, family)` and refuses unsorted input; +- a mis-strided buffer is refused rather than read shifted; +- the `RecordBatch` **points at the caller's allocation** — pointer identity is + asserted against the source buffer, so the zero-copy write is measured, not + claimed; +- the read-side `key`/`edges`/`value` ranges tile the row exactly once. + +**Not verified:** + +- **No object-store URI has been exercised.** Every test to date runs against + in-memory buffers; `Dataset::write` to an `s3://`-style location is expected + to work because Lance resolves the path through `object_store`, but expected + is not measured. The first session to run it should say so here. +- **Zero-copy is verified up to the `RecordBatch`, not through the write.** + Whether Lance's encoder streams that buffer or stages a copy of it before it + reaches the object store has not been measured. The claim in this document is + precisely "the batch borrows the caller's rows", and it should not be widened + to "no copy occurs anywhere" until someone measures the writer. +- **Whole-database round-trip.** `write_database` then `NodeTable::open` over a + multi-table buffer has no test; the pieces are tested, the composition is not. + +## The failure this document is meant to prevent + +A consumer that cannot find something in the database, and reaches back up the +chain to get it — parsing a source artifact at runtime, or shipping its own +copy of one. It always looks local and reasonable, and it costs the property +the whole chain exists for: that every reader sees the same bytes, and that the +expensive stages ran once. + +If something is missing from the calcified form, the fix is upstream, and it is +someone's job. Say so; do not route around it.