Skip to content

Put the loading index on a node-local NVMe ephemeral volume (blocked on #280)#281

Open
gaurav wants to merge 1 commit into
mainfrom
nvme-ephemeral-loading
Open

Put the loading index on a node-local NVMe ephemeral volume (blocked on #280)#281
gaurav wants to merge 1 commit into
mainfrom
nvme-ephemeral-loading

Conversation

@gaurav

@gaurav gaurav commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

⚠️ Do not merge until #280 is done

This needs nvme-ephemeral generic ephemeral volumes to work in the data-loading
namespace, and they currently do not — see #280. Merging before that is sorted out
leaves a loading pod that cannot be scheduled.

Stacked on speed-up-loads (#278), so the diff here is only the NVMe change. Once
#278 merges and its branch is deleted, GitHub retargets this to main.

What this does

Moves the loading Solr index onto a node-local NVMe ephemeral volume, declared inline
in nameres-loading.k8s.yaml, instead of the nameres-loading-solr PVC on network
storage.

After the parallelisation work in #278, a load is bounded by its two disk-bound
phases: merging, and the final optimize=true, which reads ~130G and writes ~130G in
a single forced merge. This is the volume that absorbs all of it, so its throughput
decides how long a load takes — more than anything left to tune in the Makefile.

Only the index moves, because the two volumes want opposite things:

  • /var/solr benefits most from being fast, and is pure intermediate state. Once
    the tarball exists the index is worthless, and losing it with the pod costs only the
    time to rebuild it from the synonym files on the other volume.
  • data/ holds the ~130G download and the finished backup tarball — the only
    thing the whole exercise produces — and stays persistent.

So deleting the pod throws away the fast, rebuildable half and keeps the slow,
irreplaceable half.

fsGroup: 1000 matches the image's nru user; without it the volume arrives
root-owned and Solr cannot write to it.

Notes

  • nameres-loading-solr.k8s.yaml is deleted, not kept as a fallback. A manifest
    nobody applies is a manifest nobody maintains, and it would drift out of step with
    the pod it pairs with. The README documents the one-command recovery from git
    history for the only case that needs it: no node with 600Gi of local NVMe free.
  • 600Gi, not 400Gi. Sized on speed-up-loads for an index well above the ~127Gi
    measured for Babel 2025nov4, since releases grow and the sizing has to hold for the
    release being loaded. optimize=true writes the new segment before dropping the old
    ones, so peak is 2–3x the finished index, and running out of room happens at the
    last step of a multi-hour load.

What to check once #280 lands

  • The pod schedules — some node has 600Gi of local NVMe free alongside 32 CPU and
    128Gi. If not, lower the request, but not below roughly 2x the finished index.
  • The merge and optimize phases are visibly shorter in Grafana. If disk throughput on
    the Solr volume still flat-tops, the claim did not bind to an NVMe class — check
    with kubectl get pvc while the pod runs.

The index is the volume that wants speed -- it takes the whole write load of the
indexing run and then the read-and-write of the optimize -- and it is also the
one volume we can afford to lose, since once the tarball exists the index is
worthless and could be rebuilt from the synonym files next door. That makes it an
exact fit for nvme-ephemeral: created with the pod, destroyed with it.

data/ stays on a persistent PVC. It holds the ~130G download and the backup
tarball, which is the only thing the exercise produces.

fsGroup 1000 matches the image's nru user; without it the volume arrives
root-owned and Solr cannot write to it.

nameres-loading-solr.k8s.yaml is deleted rather than kept as a fallback: a
manifest nobody applies is a manifest nobody maintains, and it would sit there
slowly drifting out of step with the pod it is supposed to pair with. The README
explains how to get it back out of git history in the one case that needs it --
no node with 600Gi of local NVMe free.

Blocked on #280: the namespace cannot currently create these volumes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base automatically changed from speed-up-loads to main July 23, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant