perf(datahub): match export rows against a set of listed subject ids

[gdevenyi]

Addresses item 1 of #1412.

The bug

The export filter built its list of subject ids by iterating each row's visible cells:

.rows.flatMap((row) => row.getVisibleCells().map((cell) => removeSubjectIdScope(cell.row.original.id)))

The callback ignores cell entirely and reads cell.row.original.id — the same value for every cell in the row. So the row's id is emitted once per rendered column, and removeSubjectIdScope runs once per column too.

That array was then the right-hand side of an includes inside a filter over the export, making the match a linear scan per exported row, against a list several times longer than it needed to be.

Measured, not estimated — and the issue understated it

I rendered the real table and counted. It has three data columns plus the row-actions column libui injects, which #1412 didn't account for:

cellsPerRow=4  rows=5000
oldArrayLength=20000  newSetSize=5000  factor=4.0

So the duplication is 4×, not the "at least three" I wrote in the issue.

Filtering 200,000 export rows against 5,000 listed subjects, identical results both ways (167,000 matched):

before	3,070 ms
after	17 ms

That is ~180×, and it is conservative: a 25,000-record group with 30 measures per record produces roughly 750,000 export rows, ~3.75× the size benchmarked here.

The change

const getListedSubjectIds = (table: TanstackTable.Table<Subject>): Set<string> => {
  return new Set(table.getPrePaginationRowModel().rows.map((row) => removeSubjectIdScope(row.original.id)));
};

and listedSubjects.has(...) at the call site. Extracted to a named helper so the reason it reads one id per row is documented where someone would otherwise reintroduce the cell loop.

Tests

Three, driving the real DataTable with the master table's shape (three columns + row actions) rather than a hand-built stand-in — so the duplication claim is verified rather than assumed:

• one id per row, not one per cell (asserts getVisibleCells() genuinely over-counts, and that the current extraction doesn't);
• membership matches the right export rows;
• an empty table excludes everything, rather than accidentally matching.

Checks

• tsc --noEmit on apps/web — clean
• eslint src — clean
• prettier --check — clean
• vitest run (whole workspace) — 255 passed, 1 skipped, 1 failed

The one failure is instrument-records.service.spec.ts > upload > should create records with pending set. This branch is cut from main, so it carries that pre-existing failure; #1422 fixes it.

What is still open in #1412

The filtering is still client-side. The export is fetched for the entire group and narrowed in the browser, so the sex filter, the date-of-birth range, the search string and the "with records only" toggle are all honoured only after the payload has crossed the network. Pushing the subject ids (or the filter criteria) to /v1/instrument-records/export is the real fix, and it composes with the streaming rework proposed in #1407 — worth doing together rather than bolting a subjectIds parameter onto an endpoint that is about to be restructured.

Item 3 of #1412 — the same anti-pattern in the audit log download — is already handled in #1419, which replaces it with a server-side fetch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bfen9VQemZanJLXinJ6MMG

[Copilot]

Pull request overview

This PR improves Datahub export filtering performance by extracting one subject ID per listed table row (instead of per visible cell) and using a Set for constant-time membership checks when filtering exported rows client-side.

Changes:

• Replace per-cell subject ID collection (getVisibleCells().flatMap(...)) with a per-row extraction helper that deduplicates via Set.
• Update export filtering to use Set.has(...) instead of Array.includes(...).
• Add a new test file that renders a real DataTable shape (including row actions) to validate the underlying “per-cell duplicates IDs” behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
apps/web/src/routes/_app/datahub/index.tsx	Switch export filtering to per-row ID extraction and Set membership checks for large performance gains.
apps/web/src/routes/_app/datahub/tests/index.test.tsx	Add tests around table row-model behavior and basic set membership filtering.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.