fix(api): resolve the instrument subject filter without a relation join

[gdevenyi]

Fixes #1416.

GET /v1/instruments/info?subjectId= returns HTTP 500 — and the datahub subject view stops rendering entirely — once a single instrument accumulates roughly 182,000 records.

Cause

InstrumentsService.find expressed "instruments this subject has records for" as a prisma relation filter:

records: query.subjectId ? { some: { subjectId: query.subjectId } } : undefined

On MongoDB, prisma compiles that into a $lookup which materialises every record belonging to an instrument into a single array before applying the predicate. MongoDB caps one document's $lookup output at 100 MiB and aborts the whole aggregation past it (error 4568):

Executor error during aggregate command on namespace: <db>.InstrumentModel
:: caused by :: Total size of documents in InstrumentRecordModel matching
pipeline's $lookup stage exceeds 104857600 bytes

The limit cannot be raised, and allowDiskUse does not apply to it.

Two things make this worse than the raw number suggests:

• The threshold is per-instrument, not per-collection, so the busiest instrument hits it first. A core intake questionnaire administered at every visit reaches 182,000 records at 5,000 subjects × 36 visits — an ordinary size here.
• There is no partial degradation. useInstrumentVisualization calls this endpoint on mount, and it backs both /datahub/$subjectId/table and /datahub/$subjectId/graph, so both pages fail and the user has no workaround.

The fix

Query the child collection instead of joining from the parent. The work is then bounded by the subject's records (tens) rather than the instrument's (hundreds of thousands), and there is no size ceiling:

const subjectInstrumentIds = query.subjectId ? await this.findInstrumentIdsBySubject(query.subjectId) : null;

const instruments = await this.instrumentModel.findMany({
  where: {
    AND: [subjectInstrumentIds ? { id: { in: subjectInstrumentIds } } : {}, accessibleQuery(ability, 'read', 'Instrument')]
  }
});

Verified against a live instance

Real API, real MongoDB replica set, database provisioned through the app's own POST /v1/setup, then one instrument pushed past the ceiling.

Above the ceiling — busiest instrument at 105.7 MiB, identical database for both:

	result
main	HTTP 500 (three times), exceeds 104857600 bytes
this branch	HTTP 200 in 77 ms, 811 bytes

Below the ceiling — same 55.3 MiB instrument, where main still answers:

	latency
main	1.83s / 1.69s / 1.66s
this branch	0.10s / 0.042s / 0.041s

So this is not only an outage fix: even well under the limit the endpoint was taking ~1.7 seconds to return 811 bytes, and it now takes ~40 ms.

Tests

Three added to instruments.service.spec.ts:

• the subject filter is resolved against InstrumentRecord and the instrument query contains no records relation filter (this is the regression guard — it asserts the absence of the construct that fails);
• no record query is issued when no subject is given;
• a subject with no records yields { id: { in: [] } } rather than an unfiltered query.

The existing findInfo tests are unchanged and still pass.

Checks

• tsc --noEmit on apps/api — clean
• eslint src — clean
• prettier --check — clean
• vitest run (whole workspace) — 255 passed, 1 skipped, 1 failed

The one failure is instrument-records.service.spec.ts > upload > should create records with pending set. This branch is cut from main, so it still carries that pre-existing failure — #1422 fixes it, and the suite is fully green there.

Notes

• @@index([instrumentId]) on InstrumentRecord (proposed in Performance: no indexes are ever created on any MongoDB collection #1406) is a useful mitigation for the old query but not a fix — it does not remove the 100 MiB ceiling. This change removes the ceiling from the picture entirely; the index still helps the new query.
• Because the failure only appears above a data-volume threshold, no small-fixture test would have caught it. A seeded at-scale test in the e2e suite would be worth adding separately.
• I swept for other relation filters on high-cardinality relations. FilesService.find uses files: { every: ... }, which is bounded by files-per-record and is safe today; it is the only other instance.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bfen9VQemZanJLXinJ6MMG

[Copilot]

Pull request overview

This PR fixes a MongoDB/Prisma scaling failure in the API’s instrument info query (GET /v1/instruments/info?subjectId=) by avoiding a high-cardinality relation filter that Prisma compiles into a $lookup capable of exceeding MongoDB’s 100 MiB per-document limit. Instead of joining from Instrument to InstrumentRecord, it first queries the child collection to find relevant instrument IDs for the subject, then filters instruments by those IDs.

Changes:

• Replace the records: { some: { subjectId } } Prisma relation filter with a two-step query: InstrumentRecord distinct instrument IDs → Instrument filtered by id in [...].
• Add a private helper (findInstrumentIdsBySubject) to encapsulate the distinct ID lookup.
• Add unit tests guarding against reintroducing the relation filter and verifying the new query behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
apps/api/src/instruments/instruments.service.ts	Reworks subject filtering to query InstrumentRecord for distinct instrument IDs, avoiding a MongoDB $lookup blowup.
apps/api/src/instruments/tests/instruments.service.spec.ts	Adds regression tests ensuring the subject filter is resolved via InstrumentRecord and that no record query runs when subjectId is absent.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.