perf(api): group the has-record subject ids in the database

[gdevenyi]

Fixes #1413.

The "with records only" datahub filter resolved its subject list with:

findMany({ distinct: ['subjectId'], select: { subjectId: true }, where })

Prisma applies distinct in the query engine, not in MongoDB, so the driver receives one row per record and collapses them to one row per subject only after they have arrived. On a group holding 100k records that is 100k rows transferred to produce a list of at most a few thousand ids.

groupBy asks MongoDB for the distinct values directly.

Measured over HTTP, live instance, 200,300 records, median of 7

	main	this branch
GET /v1/subjects?hasRecord=true	366 ms	42 ms
GET /v1/subjects?groupId=<g>&hasRecord=true	67 ms	37 ms
GET /v1/subjects?groupId=<g> (filter off, for reference)	4 ms	3 ms

The unscoped case is where the difference shows, and it widens with the collection: groupBy returns one row per subject regardless of how many records back it, so its cost is flat in records-per-subject where the old form was linear.

Confirmed at the query level too — the same comparison run directly through the Prisma client:

findMany distinct (current impl), scoped        197ms  -> 55 ids
findMany distinct (current impl), unscoped     1229ms  -> 100 ids
groupBy by subjectId, scoped                     95ms  -> 55 ids
groupBy by subjectId, unscoped                  100ms  -> 100 ids

Note distinct going 197 → 1229 ms as the scope widens while groupBy stays at ~100 ms.

The issue's "better still" suggestion is wrong — please don't take it

#1413 floated expressing this as a relation filter on Subject:

{ instrumentRecords: { some: groupId ? { groupId } : {} } }

and said it was "the version worth trying first". I measured it on the same dataset and it takes 26,849 ms — 270× slower than the code being replaced, let alone the replacement:

subject.findMany({instrumentRecords:{some}})   26849ms  -> 55 subjects

Prisma compiles a relation filter on MongoDB into a $lookup over the whole record collection. That is the same construct that fails outright at scale in #1416 (fixed in #1423 by removing it), and it carries the same 100 MiB per-document ceiling. I've left a comment on the method saying so, and corrected the issue.

Also fixed: the record query now respects the caller's ability

querySubjectIdsWithRecords ignored ability entirely. The outer subject query already constrained the result, so this was not a data leak — but the inner query read records the caller had no permission to read, and was inconsistent with every other query in the service.

Tests

The three existing hasRecord tests are updated to the new call and still assert the same outcomes. Two added:

• the ids are grouped in the database, asserting groupBy is used and findMany is not — a guard against the distinct form coming back;
• the record query carries the caller's ability rather than being unconstrained.

Checks

• tsc --noEmit on apps/api — clean
• eslint src — clean
• prettier --check — clean
• vitest run (whole workspace) — 254 passed, 1 skipped, 1 failed

The one failure is instrument-records.service.spec.ts > upload > should create records with pending set. This branch is cut from main so it carries that pre-existing failure; #1422 fixes it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bfen9VQemZanJLXinJ6MMG

[Copilot]

Pull request overview

This PR improves the performance of the API’s “with records only” subject filter by moving subject-id deduplication into MongoDB via Prisma groupBy, avoiding Prisma’s client-side distinct behavior that transfers one row per record over the wire. It also updates the query to respect the caller’s ability when resolving subject ids from instrument records.

Changes:

• Replace findMany({ distinct: [...] }) with instrumentRecord.groupBy({ by: ['subjectId'] }) when resolving “subjects with records”.
• Apply accessibleQuery(ability, 'read', 'InstrumentRecord') to the record-side query for consistency and correctness.
• Update and extend unit tests to guard against regressions (ensuring groupBy is used and ability is threaded through).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
apps/api/src/subjects/subjects.service.ts	Switch subject-id resolution to Prisma groupBy and apply ability to the instrument record query.
apps/api/src/subjects/tests/subjects.service.spec.ts	Update existing hasRecord tests for groupBy and add regression tests for groupBy usage and ability propagation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.