feat(backend): adopt climate_ref.results.Reader for metric values

[lewisjared]

Summary

Migrates the backend's two /values endpoints off hand-rolled SQLAlchemy queries and onto the new climate_ref.results.Reader read-layer facade — the consumer that read layer was explicitly designed for. This is the first, keystone slice of a phased adoption (plan tracked separately); it targets the biggest duplication: metric-value filtering, source-id-aware IQR outlier detection, facet collection, and the load-all-then-paginate rule now all live upstream.

• GET /executions/{group_id}/values and GET /diagnostics/{provider}/{diagnostic}/values now query app_context.reader.values.{scalar,series}_values(...) via MetricValueFilter + OutlierPolicy.
• New get_reader dependency on AppContext (additive; the raw session dependency is untouched and still used by the not-yet-migrated routes).
• New core/reader_values.py for CSV rendering from reader collections and CV-dimension query-param gating; MetricValueCollection.build_{scalar,series}_from_reader adapters in models.py.

Response shapes are unchanged. promoted_only=False and include_retracted=True are set explicitly to preserve the previous unfiltered behaviour rather than adopt the reader's stricter defaults — so this is behaviour-preserving, which the existing route tests confirm.

Dependency / environment changes (please review)

• Requires climate-ref >= 0.16.0 (>=0.16.0,<0.17). v0.16.0 is tagged and GitHub-released but is not yet on PyPI (PyPI currently tops out at 0.14.6 — 0.14.7/0.15.0/0.16.0 all have green Release runs but no published artifact). It is therefore sourced from the v0.16.0 git tag via [tool.uv.sources]; the version constraint is already correct, so that source line can simply be deleted once 0.16.0 lands on PyPI.
• Python floor moves 3.11 → 3.12, because climate-ref 0.16 requires >=3.12.
• The decimated test-fixture DB is migrated to the current schema (it was 4 alembic revisions behind; diagnostic.promoted_version was missing). This was done with climate-ref's own migrations — the same path a deployed 0.13-era database will take. Deployed REF databases must be alembic-migrated to head before a reader-based backend can read them.

Testing

• Full backend suite green: 185 passed, 1 xfailed (unchanged from the pre-migration baseline), against climate-ref v0.16.0.
• mypy (strict) and ruff clean.

Follow-ups (later phases)

• Migrate the executions/diagnostics list routes, datasets, and artifacts endpoints.
• Delete the now-dead core/outliers.py and the query half of core/metric_values.py once nothing references them.
• Harden the scalar adapter against null-valued scalars (float(item.value) would raise; none in the current fixture).

Summary by CodeRabbit

• New Features

  • Added consistent metric-value retrieval for diagnostic and execution endpoints.
  • Added CSV downloads for scalar and series values, including dimensions, metadata and outlier details where applicable.
  • Added support for dimension filters, ID inclusion/exclusion, pagination and outlier detection.
  • Preserved existing API response formats while improving value and series handling.

• Refactor

  • Centralised metric-value reading and response construction for more consistent results.

[netlify]

✅ Deploy Preview for climate-ref canceled.

Name	Link
🔨 Latest commit	e7f862c
🔍 Latest deploy log	https://app.netlify.com/projects/climate-ref/deploys/6a54db5380e8230008b36270

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5beab3ed-ab74-4975-b489-ed6680db9339

📥 Commits

Reviewing files that changed from the base of the PR and between e9a6ca3 and e7f862c.

📒 Files selected for processing (4)

• backend/src/ref_backend/api/routes/diagnostics.py
• backend/src/ref_backend/api/routes/executions.py
• backend/src/ref_backend/models.py
• backend/tests/test_core/test_metric_values.py

🚧 Files skipped from review as they are similar to previous changes (3)

• backend/src/ref_backend/api/routes/diagnostics.py
• backend/src/ref_backend/api/routes/executions.py
• backend/src/ref_backend/models.py

---

📝 Walkthrough

Walkthrough

The change wires a shared Reader into application context, adds reader-based filtering, CSV generation, and response builders, and refactors diagnostic and execution metric-value endpoints to use the reader workflow.

Changes

Reader-backed metric values

Layer / File(s)	Summary
Reader dependency wiring backend/src/ref_backend/api/deps.py	Constructs a Reader from the database and results path, then exposes it through AppContext.
Reader value helpers and response builders backend/src/ref_backend/core/reader_values.py, backend/src/ref_backend/models.py, backend/tests/test_core/test_metric_values.py	Adds dimension parsing, scalar/series CSV generation, reader-to-response mapping, and tests for kinds, outliers, attributes, and references.
Diagnostics metric-value retrieval backend/src/ref_backend/api/routes/diagnostics.py	Delegates filtering, outlier handling, retrieval, pagination, CSV generation, and JSON construction to reader-layer APIs.
Execution metric-value retrieval backend/src/ref_backend/api/routes/executions.py, changelog/39.trivial.md	Applies the reader workflow to execution-scoped values and records the data-access change in the changelog.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant MetricValueRoute
  participant Reader
  participant MetricValueCollection

  Client->>MetricValueRoute: request metric values
  MetricValueRoute->>Reader: apply MetricValueFilter and retrieve values
  Reader-->>MetricValueRoute: scalar or series collection
  MetricValueRoute->>MetricValueCollection: build API response
  MetricValueCollection-->>Client: JSON or CSV response

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: switching backend metric-value endpoints to climate_ref.results.Reader.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/adopt-results-reader

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

backend/src/ref_backend/api/routes/executions.py (1)

341-375: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider extracting the shared scalar/series/CSV branching.

This block is essentially identical to diagnostics.py's list_metric_values (filter build aside). A shared helper taking a MetricValueFilter plus the value-type/format/pagination/outlier args would remove the duplication and keep the two endpoints' response behaviour in lock-step as the reader layer evolves.

backend/src/ref_backend/core/reader_values.py (1)

37-62: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

CSV generators buffer the entire output in memory before yielding.

Both generate_csv closures write all rows into a single io.StringIO and yield once at the end. Since CSV export deliberately returns all results without pagination, the full collection is already in memory, and the buffer roughly doubles that footprint. For large diagnostics this can cause significant memory pressure on the worker.

Consider yielding rows (or batches of rows) as they are written so StreamingResponse can flush incrementally:

♻️ Proposed refactor: yield per-row instead of buffering

     def generate_csv() -> Generator[str]:
         output = io.StringIO()
         writer = csv.writer(output)

         items = collection.items
         if not items:
             yield ""
             return

         dimensions = sorted(items[0].dimensions.keys())
         header = [*dimensions, "value", "type"]
         if detection_ran:
             header.extend(["is_outlier", "verification_status"])
         writer.writerow(header)
+        yield output.getvalue()
+        output.seek(0)
+        output.truncate(0)

         for item in items:
             row = [item.dimensions.get(d) for d in dimensions] + [
                 sanitize_float_value(item.value),
                 "scalar",
             ]
             if detection_ran:
                 row.extend([item.is_outlier, item.verification_status])
             writer.writerow(row)
+            yield output.getvalue()
+            output.seek(0)
+            output.truncate(0)
-
-        output.seek(0)
-        yield output.read()

The same pattern applies to generate_csv_response_series (lines 87-112). For fewer, larger chunks, accumulate N rows before yielding.

Also applies to: 87-112

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 62c6e7e3-e968-4cd4-a930-25a867d6132a

📥 Commits

Reviewing files that changed from the base of the PR and between e501f1d and e9a6ca3.

⛔ Files ignored due to path filters (2)

• backend/tests/test-data/tests.integration.test_cmip7_aft/test_solve_cmip7_aft/db/climate_ref.db is excluded by !**/*.db
• backend/uv.lock is excluded by !**/*.lock

📒 Files selected for processing (8)

• backend/pyproject.toml
• backend/src/ref_backend/api/deps.py
• backend/src/ref_backend/api/routes/diagnostics.py
• backend/src/ref_backend/api/routes/executions.py
• backend/src/ref_backend/core/reader_values.py
• backend/src/ref_backend/models.py
• changelog/39.breaking.md
• changelog/39.trivial.md