Metadata Translation Layers for API Endpoints

[grantfitzsimmons]

Problem Statement

Today, when an API consumer fetches a resource from /api/specify/<model>/<id>/, the response JSON uses internal Django model field names as keys:

{
  "id": 1001,
  "catalogNumber": "CAS-12345",
  "fieldNumber": "GRF-2023-001",
  "collectionObjectAttribute": "/api/specify/collectionobjectattribute/500/",
  "resource_uri": "/api/specify/collectionobject/1001/"
}

To display human-readable field labels (e.g., "Catalog Number" in English, "Numéro de catalogue" in French, user-assigned overrides), the consumer must:

1. Fetch /context/schema_localization.json?lang=fr to get the localized schema metadata
2. Look up each field name in the nested tables → collectionobject → items → catalogNumber → name path
3. Manually map every key in every API response

This two-step workflow is error-prone, adds latency, and requires every consumer to reimplement the same mapping logic.

The goal is to allow API consumers to receive localized field labels directly in API data responses, eliminating the need for a separate schema localization fetch. This must support all languages currently available in Splocaleitemstr (user-assigned captions), with sensible fallback to English and then to the internal field name. Full backward compatibility must be maintained — existing API responses must remain unchanged unless the consumer explicitly opts in. The feature must work for both read (GET single resource, GET collection list) and write (POST, PUT) endpoints, support nested/inlined dependent objects, minimize performance impact, and respect collection-level and discipline-level schema customization.

---

Out of Scope

• Localizing values (e.g., translating picklist item text). This feature is only about field labels (captions/names).
• Changing the internal data model or database schema
• Modifying the serialized keys to be localized (keys remain internal field names for programmatic stability)
• Providing translations for dynamic/computed fields whose names are not in the schema configuration tables
• Adding localization to attachment, report, or other non-data endpoints

---

Opt-In Mechanism

The API consumer SHALL opt in to receiving localized field labels by including a query parameter fieldLabels=<lang> on any data endpoint request.

• Example: GET /api/specify/collectionobject/1001/?fieldLabels=fr
• The <lang> value SHALL use the same ll[-cc] format as the existing schema_localization endpoint (e.g., en, en-us, fr, uk-ua).

When fieldLabels is absent, the response SHALL be identical to current behavior (no localization metadata included).

When fieldLabels is present but the requested language has no translations, the system SHALL fall back following this chain:

1. Requested language-country (exact match, e.g., en-us)
2. Requested language with no country (e.g., en)
3. en with no country (English fallback)
4. The internal field name as stored in splocalecontaineritem.name (the untranslated name)

---

Response Format for Single Resources

For single-resource responses (GET /api/specify/<model>/<id>/), the response SHALL include a top-level _fieldLabels object when fieldLabels is specified:

{
  "id": 1001,
  "catalogNumber": "CAS-12345",
  "fieldNumber": "GRF-2023-001",
  "collectionObjectAttribute": "/api/specify/collectionobjectattribute/500/",
  "resource_uri": "/api/specify/collectionobject/1001/",
  "_fieldLabels": {
    "id": "ID",
    "catalogNumber": "Numéro de catalogue",
    "fieldNumber": "Numéro de terrain",
    "collectionObjectAttribute": "Attribut de l'objet de collection"
  }
}

The _fieldLabels key SHALL be a flat object mapping each top-level field name to its localized caption (the name from splocalecontaineritem plus its Splocaleitemstr translation for the requested language). For fields that reference related objects (foreign keys shown as URIs), the label SHALL be the localized table/relationship name from the schema container for the parent table. The resource_uri and _fieldLabels keys SHALL NOT themselves have entries in _fieldLabels.

---

Response Format for Collections

For collection responses (GET /api/specify/<model>/), the response SHALL include _fieldLabels at the same level as objects (the metadata object, not inside each individual object):

{
  "objects": [
    {
      "id": 1001,
      "catalogNumber": "CAS-12345",
      ...
    },
    ...
  ],
  "meta": {
    "limit": 20,
    "offset": 0,
    "total_count": 156
  },
  "_fieldLabels": {
    "id": "ID",
    "catalogNumber": "Numéro de catalogue",
    ...
  }
}

The _fieldLabels SHALL NOT be repeated inside each object in the objects array — it SHALL appear once at the top level of the response.

---

Nested / Inlined Objects

When a dependent (inlined) object is included in the response, its fields SHALL have their own _fieldLabels block nested within the dependent object:

{
  "id": 1001,
  "catalogNumber": "CAS-12345",
  "collectionObjectAttribute": {
    "id": 500,
    "text1": "Some note",
    "integer1": 3,
    "resource_uri": "/api/specify/collectionobjectattribute/500/",
    "_fieldLabels": {
      "id": "ID",
      "text1": "Remarque",
      "integer1": "Nombre d'éléments"
    }
  },
  "resource_uri": "/api/specify/collectionobject/1001/",
  "_fieldLabels": {
    "id": "ID",
    "catalogNumber": "Numéro de catalogue",
    "collectionObjectAttribute": "Attribut de l'objet de collection"
  }
}

The _fieldLabels SHALL be included at every nesting level where objects are inlined (i.e., where the full object data is present, not just a URI reference).

---

Write Operations

POST and PUT responses that return the created/updated object SHALL include _fieldLabels in the response body when the fieldLabels query parameter is present on the request. The fieldLabels parameter SHALL be accepted but ignored for the purposes of request parsing — it only affects the response format.

---

Language Discovery

The existing /context/schema/language/ endpoint SHALL continue to serve as the discovery mechanism for available schema localization languages. If a consumer requests a language that exists in the Django LANGUAGES setting but has no entries in Splocalecontaineritem for the collection's discipline, the fallback chain SHALL apply, ultimately falling back to the internal (untranslated) field name.

---

Backend Implementation

Localization data SHALL be sourced from the same Splocalecontainer / Splocalecontaineritem / Splocaleitemstr tables used by schema_localization.py today. The implementation SHALL reuse the existing get_schema_localization() function or its core logic to avoid duplicating the localization query logic.

Field label lookup SHALL use a precomputed dictionary (keyed by table_name→field_name) built once per request, not per object, to avoid N+1 database queries. The localization lookup SHALL be implemented as a post-processing step on the serialized data dictionary, not by modifying Django model serialization internals directly. The feature SHALL be opt-in via query parameter — no new URL routes are required.

Caching headers SHALL include Vary: Accept-Language or appropriate cache key variation so that CDN/proxy caches do not serve localized responses for the wrong language.

---

Performance Constraints

For a collection response of 100 objects, the overhead of adding _fieldLabels SHALL add no more than 50ms of server processing time compared to the same request without fieldLabels. The localization data for a single discipline SHALL be fetched in at most 2 database queries (containers + items), regardless of the number of objects in the response.

---

Compatibility

The fieldLabels parameter SHALL be documented in the OpenAPI schema for all data endpoints. The internal field names used as JSON keys SHALL NOT change — this is a purely additive feature. All existing tests SHALL continue to pass without modification. The _fieldLabels key SHALL be prefixed with underscore to indicate it is metadata, following the existing convention (resource_uri).

---

Security and Access Control

Field labels SHALL respect the same collection access controls as the data — users can only see labels for fields they have permission to read. Hidden fields (ishidden=true in schema config) SHALL still have their labels returned in _fieldLabels if the user can see the field value — _fieldLabels reflects what is in the response, not what is configured as hidden. No new database write operations are introduced — this feature is read-only for the localization tables.

---

Error Handling

If the fieldLabels parameter has an invalid format (not matching ll[-cc]), the endpoint SHALL return a 400 Bad Request with a descriptive error message. If the schema localization data cannot be fetched (e.g., database error), the endpoint SHALL still return the data response but omit _fieldLabels and log a warning — data retrieval MUST NOT fail because of localization failures. If a particular field has no localization entry (field exists in Django model but not in splocalecontaineritem), _fieldLabels SHALL use the internal field name as the label with no error.

---

Example Workflows

Current (Two Requests):

Request 1: GET /context/schema_localization.json?lang=fr
Request 2: GET /api/specify/collectionobject/1001/
# Consumer manually maps field names using Request 1 data

New (One Request):

Request: GET /api/specify/collectionobject/1001/?fieldLabels=fr
# Response includes _fieldLabels — consumer uses it directly

Collection Browsing:

GET /api/specify/collectionobject/?fieldLabels=fr&limit=50

Response includes _fieldLabels once at the top level — the consuming application can label all columns in a table view without per-object overhead.

---

Acceptance Criteria

• AC: GET /api/specify/collectionobject/<id>/ without fieldLabels returns the same JSON structure as before this feature was implemented
• AC: GET /api/specify/collectionobject/<id>/?fieldLabels=fr returns a response with _fieldLabels containing French field labels
• AC: GET /api/specify/collectionobject/?fieldLabels=en&limit=10 returns _fieldLabels at the top level (not inside each object)
• AC: When a dependent object is inlined (e.g., CollectionObjectAttribute), its fields have their own nested _fieldLabels
• AC: GET /api/specify/collectionobject/<id>/?fieldLabels=zz (nonexistent language) falls back to English labels
• AC: fieldLabels does not affect POST/PUT request parsing — only the response body
• AC: The existing test suite passes without modification
• AC: Response time for a single resource with fieldLabels is within 50ms of the same request without fieldLabels
• AC: The feature is discoverable via the OpenAPI documentation at /context/api_endpoints.json

---

Dependencies

• specifyweb/backend/context/schema_localization.py — get_schema_localization() function
• specifyweb/specify/api/serializers.py — _obj_to_data() and related serialization functions
• specifyweb/specify/api/crud.py — GET resource/collection handlers
• specifyweb/specify/models.py — Splocalecontainer, Splocalecontaineritem, Splocaleitemstr Django models
• specifyweb/backend/context/views.py — OpenAPI schema generation for endpoints

---

Open Questions

• Should _fieldLabels be included in PATCH responses? PATCH typically returns 204 No Content or the updated fields only. Recommendation: Include _fieldLabels if a response body is returned.
• Should we provide a fieldLabels=* option to request labels in the user's currently authenticated language? This would avoid the client needing to track language separately. The server knows the authenticated user's language from the session/cookie.
• Should _fieldLabels include the desc (description) field from the schema localization in addition to name? This would require a nested object like "_fieldLabels": {"catalogNumber": {"name": "Numéro de catalogue", "desc": "Le numéro de catalogue unique"}} instead of a flat string mapping. This adds complexity but may be valuable for UI tooltips.
• Should we support fieldLabels on the /api/specify_rows/ query endpoint? This endpoint already returns column names — it may benefit significantly from localization support.

---

Implementation Notes

• The _fieldLabels construction should happen as a post-processing step after the full response dictionary is built, keeping it cleanly separated from the core serialization logic.
• Consider implementing a utility function annotate_field_labels(data: dict, model_name: str, language: str, country: str | None) -> dict that can be called from the view/crud layer, or in specifyweb/backend/context/schema_localization.py.
• Cache the per-request localization dictionary on the request object to avoid rebuilding it for nested/inlined objects within the same request.