Document where NodeNorm's data comes from, and serve the API docs that were being discarded

CLAUDE.md

@@ -71,7 +71,7 @@ Babel Compendia Files → loader.py → Redis (7 DBs) → FastAPI (server.py) 
 
 ### Key Modules
 
-- **`node_normalizer/server.py`** — FastAPI app with all REST endpoints. Uses lifespan events for Redis connection setup/teardown. Root path is `/1.3`.
+- **`node_normalizer/server.py`** — FastAPI app with all REST endpoints. Uses lifespan events for Redis connection setup/teardown. Root path is `/`.
 - **`node_normalizer/normalizer.py`** — Core logic: `get_normalized_nodes()`, `normalize_message()` (for TRAPI), and equivalent CURIE discovery. Traverses Biolink Model ancestors for semantic type expansion.
 - **`node_normalizer/loader/`** — the data loader: synchronous module-level functions (`load_all`, `load_compendium`, `load_conflation`, `merge_semantic_meta_data`) that read flat compendia files and populate Redis. Validates input against `resources/valid_data_format.json`. Batch size: 100,000. Invoked via the root `load.py`. `load_all` disables Redis periodic saves (`CONFIG SET save ""`) for the duration of the load — the backend DBs are write-once and persisted by a manual BGSAVE, so snapshotting during the load is wasted work. See `documentation/Loader.md`.
 - **`node_normalizer/config.py`** — shared repo-relative paths (`config.json`, `redis_config.yaml`, `resources/`) and `get_config()`. Used by both frontend and loader.

README.md

@@ -3,7 +3,7 @@
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.18489030.svg)](https://doi.org/10.5281/zenodo.18489030) [![arXiv](https://img.shields.io/badge/arXiv-2601.10008-b31b1b.svg)](https://arxiv.org/abs/2601.10008)
 
 <!-- Previous badges that aren't very useful:
-[![Publish to GitHub Packages](https://github.com/NCATSTranslator/NodeNormalization/actions/workflows/release.yml/badge.svg)](https://github.com/TranslatorSRI/NodeNormalization/actions/workflows/release.yml)
+[![Publish to GitHub Packages](https://github.com/NCATSTranslator/NodeNormalization/actions/workflows/release.yml/badge.svg)](https://github.com/NCATSTranslator/NodeNormalization/actions/workflows/release.yml)
 -->
 
 ## Introduction
@@ -14,11 +14,15 @@ Node normalization takes a CURIE, and returns:
 * All other known equivalent identifiers for the entity
 * Semantic types for the entity as defined by the [Biolink Model](https://biolink.github.io/biolink-model/)
 
-The data currently served by Node Normalization is created by the prototype project
-[Babel](https://github.com/NCATSTranslator/Babel), which attempts to find identifier equivalences,
-and makes sure that CURIE prefixes are BioLink Model compliant.  The NodeNormalization service, however,
-is independent of Babel and as improved identifier equivalence tools are developed, their results
-can be easily incorporated.
+The data served by Node Normalization is created by [Babel](https://github.com/NCATSTranslator/Babel),
+which finds identifier equivalences and makes sure that CURIE prefixes are Biolink Model compliant.
+NodeNorm itself is independent of Babel, and as improved identifier equivalence tools are developed
+their results can be easily incorporated.
+
+Babel decides which identifiers are equivalent, which one is preferred, and what a clique is called —
+so most "why did I get this answer?" questions are really questions about Babel.
+[Where NodeNorm's data comes from](documentation/Babel.md) explains the parts of Babel that show up
+in a NodeNorm response.
 
 To determine whether Node Normalization is likely to be useful, check /get_semantic_types, which lists the BioLink
 semantic types for which normalization has been attempted, and /get_curie_prefixes,
@@ -46,11 +50,12 @@ Install requirements
 ```
 ## Generating equivalence data
 
-The equivalence data can be generated by running [Babel](https://github.com/NCATSTranslator/Babel). An example of the contents of a compendia file is shown below:
-```
-    {"id": {"identifier": "PUBCHEM:50986940"}, "equivalent_identifiers": [{"identifier": "PUBCHEM:50986940"}, {"identifier": "INCHIKEY:CYMOSKLLKPIPCD-UHFFFAOYSA-N"}], "type": ["chemical_substance", "named_thing", "biological_entity", "molecular_entity"]}
-    {"id": {"identifier": "CHEMBL.COMPOUND:CHEMBL1546789", "label": "CHEMBL1546789"}, "equivalent_identifiers": [{"identifier": "CHEMBL.COMPOUND:CHEMBL1546789", "label": "CHEMBL1546789"}, {"identifier": "PUBCHEM:4879549"}, {"identifier": "INCHIKEY:FUIYIXDZTPMQEH-UHFFFAOYSA-N"}], "type": ["chemical_substance", "named_thing", "biological_entity", "molecular_entity"]}
-```
+The equivalence data can be generated by running [Babel](https://github.com/NCATSTranslator/Babel),
+or downloaded from a published Babel build. The loader reads Babel's compendium and conflation
+files; both formats are specified in
+[Where NodeNorm's data comes from](documentation/Babel.md), which links to Babel's format
+documentation.
+
 ## Creating and loading a Redis container with data 
 
 A running instance of Redis is needed to house the node normalization data. a Redis Docker container image can be downloaded from [Docker hub](https://hub.docker.com/_/redis). The Redis caonteriner can be started with thie following docker command: 
@@ -59,21 +64,13 @@ A running instance of Redis is needed to house the node normalization data. a Re
 ```
 Note that the dataset for Node normalization is quite large and 256Gb of memory and disk space should be made available to the Redis instance to insure proper loading of the complete compendia.
 ### Configuration
-Insure that the `./config.json` file is created and contains the parameters for the node normalization load specific to your environment. 
 
-The configuration parameters `compendium_directory` and `data_files` specify the location of the compendia files. An example of the files' contents  
-are listed below:
-```
-    {
-        "compendium_directory": "<path to files>",
-        "data_files": "anatomy.txt,BiologicalProcess.txt,cell.txt,cellular_component.txt,disease.txt,gene_compendium.txt,gene_family_compendium.txt,MolecularActivity.txt,pathways.txt,phenotypes.txt,taxon_compendium.txt",
-        "redis_host": "<Redis host server name>",
-        "redis_port": <Redis connection port>,
-        "redis_password": "<Redis password",
-        "test_mode": 1,
-        "debug_messages": 0
-    }
-```
+The loader reads `./config.json` for the location and list of Babel files to load, and
+`./redis_config.yaml` for the Redis connection details of each of the seven backend databases.
+Redis credentials live only in `redis_config.yaml`, never in `config.json`. See
+[Loader.md](documentation/Loader.md) for the keys each file supports, and
+[Redis.md](documentation/Redis.md) for the database layout.
+
 ### Loading of the Redis server with compendia data
 
 The load.py script reads the configuration file for load parameters and the loads the compendia data into the Redis instance. 
@@ -145,6 +142,13 @@ NodeNorm can be configured by setting environmental variables:
 * `SERVER_ROOT`: The server root (defaults to `/`)
 * `LOG_LEVEL`: The log level (defaults to `ERROR`)
 * `TRAPI_VERSION`: The TRAPI version this version of NodeNorm supports.
+* `BIOLINK_MODEL_TAG`: The tag, branch or commit of the [biolink-model](https://github.com/biolink/biolink-model)
+  repository used to expand semantic types into their ancestors (defaults to `master`). Reported by
+  `/status`. The loader pins its own Biolink version separately in `config.json` — see
+  [Loader.md](documentation/Loader.md).
+* `BABEL_VERSION` and `BABEL_VERSION_URL`: The name of the [Babel](https://github.com/NCATSTranslator/Babel)
+  build loaded into this instance's databases (e.g. `2025sep1`) and a URL describing it. Reported by
+  `/status`; see [Where NodeNorm's data comes from](documentation/Babel.md).
 * `MATURITY_VALUE`: How mature is this NameRes (defaults to `maturity`, e.g. `development`)
 * `LOCATION_VALUE`: Where is this NameRes setup (defaults to `location`, e.g. `RENCI`)
 * `EQ_BATCH_SIZE`: The size of the `get_eqids_and_types()` batch size (defaults to `2500`)

documentation/API.md

@@ -6,21 +6,34 @@ prefixes. The [NodeNorm FastAPI Documentation](https://nodenormalization-sri.ren
 information about the parameters for calling each endpoint, but this document will describe the
 intended function of each endpoint, suggestions for use and descriptions of the JSON documents returned.
 
+The identifiers, preferred names, types and information content values these endpoints return are all
+computed upstream by [Babel](https://github.com/NCATSTranslator/Babel) and loaded into NodeNorm as a
+fixed snapshot. If you are trying to work out *why* a particular clique looks the way it does, start
+with [Where NodeNorm's data comes from](Babel.md).
+
 ## Identifier/Node Normalization
 
 ### `/get_normalized_nodes`
 
 * Method: [GET](https://nodenormalization-sri.renci.org/docs#/default/get_normalized_node_handler_get_normalized_nodes_get)
   * Parameters:
     * `curie` (e.g. `curie=MESH:D014867&curie=NCIT:C34373`): The identifiers to normalize.
-    * `conflate` (e.g. `conflate=true`): Whether to apply GeneProtein conflation.
-    * `drug_chemical_conflate` (e.g. `drug_chemical_conflate=true`): Whether to apply DrugChemical conflation.
-    * `description` (e.g. `description=false`): Whether to include descriptions for nodes that have descriptions.
-    * `individual_types` (e.g. `individual_types=true`): When returning a conflated result, should Biolink types be
+    * `conflate` (default `true`): Whether to apply GeneProtein conflation.
+    * `drug_chemical_conflate` (default `true`): Whether to apply DrugChemical conflation.
+    * `description` (default `false`): Whether to include descriptions for nodes that have descriptions.
+    * `individual_types` (default `false`): When returning a conflated result, should Biolink types be
       returned for each individual identifier.
+    * `include_taxa` (default `true`): Whether to include the taxa associated with each equivalent identifier.
 * Method: [POST](https://nodenormalization-sri.renci.org/docs#/default/get_normalized_node_handler_post_get_normalized_nodes_post)
   * POST Body: A JSON object with the same parameters as the GET method, with a `curies` list instead of individual
     `curie` entries.
+  * **Note that `drug_chemical_conflate` defaults to `false` on POST but `true` on GET**
+    ([#398](https://github.com/NCATSTranslator/NodeNormalization/issues/398)). Until that is resolved,
+    set it explicitly if you use the POST method and care which conflations are applied.
+
+Conflation changes which identifier is preferred and merges several cliques into one flat list of
+equivalent identifiers; see [Where NodeNorm's data comes from](Babel.md#conflation) for what each
+conflation guarantees about ordering.
 
 Example output:
 
@@ -71,31 +84,44 @@ Example output:
 ```
 
 * Output values: the output is a dictionary with queried CURIEs as the keys and with JSON objects
-  as the values, containing the following keys:
+  as the values. **A CURIE that could not be normalized is present as a key with a `null` value**, rather than
+  being left out — check for `null` values rather than for missing keys. Each non-null value contains the
+  following keys:
   * `id`: A JSON object that provides the preferred identifier and labels for this clique.
     * `identifier`: The preferred CURIE for this clique. Every Biolink class includes a list of
       preferred prefixes (e.g.
       [valid ID prefixes for SmallMolecule](https://biolink.github.io/biolink-model/SmallMolecule/#valid-id-prefixes)),
       and this is used to choose the preferred CURIE for this clique.
-    * `label`: The preferred label for this clique. Note that this is not necessarily the label associated with the
-      preferred CURIE: for some classes (such as chemicals), we choose the best label in a different prefix order than
-      the Biolink Model preferred prefix order, based on which sources tend to have the best labels.
+    * `label`: The preferred label for this clique, as computed by Babel (its `preferred_name`). Note that this is
+      not necessarily the label associated with the preferred CURIE: for some classes (such as chemicals), the best
+      label is chosen in a different prefix order than the Biolink Model preferred prefix order, based on which
+      sources tend to have the best labels. See
+      [Where NodeNorm's data comes from](Babel.md#cliques-preferred-identifiers-and-labels).
+      (Older database loads that predate Babel supplying `preferred_name` fall back to a legacy label-selection
+      algorithm inside NodeNorm, which does not reproduce Babel's choice for conflated cliques.)
     * `description`: One of the descriptions for the identifiers within this clique.
   * `equivalent_identifiers`: a list of identifiers that are part of this clique given the conflation options.
     Each identifier includes an `identifier` (a CURIE), a `label` (which corresponds to the label of the CURIE as per
     its authoritative source), a `description` (currently only taken from UberGraph), and (if `individual_types` is set)
     the Biolink type of each identifier. This list is ordered in the Biolink Model's preferred prefix order for this class.
   * `descriptions`: a list of unique descriptions for the identifiers within this clique. This list is ordered in the same
-    order as `equivalent_identifiers`.
+    order as `equivalent_identifiers`. Only returned when `description` is set.
+  * `taxa`: every taxon associated with any identifier in this clique, as NCBITaxon CURIEs. Only returned when
+    `include_taxa` is set (the default), and omitted when no identifier in the clique has a known taxon. Individual
+    taxa are also returned on each entry in `equivalent_identifiers`.
   * `type`: The list of Biolink classes for this clique, starting with the most specific type (in this example,
     `biolink:SmallMolecule`), and ending with any mixins that include this class.
-  * `information_content`: the information content value between 0 and 100. This is calculated by retrieving the
+  * `information_content`: the information content value between 0 and 100. This is calculated by Babel, which retrieves the
     [normalized information content value](https://github.com/INCATools/ubergraph/?tab=readme-ov-file#graph-organization)
-    for each identifier that is present in UberGraph, and then calculating the lowest information content value of
-    any identifier in this clique for which UberGraph has an identifier value. According to UberGraph's documentation,
+    for each identifier that is present in UberGraph, and then stores the lowest information content value of
+    any identifier in this clique for which UberGraph has a value. NodeNorm returns that stored value unchanged.
+    According to UberGraph's documentation,
     the normalized information content value is "Precomputed information content score for each ontology class, based
     on the count of terms related to a given term via rdfs:subClassOf or any existential relation. The scores are
-    xsd:decimal values scaled from 0 to 100 (e.g., a very specific term with no subclasses)."
+    xsd:decimal values scaled from 0 to 100 (e.g., a very specific term with no subclasses)." A value of 0.0 therefore
+    means a broad, high-level concept and 100.0 a very specific one. Not every clique has one, and the field is
+    omitted when it is unknown. See [Where NodeNorm's data comes from](Babel.md#information-content) — including the
+    caveat that information content from Babel builds before `2025sep1` is unreliable.
   * Internally, conflation is represented as sets of cliques that should be combined when that conflation is turned on.
     This means that a conflated clique will be represented by a single list of equivalent identifiers, starting with the
     equivalent identifiers from the first clique, followed by the equivalent identifiers from the second clique, and so
@@ -173,8 +199,15 @@ Example output:
 ```json
 {
   "status": "running",
+  "version": "2.5.1",
+  "backend": "redis",
   "babel_version": "2025mar31",
-  "babel_version_url": "https://github.com/NCATSTranslator/Babel/blob/master/releases/2025mar31.md",
+  "babel_version_url": "https://github.com/NCATSTranslator/Babel/blob/main/releases/2025mar31.md",
+  "biolink_model": {
+    "tag": "v4.2.6-rc2",
+    "url": "https://github.com/biolink/biolink-model/tree/v4.2.6-rc2",
+    "download_url": "https://raw.githubusercontent.com/biolink/biolink-model/v4.2.6-rc2/biolink-model.yaml"
+  },
   "databases": {
     "eq_id_to_id_db": {
       "dbname": "id-id",
@@ -190,11 +223,22 @@ Example output:
 Output values:
 
 * `status` (example: `running`): Whether or not the service is running.
+* `version` (example: `2.5.1`): The version of NodeNorm itself.
+* `backend` (example: `redis`): The storage backend in use.
 * `babel_version` (example: `2025mar31`): The version of [Babel](https://github.com/NCATSTranslator/Babel) used to generate
   the cliques being presented. These are usually date-based versions indicating approximately when the Babel build was
-  completed.
-* `babel_version_url` (example: https://github.com/NCATSTranslator/Babel/blob/master/releases/2025mar31.md): A URL you
+  completed. Since the backend databases are written once and never updated in place, this identifies the data
+  this instance will return until an operator loads a newer build — record it if you need reproducible results.
+* `babel_version_url` (example: https://github.com/NCATSTranslator/Babel/blob/main/releases/2025mar31.md): A URL you
   can use to learn more about this version of Babel, and how it differs from previous and future versions.
+* `biolink_model`: The version of the [Biolink Model](https://github.com/biolink/biolink-model) this instance uses to
+  expand a clique's most specific type into the full list of ancestors returned in `type`. Set with the
+  `BIOLINK_MODEL_TAG` environment variable.
+  * `tag`: The tag, branch or commit name (example: `v4.2.6-rc2`).
+  * `url`: The biolink-model repository at that tag.
+  * `download_url`: The `biolink-model.yaml` actually loaded.
+  * Note that the loader pins its own Biolink version separately (`biolink_version` in `config.json`), so that
+    ancestors are computed against the model Babel built the data with. The two can differ.
 * `databases`: A dictionary of Redis key-value databases used by this NodeNorm instance (currently: 7). Each database
   uses the internal name of this database as its key, along with the following information:
   * `dbname`: A second name used for this database.
@@ -307,7 +351,14 @@ Example output:
 
 ## TRAPI Normalization (deprecated)
 
-These methods 
+These two endpoints normalize an entire TRAPI message rather than a list of CURIEs. Both are
+deprecated ([PR #323](https://github.com/NCATSTranslator/NodeNormalization/pull/323)): they are no
+longer actively maintained, and will be removed once the Workflow Runner stops using them. New
+callers should extract the CURIEs they care about and use `/get_normalized_nodes` instead.
+
+Note that `/query` also attaches each clique's information content to the normalized node as a TRAPI
+attribute (`attribute_type_id: biolink:has_numeric_value`, `original_attribute_name:
+information_content`), rounded to one decimal place.
 
 ### `/query`