diff --git a/CLAUDE.md b/CLAUDE.md index e213c6a8..8d7f5cc5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,7 +71,7 @@ Babel Compendia Files → loader.py → Redis (7 DBs) → FastAPI (server.py) ### Key Modules -- **`node_normalizer/server.py`** — FastAPI app with all REST endpoints. Uses lifespan events for Redis connection setup/teardown. Root path is `/1.3`. +- **`node_normalizer/server.py`** — FastAPI app with all REST endpoints. Uses lifespan events for Redis connection setup/teardown. Root path is `/`. - **`node_normalizer/normalizer.py`** — Core logic: `get_normalized_nodes()`, `normalize_message()` (for TRAPI), and equivalent CURIE discovery. Traverses Biolink Model ancestors for semantic type expansion. - **`node_normalizer/loader/`** — the data loader: synchronous module-level functions (`load_all`, `load_compendium`, `load_conflation`, `merge_semantic_meta_data`) that read flat compendia files and populate Redis. Validates input against `resources/valid_data_format.json`. Batch size: 100,000. Invoked via the root `load.py`. `load_all` disables Redis periodic saves (`CONFIG SET save ""`) for the duration of the load — the backend DBs are write-once and persisted by a manual BGSAVE, so snapshotting during the load is wasted work. See `documentation/Loader.md`. - **`node_normalizer/config.py`** — shared repo-relative paths (`config.json`, `redis_config.yaml`, `resources/`) and `get_config()`. Used by both frontend and loader. diff --git a/README.md b/README.md index 4f7637b0..f2e3d4b6 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.18489030.svg)](https://doi.org/10.5281/zenodo.18489030) [![arXiv](https://img.shields.io/badge/arXiv-2601.10008-b31b1b.svg)](https://arxiv.org/abs/2601.10008) ## Introduction @@ -14,11 +14,15 @@ Node normalization takes a CURIE, and returns: * All other known equivalent identifiers for the entity * Semantic types for the entity as defined by the [Biolink Model](https://biolink.github.io/biolink-model/) -The data currently served by Node Normalization is created by the prototype project -[Babel](https://github.com/NCATSTranslator/Babel), which attempts to find identifier equivalences, -and makes sure that CURIE prefixes are BioLink Model compliant. The NodeNormalization service, however, -is independent of Babel and as improved identifier equivalence tools are developed, their results -can be easily incorporated. +The data served by Node Normalization is created by [Babel](https://github.com/NCATSTranslator/Babel), +which finds identifier equivalences and makes sure that CURIE prefixes are Biolink Model compliant. +NodeNorm itself is independent of Babel, and as improved identifier equivalence tools are developed +their results can be easily incorporated. + +Babel decides which identifiers are equivalent, which one is preferred, and what a clique is called — +so most "why did I get this answer?" questions are really questions about Babel. +[Where NodeNorm's data comes from](documentation/Babel.md) explains the parts of Babel that show up +in a NodeNorm response. To determine whether Node Normalization is likely to be useful, check /get_semantic_types, which lists the BioLink semantic types for which normalization has been attempted, and /get_curie_prefixes, @@ -46,11 +50,12 @@ Install requirements ``` ## Generating equivalence data -The equivalence data can be generated by running [Babel](https://github.com/NCATSTranslator/Babel). An example of the contents of a compendia file is shown below: -``` - {"id": {"identifier": "PUBCHEM:50986940"}, "equivalent_identifiers": [{"identifier": "PUBCHEM:50986940"}, {"identifier": "INCHIKEY:CYMOSKLLKPIPCD-UHFFFAOYSA-N"}], "type": ["chemical_substance", "named_thing", "biological_entity", "molecular_entity"]} - {"id": {"identifier": "CHEMBL.COMPOUND:CHEMBL1546789", "label": "CHEMBL1546789"}, "equivalent_identifiers": [{"identifier": "CHEMBL.COMPOUND:CHEMBL1546789", "label": "CHEMBL1546789"}, {"identifier": "PUBCHEM:4879549"}, {"identifier": "INCHIKEY:FUIYIXDZTPMQEH-UHFFFAOYSA-N"}], "type": ["chemical_substance", "named_thing", "biological_entity", "molecular_entity"]} -``` +The equivalence data can be generated by running [Babel](https://github.com/NCATSTranslator/Babel), +or downloaded from a published Babel build. The loader reads Babel's compendium and conflation +files; both formats are specified in +[Where NodeNorm's data comes from](documentation/Babel.md), which links to Babel's format +documentation. + ## Creating and loading a Redis container with data A running instance of Redis is needed to house the node normalization data. a Redis Docker container image can be downloaded from [Docker hub](https://hub.docker.com/_/redis). The Redis caonteriner can be started with thie following docker command: @@ -59,21 +64,13 @@ A running instance of Redis is needed to house the node normalization data. a Re ``` Note that the dataset for Node normalization is quite large and 256Gb of memory and disk space should be made available to the Redis instance to insure proper loading of the complete compendia. ### Configuration -Insure that the `./config.json` file is created and contains the parameters for the node normalization load specific to your environment. -The configuration parameters `compendium_directory` and `data_files` specify the location of the compendia files. An example of the files' contents -are listed below: -``` - { - "compendium_directory": "", - "data_files": "anatomy.txt,BiologicalProcess.txt,cell.txt,cellular_component.txt,disease.txt,gene_compendium.txt,gene_family_compendium.txt,MolecularActivity.txt,pathways.txt,phenotypes.txt,taxon_compendium.txt", - "redis_host": "", - "redis_port": , - "redis_password": " Dict[str, str]: api_docs = load(apd_file, Loader=SafeLoader) if app.openapi_schema: - return app.openapi_schema() + return app.openapi_schema open_api_schema = get_openapi( title=api_docs['info']['title'], @@ -64,6 +68,9 @@ def construct_open_api_schema(app) -> Dict[str, str]: if 'description' in api_docs['info']: open_api_schema['info']['description'] = api_docs['info']['description'] + if 'license' in api_docs['info']: + open_api_schema['info']['license'] = api_docs['info']['license'] + # adds support to override server root path server_root = os.environ.get('SERVER_ROOT', '/') # make sure not to add double slash at the end. diff --git a/node_normalizer/examples.py b/node_normalizer/examples.py index aba64c99..48bfba28 100644 --- a/node_normalizer/examples.py +++ b/node_normalizer/examples.py @@ -138,3 +138,34 @@ ], } } + + +# An example response from /get_normalized_nodes, used to document the endpoint. Shows a CURIE that +# normalizes (MESH:D014867 -> CHEBI:15377) and one that does not (RUBBISH:1234 -> null). +EXAMPLE_NORMALIZED_NODES = { + "MESH:D014867": { + "id": { + "identifier": "CHEBI:15377", + "label": "Water", + }, + "equivalent_identifiers": [ + {"identifier": "CHEBI:15377", "label": "water", "type": "biolink:SmallMolecule"}, + {"identifier": "UNII:059QF0KO0R", "label": "WATER", "type": "biolink:SmallMolecule"}, + {"identifier": "PUBCHEM.COMPOUND:962", "label": "Water", "type": "biolink:SmallMolecule"}, + {"identifier": "MESH:D014867", "label": "Water", "type": "biolink:SmallMolecule"}, + ], + "type": [ + "biolink:SmallMolecule", + "biolink:MolecularEntity", + "biolink:ChemicalEntity", + "biolink:PhysicalEssence", + "biolink:ChemicalOrDrugOrTreatment", + "biolink:ChemicalEntityOrGeneOrGeneProduct", + "biolink:ChemicalEntityOrProteinOrPolypeptide", + "biolink:NamedThing", + "biolink:PhysicalEssenceOrOccurrent", + ], + "information_content": 47.7, + }, + "RUBBISH:1234": None, +} diff --git a/node_normalizer/model/__init__.py b/node_normalizer/model/__init__.py index 457a5369..5f9ba4b9 100644 --- a/node_normalizer/model/__init__.py +++ b/node_normalizer/model/__init__.py @@ -3,4 +3,12 @@ """ from .input import CurieList, SemanticTypesInput, SetIDQuery -from .response import CuriePivot, SemanticTypes, ConflationList, SetIDResponse +from .response import ( + CuriePivot, + SemanticTypes, + ConflationList, + SetIDResponse, + EquivalentIdentifier, + NormalizedNodeId, + NormalizedNode, +) diff --git a/node_normalizer/model/input.py b/node_normalizer/model/input.py index 6a816ec9..3fda9f90 100644 --- a/node_normalizer/model/input.py +++ b/node_normalizer/model/input.py @@ -12,33 +12,40 @@ class CurieList(BaseModel): curies: List[str] = Field( ..., # Ellipsis means field is required - title='List of CURIEs to normalize', + description="The CURIEs to normalize. Any CURIE that cannot be normalized is returned with a null value " + "rather than being left out of the response.", min_items=1 ) - conflate:bool = Field( + conflate: bool = Field( True, - title="Whether to apply gene/protein conflation" + description="Whether to apply GeneProtein conflation, which merges a gene with the protein it encodes. " + "The gene always comes first in the combined clique." ) description: bool = Field( False, - title="Whether to return CURIE descriptions when possible" + description="Whether to return CURIE descriptions when possible. Descriptions currently come only from " + "UberGraph, so most identifiers have none." ) drug_chemical_conflate: bool = Field( False, - title="Whether to apply drug/chemical conflation" + description="Whether to apply DrugChemical conflation, which merges a drug with its active ingredient. " + "The active ingredient comes before any formulations in the combined clique. " + "Note that this defaults to false here but true on the GET method of this endpoint " + "(https://github.com/NCATSTranslator/NodeNormalization/issues/398)." ) individual_types: bool = Field( False, - title="Whether to return individual types for equivalent identifiers" + description="Whether to return the Biolink type of each equivalent identifier. Useful for telling apart " + "the members of a conflated clique, which are otherwise returned as one flat list." ) include_taxa: bool = Field( True, - title="Whether to return taxa for equivalent identifiers" + description="Whether to return the taxa associated with each equivalent identifier, as NCBITaxon CURIEs." ) class Config: @@ -48,6 +55,8 @@ class Config: "conflate": True, "description": False, "drug_chemical_conflate": True, + "individual_types": False, + "include_taxa": True, } } @@ -57,7 +66,7 @@ class SemanticTypesInput(BaseModel): semantic_types: List[str] = Field( ..., # required field - title='list of semantic types', + description="The Biolink semantic types to report on. Pass an empty list for every semantic type.", ) class Config: @@ -79,6 +88,6 @@ class SetIDQuery(BaseModel): conflations: List[str] = Field( [], - description="Set of conflations to apply", + description="Set of conflations to apply. See /get_allowed_conflations for the valid values.", example=["GeneProtein", "DrugChemical"], ) diff --git a/node_normalizer/model/response.py b/node_normalizer/model/response.py index 60d89c50..6206fffa 100644 --- a/node_normalizer/model/response.py +++ b/node_normalizer/model/response.py @@ -1,8 +1,15 @@ """ API Response Models not described in reasoner-pydantic + +Note that the normalization models below (`EquivalentIdentifier`, `NormalizedNodeId`, +`NormalizedNode`) are used to *document* the response of /get_normalized_nodes, via the +`responses=` argument of the route decorators, rather than as a `response_model=`. They are +deliberately not enforced: `create_node()` builds these dictionaries by hand and omits keys that +don't apply, and making FastAPI filter its output through a model risks silently dropping a field +the model forgot. Keep them in sync with `create_node()` in `normalizer.py`. """ -from pydantic import BaseModel +from pydantic import BaseModel, Field from typing import Dict, List, Optional @@ -25,20 +32,141 @@ class Config: class CuriePivot(BaseModel): - curie_prefix: Dict[str, str] + curie_prefix: Dict[str, str] = Field( + ..., + description="A mapping from CURIE prefix to the number of times it appears in an equivalent identifier " + "for this semantic type. Note that the counts are strings, not numbers.", + ) + + class Config: + schema_extra = { + "example": { + "curie_prefix": { + "PUBCHEM.COMPOUND": "119397095", + "INCHIKEY": "115661650", + "CHEBI": "200507", + } + } + } class ConflationList(BaseModel): - conflations: List + conflations: List[str] = Field( + ..., + description="The conflations supported by this NodeNorm instance.", + ) + + class Config: + schema_extra = {"example": {"conflations": ["GeneProtein", "DrugChemical"]}} + + +class EquivalentIdentifier(BaseModel): + """A single identifier belonging to a normalized clique.""" + + identifier: str = Field(..., description="The CURIE for this identifier.") + label: Optional[str] = Field( + None, + description="The label for this identifier as given by its authoritative source. Omitted if that source " + "provides no label.", + ) + description: Optional[str] = Field( + None, + description="A description of this identifier, currently always from UberGraph. Only returned when the " + "`description` parameter is set, and omitted when no description is known.", + ) + taxa: Optional[List[str]] = Field( + None, + description="The taxa this identifier is associated with, as NCBITaxon CURIEs. Only returned when the " + "`include_taxa` parameter is set, and omitted when no taxa are known.", + ) + type: Optional[str] = Field( + None, + description="The most specific Biolink type of this individual identifier. Only returned when the " + "`individual_types` parameter is set. Useful for telling apart the members of a conflated " + "clique, e.g. which identifiers are the gene and which the protein.", + ) + + +class NormalizedNodeId(BaseModel): + """The preferred identifier and label for a normalized clique.""" + + identifier: str = Field( + ..., + description="The preferred CURIE for this clique: the first identifier in the Biolink Model's preferred " + "prefix order for this clique's type. With a conflation applied, this is the preferred " + "identifier of the first clique being conflated (for GeneProtein, always the gene).", + ) + label: Optional[str] = Field( + None, + description="The preferred label for this clique, as computed by Babel. Note that this is not necessarily " + "the label of the preferred identifier — for chemicals in particular, a label may be taken from " + "a different identifier. Omitted if nothing in the clique has a label.", + ) + description: Optional[str] = Field( + None, + description="One of the descriptions for the identifiers in this clique. Only returned when the " + "`description` parameter is set.", + ) + + +class NormalizedNode(BaseModel): + """A normalized clique: everything NodeNorm knows about one concept.""" + + id: NormalizedNodeId = Field(..., description="The preferred identifier and label for this clique.") + equivalent_identifiers: List[EquivalentIdentifier] = Field( + ..., + description="Every identifier in this clique, in the Biolink Model's preferred prefix order for this type. " + "When a conflation is applied this is a single flat list — all the identifiers of the first " + "clique, then those of the second, and so on — with no marker for where one clique ends and the " + "next begins.", + ) + type: List[str] = Field( + ..., + description="The Biolink classes for this clique, starting with the most specific type and ending with any " + "mixins. Note that `biolink:Entity` is deliberately excluded.", + ) + descriptions: Optional[List[str]] = Field( + None, + description="The unique descriptions of the identifiers in this clique, in the same order as " + "`equivalent_identifiers`. Only returned when the `description` parameter is set.", + ) + taxa: Optional[List[str]] = Field( + None, + description="Every taxon associated with any identifier in this clique, as NCBITaxon CURIEs. Only returned " + "when the `include_taxa` parameter is set.", + ) + information_content: Optional[float] = Field( + None, + description="How specific this concept is, from 0.0 (a broad, high-level term with many subclasses) to " + "100.0 (a very specific term with none). Taken from Ubergraph's normalizedInformationContent; " + "where several identifiers in the clique have a value, the lowest is reported. Omitted when no " + "identifier in the clique has one.", + ) class SetIDResponse(BaseModel): - curies: List[str] - conflations: List[str] - error: Optional[str] - normalized_curies: Optional[List[str]] - normalized_string: Optional[str] - setid: Optional[str] + curies: List[str] = Field(..., description="The CURIEs submitted for normalization, as submitted.") + conflations: List[str] = Field(..., description="The conflations applied, as submitted.") + error: Optional[str] = Field( + None, + description="Any error that occurred while normalizing this set. Note that a CURIE that simply cannot be " + "normalized is not an error.", + ) + normalized_curies: Optional[List[str]] = Field( + None, + description="The unique, sorted CURIEs used to construct the set ID. CURIEs that could not be normalized " + "are included unchanged.", + ) + normalized_string: Optional[str] = Field( + None, + description="The normalized CURIEs joined with `||`; this is the string the set ID is calculated from.", + ) + setid: Optional[str] = Field( + None, + description="The set ID: a UUID (prefixed with `uuid:`) derived from `normalized_string`. The same set of " + "CURIEs given to the same version of NodeNorm always produces the same set ID, but a different " + "Babel build may normalize them differently and so produce a different one.", + ) # base64: Optional[str] # base64zlib: Optional[str] # sha224hash: Optional[str] \ No newline at end of file diff --git a/node_normalizer/resources/openapi.yml b/node_normalizer/resources/openapi.yml index a2bf9571..de72d289 100644 --- a/node_normalizer/resources/openapi.yml +++ b/node_normalizer/resources/openapi.yml @@ -12,13 +12,16 @@ info:
  • Semantic types for the entity as defined by the Biolink Model The data served by Node Normalization is created by Babel, which attempts to find identifier equivalences, and makes sure that CURIE prefixes - are BioLink Model Compliant. To determine whether Node Normalization is likely + are BioLink Model Compliant. Babel decides which identifiers are equivalent, which one is + preferred, and what a clique is called, so most questions about why a clique looks the way it + does are really questions about Babel: see + Where NodeNorm''s data comes from. + The babel_version reported by /status identifies exactly which Babel build this + instance is serving. To determine whether Node Normalization is likely to be useful, check /get_semantic_types, which lists the BioLink semantic types for which normalization has been attempted, and /get_curie_prefixes, which lists the number of times each prefix is used for a semantic type. You can find out more about these API methods at the - Node Normalization API documentation. - To learn more about how Babel creates cliques of equivalent identifiers, you can - read its documentation.' + Node Normalization API documentation.' license: name: MIT url: https://opensource.org/licenses/MIT @@ -43,238 +46,3 @@ tags: - name: translator - name: Interfaces - name: trapi -paths: - /status: - get: - description: 'Returns the status of the Node Normalization service. You can read more about this endpoint in the - NodeNorm API documentation.' - responses: - '200': - description: Successful result describing this NodeNorm instance. - content: - application/json: - schema: - example: - status: running - babel_version: 2025mar31 - babel_version_url: https://github.com/NCATSTranslator/Babel/blob/master/releases/2025mar31.md - databases: - eq_id_to_id_db: - dbname: id-id - count: 677731045 - used_memory_rss_human: 68.83G - is_cluster: false - /get_curie_prefixes: - get: - description: 'Returns the curies and their hit count for a semantic type(s). You can read more about this endpoint - in the NodeNorm API documentation.' - parameters: - - in: query - name: semantic_type - schema: - example: - - biolink:ChemicalEntity - - biolink:AnatomicalEntity - items: - type: string - type: array - responses: - '200': - content: - application/json: - schema: - example: - "biolink:ChemicalEntity": - curie_prefix: - "PUBCHEM.COMPOUND": "123887334" - "INCHIKEY": "115975484" - "CAS": "4112274" - "HMDB": "217920" - "CHEMBL.COMPOUND": "2479770" - "UNII": "138975" - "CHEBI": "218762" - "MESH": "256235" - "UMLS": "603550" - "DrugCentral": "4995" - "GTOPDB": "13265" - "RXCUI": "124800" - "DRUGBANK": "15274" - "KEGG.COMPOUND": "16039" - "biolink:AnatomicalEntity": - curie_prefix: - "UMLS": "159941" - "FMA": "98631" - "UBERON": "14564" - "ZFA": "606" - "NCIT": "10286" - "MESH": "1992" - "EMAPA": "966" - "FBbt": "117" - "WBbt": "18" - "GO": "4041" - "SNOMEDCT": "1422" - "CL": "3043" - type: object - description: Results - summary: Return the number of times each CURIE prefix appears in an equivalent - identifier for a semantic type - tags: - - Interfaces - /get_normalized_nodes: - get: - description: 'Returns the equivalent identifiers and semantic types for the CURIEs entered. - You can optionally conflate identifiers if needed. - You can read more about this endpoint in the - NodeNorm API documentation.' - parameters: - - in: query - name: curie - schema: - example: - - MESH:D014867 - - NCIT:C34373 - items: - type: string - type: array - responses: - '200': - content: - application/json: - schema: - example: - MESH:D014867: - id: - identifier: CHEBI:15377 - label: Water - equivalent_identifiers: - - identifier: CHEBI:15377 - label: Water - - identifier: CHEMBL.COMPOUND:CHEMBL1098659 - label: WATER - - identifier: DRUGBANK:DB09145 - - identifier: PUBCHEM:22247451 - - identifier: PUBCHEM:962 - - identifier: MESH:D014867 - - identifier: HMDB:HMDB0002111 - - identifier: INCHIKEY:XLYOFNOQVPJJNP-UHFFFAOYSA-N - - identifier: UNII:059QF0K00R - - identifier: KEGG.COMPOUND:C00001 - type: - - biolink:SmallMolecule - - biolink:MolecularEntity - - biolink:ChemicalEntity - - biolink:PhysicalEssence - - biolink:ChemicalOrDrugOrTreatment - - biolink:ChemicalEntityOrGeneOrGeneProduct - - biolink:ChemicalEntityOrProteinOrPolypeptide - - biolink:NamedThing - - biolink:PhysicalEssenceOrOccurrent - information_content: 47.5 - NCIT:C34373: - equivalent_identifiers: - - identifier: MONDO:0004976 - label: amyotrophic lateral sclerosis - - identifier: DOID:332 - - identifier: OPPHANET:803 - - identifier: UMLS:C0393554 - - identifier: MESH:D000690 - - identifier: MEDDRA:10002026 - - identifier: NCIT:C34373 - - identifier: HP:0007354 - label: Amyotrophic lateral sclerosis - id: - identifier: MONDO:0004976 - label: amyotrophic lateral sclerosis - type: - - biolink:Disease - - biolink:DiseaseorPhenotypicFeature - - biolink:BiologicalEntity - - biolink:ThingWithTaxon - - biolink:NamedThing - information_content: 74.9 - type: object - description: Results - summary: Get the equivalent identifiers and semantic types for the CURIEs entered. - tags: - - Interfaces - /get_semantic_types: - get: - description: 'Returns a distinct set of the semantic types discovered in the - compendium data. You can read more about this endpoint in the - NodeNorm API documentation.' - responses: - '200': - content: - application/json: - schema: - example: - semantic_types: - types: - - biolink:ChemicalMixture - - biolink:MacromolecularMachineMixin - - biolink:GeographicLocation - - biolink:Agent - - biolink:Protein - - biolink:DiseaseOrPhenotypicFeature - - etc. - type: object - description: Results - summary: Return a list of BioLink semantic types for which normalization has - been attempted. - tags: - - Interfaces - /get_allowed_conflations: - get: - description: 'Returns a list of allowed conflation options. Conflation allows cliques to be combined on-the-fly - on the basis of two different criteria: -
      -
    1. GeneProtein conflation merges protein-coding genes with the proteins they encode. The gene(s) always - appear first in the combined clique. -
    2. DrugChemical conflation merges chemicals based on their active ingredient. We attempt to ensure that - the active ingredient appears before any formulations in the combined clique. -
    - You can read more about conflation or - more about this endpoint.' - responses: - '200': - content: - application/json: - schema: - example: - conflations: - - GeneProtein - - DrugChemical - type: object - description: Results - summary: Return a list of named conflations. - tags: - - Interfaces - /get_setid: - get: - description: 'Returns the set ID for a given set of CURIEs. A set ID is an identifier that can be used to identify - this set of CURIEs going forward. It is currently impossible to recreate a set of CURIEs from a set ID, but the - same set of CURIEs given to the same version of Node Normalization will always return the same set ID. - You can read more about this endpoint in the - NodeNorm API documentation.' - responses: - '200': - description: Successful result - content: - application/json: - schema: - example: - curies: - - MESH:D014867 - - NCIT:C34373 - - UNII:63M8RYN44N - - RUBBISH:1234 - conflations: - - GeneProtein - - DrugChemical - error: null - normalized_curies: - - CHEBI:15377 - - MONDO:0004976 - - RUBBISH:1234 - normalized_string: "CHEBI:15377||MONDO:0004976||RUBBISH:1234" - setid: "uuid:771d3c09-9a8c-5c46-8b85-97f481a90d40" diff --git a/node_normalizer/server.py b/node_normalizer/server.py index c0ace631..6875d668 100644 --- a/node_normalizer/server.py +++ b/node_normalizer/server.py @@ -26,12 +26,13 @@ ConflationList, SetIDResponse, SetIDQuery, + NormalizedNode, ) from .normalizer import get_normalized_nodes, get_curie_prefixes, normalize_message from .set_id import generate_setid from .redis_adapter import RedisConnectionFactory from .util import LoggingUtil -from .examples import EXAMPLE_QUERY_DRUG_TREATS_ESSENTIAL_HYPERTENSION +from .examples import EXAMPLE_QUERY_DRUG_TREATS_ESSENTIAL_HYPERTENSION, EXAMPLE_NORMALIZED_NODES logger = LoggingUtil.init_logging() @@ -96,7 +97,15 @@ async def shutdown_event(): @app.get( "/status", summary="Status information on this NodeNorm instance", - description="Returns information about this NodeNorm instance and the databases it is connected to." + description="Returns information about this NodeNorm instance and the databases it is connected to, including " + "the version of Babel whose output is loaded " + "into those databases (babel_version) and the Biolink Model version used to expand " + "semantic types (biolink_model). The backend databases are written once and never " + "updated in place, so babel_version identifies the data this instance will return until " + "an operator loads a newer build. You can read more about this endpoint in the " + "" + "NodeNorm API documentation.", + response_description="Information about this NodeNorm instance and the databases it is connected to.", ) async def status_get() -> Dict: """ Return status information about this NodeNorm instance as well as its databases. """ @@ -179,8 +188,16 @@ async def status() -> Dict: @app.post( "/query", - summary="Normalizes a TRAPI response object", - description="Returns the response object with a merged knowledge graph and query graph bindings", + summary="Normalizes every identifier in a TRAPI message", + description="Normalizes the identifiers in the knowledge graph, query graph and results of a TRAPI message, " + "returning the message with a merged knowledge graph and updated bindings. Each normalized node also " + "gains the clique's information content as an attribute " + "(biolink:has_numeric_value / information_content). " + "Deprecated: this endpoint is no longer actively maintained and will be removed once " + "the Workflow Runner stops using it " + "(PR #323). New callers " + "should extract the CURIEs they care about and use /get_normalized_nodes instead.", + response_description="The submitted TRAPI message with all identifiers normalized.", response_model=reasoner_pydantic.Query, response_model_exclude_none=True, response_model_exclude_unset=True, @@ -199,8 +216,15 @@ async def query(query: Annotated[reasoner_pydantic.Query, Body(openapi_examples= @app.post( "/asyncquery", - summary="Normalizes a TRAPI response object", - description="Returns the response object with a merged knowledge graph and query graph bindings", + summary="Normalizes every identifier in a TRAPI message, returning the result to a callback URL", + description="Identical to /query, except that it returns as soon as the work has been queued and POSTs the " + "normalized TRAPI message to the callback URL given in the request body when it is " + "ready (retrying a few times if the callback fails). " + "Deprecated: this endpoint is no longer actively maintained and will be removed once " + "the Workflow Runner stops using it " + "(PR #323). New callers " + "should extract the CURIEs they care about and use /get_normalized_nodes instead.", + response_description="Confirmation that the query has been queued; the normalized message is sent to the callback URL.", deprecated=True, ) async def async_query(async_query: reasoner_pydantic.AsyncQuery): @@ -246,8 +270,22 @@ async def async_query_task(async_query: reasoner_pydantic.AsyncQuery): @app.get( "/get_allowed_conflations", - summary="Get the available conflations", - description="The returned strings can be included in an option to /get_normalized_nodes", + summary="Return a list of named conflations.", + description="Returns a list of allowed conflation options. Conflation allows cliques to be combined on-the-fly " + "on the basis of two different criteria:" + "
      " + "
    1. GeneProtein conflation merges protein-coding genes with the proteins they encode. " + "The gene(s) always appear first in the combined clique." + "
    2. DrugChemical conflation merges chemicals based on their active ingredient. We " + "attempt to ensure that the active ingredient appears before any formulations in the combined clique." + "
    " + "The returned strings can be used with the conflation parameter of /get_setid; " + "/get_normalized_nodes has a separate boolean flag for each. You can read " + "" + "more about conflation or " + "" + "more about this endpoint.", + response_description="The list of conflations supported by this NodeNorm instance.", ) async def get_conflations() -> ConflationList: """ @@ -261,10 +299,26 @@ async def get_conflations() -> ConflationList: @app.get( "/get_normalized_nodes", summary="Get the equivalent identifiers and semantic types for the CURIEs entered.", - description="Returns the equivalent identifiers and semantic types for the CURIEs entered." - "You can optionally conflate identifiers if needed." + description="Returns the equivalent identifiers and semantic types for the CURIEs entered. " + "A CURIE that cannot be normalized is returned as a key with a null value rather than " + "being left out of the response. " + "You can optionally " + "" + "conflate identifiers if needed: conflate merges genes with the proteins they encode " + "(the gene comes first), and drug_chemical_conflate merges drugs with their active " + "ingredient (the ingredient comes before any formulations). Conflated cliques are returned as a single " + "flat list of equivalent identifiers, so use individual_types if you need to tell the " + "members apart. " "You can read more about this endpoint in the " - "NodeNorm API documentation.", + "NodeNorm API documentation, " + "and about where the identifiers, labels and information content values come from in " + "" + "Where NodeNorm's data comes from.", + responses={200: { + "model": Dict[str, Optional[NormalizedNode]], + "description": "A mapping from each CURIE queried to its normalized clique, or to null if it could not be normalized.", + "content": {"application/json": {"example": EXAMPLE_NORMALIZED_NODES}}, + }}, ) async def get_normalized_node_handler( curie: List[str] = fastapi.Query( @@ -299,8 +353,21 @@ async def get_normalized_node_handler( @app.post( "/get_normalized_nodes", - summary="Get the equivalent identifiers and semantic types for the curie(s) entered.", - description="Returns the equivalent identifiers and semantic types for the curie(s). Use the `conflate` flag to choose whether to apply conflation.", + summary="Get the equivalent identifiers and semantic types for the CURIEs entered.", + description="Returns the equivalent identifiers and semantic types for the CURIEs entered. Identical to the GET " + "method of this endpoint, but takes a curies list in a JSON body instead of repeated " + "curie query parameters. " + "Note that drug_chemical_conflate currently defaults to false here " + "but true on the GET method " + "(#398); set it " + "explicitly if you care which conflations are applied. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + responses={200: { + "model": Dict[str, Optional[NormalizedNode]], + "description": "A mapping from each CURIE queried to its normalized clique, or to null if it could not be normalized.", + "content": {"application/json": {"example": EXAMPLE_NORMALIZED_NODES}}, + }}, ) async def get_normalized_node_handler_post(curies: CurieList): """ @@ -322,7 +389,18 @@ async def get_normalized_node_handler_post(curies: CurieList): @app.get( "/get_setid", response_model=SetIDResponse, - summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set." + summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set.", + description="Returns the set ID for a given set of CURIEs. CURIEs that can be normalized are normalized (using " + "the conflations provided); those that cannot are kept as-is. Duplicates are then removed, the " + "remaining CURIEs are sorted, and a hash is generated from them. " + "A set ID is an identifier that can be used to identify this set of CURIEs going forward. It is " + "currently impossible to recreate a set of CURIEs from a set ID, but the same set of CURIEs given to " + "the same version of Node Normalization will always return the same set ID. Note that a different " + "Babel build may normalize the same CURIEs differently and so produce a different set ID — see " + "babel_version in /status. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="The normalized CURIEs and the set ID calculated from them.", ) async def get_setid( curie: List[str] = fastapi.Query( @@ -343,7 +421,13 @@ async def get_setid( @app.post( "/get_setid", response_model=List[SetIDResponse], - summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set." + summary="Normalize and deduplicate a set of identifiers and return a single hash that represents this set.", + description="Identical to the GET method of this endpoint, but calculates a set ID for several sets at once. " + "Takes a list of sets, each with its own curies and optional conflations, " + "and returns a list of results in the same order. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="One result per set submitted, in the order submitted.", ) async def get_setid( sets: List[SetIDQuery] = fastapi.Body([], @@ -367,7 +451,12 @@ async def get_setid( "/get_semantic_types", response_model=SemanticTypes, summary="Return a list of BioLink semantic types for which normalization has been attempted.", - description="Returns a distinct set of the semantic types discovered in the compendium data.", + description="Returns a distinct, unordered set of the Biolink semantic types found in the " + "Babel compendia loaded into this instance. " + "Returns 404 if no semantic types could be found, which usually means the databases have not been " + "loaded. You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="The distinct Biolink semantic types present in this instance.", ) async def get_semantic_types_handler() -> SemanticTypes: # look for all biolink semantic types @@ -389,11 +478,20 @@ async def get_semantic_types_handler() -> SemanticTypes: "/get_curie_prefixes", response_model=Dict[str, CuriePivot], summary="Return the number of times each CURIE prefix appears in an equivalent identifier for a semantic type", - description="Returns the curies and their hit count for a semantic type(s).", + description="Returns the CURIE prefixes and their hit counts for one or more semantic types. Omit " + "semantic_type to get every semantic type. Counts are returned as strings. " + "These counts are tallied when the " + "Babel compendia are loaded into this " + "instance and are approximate — the load aggregates them concurrently without locking " + "(#380), so prefer " + "Babel's own reports if you need exact figures. You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="A mapping from each semantic type requested to its CURIE prefix counts.", ) async def get_curie_prefixes_handler( - semantic_type: Optional[List[str]] = fastapi.Query([], description="e.g. biolink:ChemicalEntity, " - "biolink:AnatomicalEntity") + semantic_type: Optional[List[str]] = fastapi.Query([], description="The semantic types to report on, e.g. " + "biolink:ChemicalEntity, " + "biolink:AnatomicalEntity. Omit for all types.") ) -> Dict[str, CuriePivot]: return await get_curie_prefixes(app, semantic_type) @@ -402,7 +500,10 @@ async def get_curie_prefixes_handler( "/get_curie_prefixes", response_model=Dict[str, CuriePivot], summary="Return the number of times each CURIE prefix appears in an equivalent identifier for a semantic type", - description="Returns the curies and their hit count for a semantic type(s).", + description="Identical to the GET method of this endpoint, but takes the list of semantic types in a JSON body. " + "You can read more about this endpoint in the " + "NodeNorm API documentation.", + response_description="A mapping from each semantic type requested to its CURIE prefix counts.", ) async def get_curie_prefixes_handler( semantic_types: SemanticTypesInput, diff --git a/releases/TranslatorDecember2023.md b/releases/TranslatorDecember2023.md index 8da24fbd..10437667 100644 --- a/releases/TranslatorDecember2023.md +++ b/releases/TranslatorDecember2023.md @@ -1,7 +1,7 @@ # NodeNorm Translator December 2023 Release - Babel: [2023nov5](https://stars.renci.org/var/babel_outputs/2023nov5/) -- NodeNorm: [v2.3.5](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.5) +- NodeNorm: [v2.3.5](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.5) Next release: [May 2024](TranslatorMay2024.md) @@ -24,26 +24,26 @@ Next release: [May 2024](TranslatorMay2024.md) ## Releases since [Translator October 2023 release](TranslatorOctober2023.md) -* [NodeNorm v2.2.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.2.1) +* [NodeNorm v2.2.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.2.1) * Node descriptions by @YaphetKG in #212 * Trapi 14 by @cbizon in #208 -* [NodeNorm v2.3.0](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.0) +* [NodeNorm v2.3.0](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.0) * Node descriptions by @YaphetKG in #216 * initial implementation chem conflation by @cbizon in #211 -* [NodeNorm v2.3.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.1) +* [NodeNorm v2.3.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.1) * Fix Terms of Service and service description by @gaurav in #226 -* [NodeNorm v2.3.2](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.2) +* [NodeNorm v2.3.2](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.2) * Bump httpx from 0.19.0 to 0.23.0 by @dependabot in #195 * Bump requests from 2.28.1 to 2.31.0 by @dependabot in #194 * Upgrade reasoner-pydantic and requirements.lock by @gaurav in #227 -* [NodeNorm v2.3.3](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.3) +* [NodeNorm v2.3.3](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.3) * Add NodeNorm-Loader Docker by @gaurav in #228 * Fix missing node bindings in NodeNorm /query output by @gaurav in #231 * Uniquify semantic types returned from the database by @gaurav in #232 * Remove hardcoded TRAPI version by @gaurav in #233 * Remove biolink:Entity from nodes created by NodeNorm by @gaurav in #234 -* [NodeNorm v2.3.4](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.4) +* [NodeNorm v2.3.4](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.4) * Forgot to update the version number in NodeNorm v2.3.3. -* [NodeNorm v2.3.5](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.5) +* [NodeNorm v2.3.5](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.5) * Workaround for missing type information by @gaurav in #223 * Add Open Telemetry instrumentation by @gaurav in #237 diff --git a/releases/TranslatorFuguJuly2024.md b/releases/TranslatorFuguJuly2024.md index 5d84318a..6d9900f8 100644 --- a/releases/TranslatorFuguJuly2024.md +++ b/releases/TranslatorFuguJuly2024.md @@ -1,20 +1,20 @@ # NodeNorm Translator "Fugu" July 2024 Release - Babel: [2024jul13](https://stars.renci.org/var/babel_outputs/2024jul13/) - ([Babel Translator July 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorFuguJuly2024.md)) -- NodeNorm: [v2.3.15](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.15) + ([Babel Translator July 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorFuguJuly2024.md)) +- NodeNorm: [v2.3.15](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.15) Previous release: [Translator May 2024](./TranslatorMay2024.md) Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) ## New features -* Update Biolink model by replacing bmt-lite with bmt ([#282](https://github.com/TranslatorSRI/NodeNormalization/pull/282), - [#284](https://github.com/TranslatorSRI/NodeNormalization/pull/284)) -* Added /get_setid GET endpoint need for Multi-CURIE Query (MCQ) ([#274](https://github.com/TranslatorSRI/NodeNormalization/pull/274)) +* Update Biolink model by replacing bmt-lite with bmt ([#282](https://github.com/NCATSTranslator/NodeNormalization/pull/282), + [#284](https://github.com/NCATSTranslator/NodeNormalization/pull/284)) +* Added /get_setid GET endpoint need for Multi-CURIE Query (MCQ) ([#274](https://github.com/NCATSTranslator/NodeNormalization/pull/274)) ## Technical debt * Improved database names in source code. -## Babel updates (from [Babel Translator July 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorFuguJuly2024.md)) +## Babel updates (from [Babel Translator July 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorFuguJuly2024.md)) * [Feature] Added manual disease concords, and used that to do a better job of combining opioid use disorder and alcohol use disorder. * [Feature] Moved ensembl_datasets_to_skip into the config file. * [Bugfix] Eliminated preferred prefix overrides in Babel; we now trust the preferred prefixes from the Biolink Model. @@ -28,12 +28,12 @@ Next release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) * [Update] Updated PANTHER pathways from SequenceAssociationPathway3.6.7.txt to SequenceAssociationPathway3.6.8.txt. ## Releases since [Translator May 2024 release](TranslatorMay2024.md) -* [NodeNorm v2.3.12](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.12) +* [NodeNorm v2.3.12](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.12) * Rename redis_connection[0-6] to their database names by @gaurav in #271 -* [NodeNorm v2.3.13](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.13): Incremented version, +* [NodeNorm v2.3.13](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.13): Incremented version, which I forgot to do in the previous release. -* [NodeNorm v2.3.14](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.14) +* [NodeNorm v2.3.14](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.14) * Replaced bmt-lite with bmt by @gaurav in #282 * Add a /get_setid GET endpoint by @gaurav in #274 -* [NodeNorm v2.3.15](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.15) +* [NodeNorm v2.3.15](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.15) * Fix BMT import and formatting issues by @gaurav in #284 diff --git a/releases/TranslatorGuppyAugust2024.md b/releases/TranslatorGuppyAugust2024.md index 6be46539..be7b3e20 100644 --- a/releases/TranslatorGuppyAugust2024.md +++ b/releases/TranslatorGuppyAugust2024.md @@ -1,31 +1,31 @@ # NodeNorm Translator "Guppy" August 2024 Release - Babel: [2024aug18](https://stars.renci.org/var/babel_outputs/2024aug18/) - ([Babel Translator August 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorGuppyAugust2024.md)) -- NodeNorm: [v2.3.16](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.16) + ([Babel Translator August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) +- NodeNorm: [v2.3.16](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.16) Next release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) ## New features -* Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/TranslatorSRI/NodeNormalization/pull/289) -* Add POST SetID by @gaurav in [#290](https://github.com/TranslatorSRI/NodeNormalization/pull/290) +* Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/NCATSTranslator/NodeNormalization/pull/289) +* Add POST SetID by @gaurav in [#290](https://github.com/NCATSTranslator/NodeNormalization/pull/290) ## Technical debt -* Several improvements to the compendium loader by @gaurav in [#291](https://github.com/TranslatorSRI/NodeNormalization/pull/291) +* Several improvements to the compendium loader by @gaurav in [#291](https://github.com/NCATSTranslator/NodeNormalization/pull/291) -## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorGuppyAugust2024.md)) +## Babel updates (from [Babel Translator "Guppy" August 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorGuppyAugust2024.md)) * [Feature] Added support for generating DuckDB and Parquet files from the compendium and synonym files, allowing us to run queries such as looking for all the identically labeled cliques across all the compendia. Increased Babel Outputs file size to support DuckDB. -* [Feature] Added labels from DrugBank (https://github.com/TranslatorSRI/Babel/pull/335). -* [Feature] Improved cell anatomy concords using Wikidata (https://github.com/TranslatorSRI/Babel/pull/329). -* [Feature] Added manual concords for the DrugChemical conflation (https://github.com/TranslatorSRI/Babel/pull/337). -* [Feature] Wrote a script for comparing between two summary files (https://github.com/TranslatorSRI/Babel/pull/320). +* [Feature] Added labels from DrugBank (https://github.com/NCATSTranslator/Babel/pull/335). +* [Feature] Improved cell anatomy concords using Wikidata (https://github.com/NCATSTranslator/Babel/pull/329). +* [Feature] Added manual concords for the DrugChemical conflation (https://github.com/NCATSTranslator/Babel/pull/337). +* [Feature] Wrote a script for comparing between two summary files (https://github.com/NCATSTranslator/Babel/pull/320). * [Feature] Added timestamping as an option to Wget. * [Feature] Reorganized primary label determination so that we can include it in compendia files as well. * This isn't currently used by the loader, but might be in the future. For now, this is only useful in helping track what labels are being chosen as the preferred label. -* [Bugfixes] Added additional ENSEMBL datasets to skip (https://github.com/TranslatorSRI/Babel/pull/297). +* [Bugfixes] Added additional ENSEMBL datasets to skip (https://github.com/NCATSTranslator/Babel/pull/297). * [Bugfixes] Fixed a bug in recognizing the end of file when reading the PubChem ID and SMILES files. * [Bugfixes] Fixed the lack of `clique_identifier_count` in leftover UMLS output. * [Bugfixes] Fixed unraised exception in Ensembl BioMart download. @@ -35,10 +35,10 @@ Previous release: [Translator "Fugu" July 2024](./TranslatorFuguJuly2024.md) * [Updates] Changed DrugBank ID types from 'ChemicalEntity' to 'Drug'. ## Releases since [Translator Fugu July 2024](./TranslatorFuguJuly2024.md) -* [2.3.16](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.16) - * Added TranslatorFuguJuly2024 release information by @gaurav in [#287](https://github.com/TranslatorSRI/NodeNormalization/pull/287) - * Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/TranslatorSRI/NodeNormalization/pull/289) - * Bump requests from 2.31.0 to 2.32.0 by @dependabot in [#261](https://github.com/TranslatorSRI/NodeNormalization/pull/261) - * Add POST SetID by @gaurav in [#290](https://github.com/TranslatorSRI/NodeNormalization/pull/290) - * Several improvements to the compendium loader by @gaurav in [#291](https://github.com/TranslatorSRI/NodeNormalization/pull/291) +* [2.3.16](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.16) + * Added TranslatorFuguJuly2024 release information by @gaurav in [#287](https://github.com/NCATSTranslator/NodeNormalization/pull/287) + * Optionally return types for every equivalent identifier by @gaurav in [#289](https://github.com/NCATSTranslator/NodeNormalization/pull/289) + * Bump requests from 2.31.0 to 2.32.0 by @dependabot in [#261](https://github.com/NCATSTranslator/NodeNormalization/pull/261) + * Add POST SetID by @gaurav in [#290](https://github.com/NCATSTranslator/NodeNormalization/pull/290) + * Several improvements to the compendium loader by @gaurav in [#291](https://github.com/NCATSTranslator/NodeNormalization/pull/291) diff --git a/releases/TranslatorHammerheadNovember2024.md b/releases/TranslatorHammerheadNovember2024.md index b21f1c67..ede4016b 100644 --- a/releases/TranslatorHammerheadNovember2024.md +++ b/releases/TranslatorHammerheadNovember2024.md @@ -1,31 +1,31 @@ # NodeNorm Translator "Hammerhead" November 2024 Release - Babel: [2024oct24](https://stars.renci.org/var/babel_outputs/2024oct24/) - ([Babel Translator November 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorHammerheadNovember2024.md)) -- NodeNorm: [v2.3.18](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.18) + ([Babel Translator November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) +- NodeNorm: [v2.3.18](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.18) Next release: _None as yet_ Previous release: [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) ## New features -* Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/TranslatorSRI/NodeNormalization/pull/268)). -* Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/TranslatorSRI/NodeNormalization/pull/298)). +* Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/NCATSTranslator/NodeNormalization/pull/268)). +* Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/NCATSTranslator/NodeNormalization/pull/298)). ## Updates -* Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/TranslatorSRI/NodeNormalization/pull/300)). +* Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/NCATSTranslator/NodeNormalization/pull/300)). ## Documentation -* Added Guppy release notes ([#294](https://github.com/TranslatorSRI/NodeNormalization/pull/294)). +* Added Guppy release notes ([#294](https://github.com/NCATSTranslator/NodeNormalization/pull/294)). -## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/TranslatorSRI/Babel/blob/master/releases/TranslatorHammerheadNovember2024.md)) -- [New features] Added taxon information to proteins ([#349](https://github.com/TranslatorSRI/Babel/pull/349)) +## Babel updates (from [Babel Translator "Hammerhead" November 2024 Release](https://github.com/NCATSTranslator/Babel/blob/main/releases/TranslatorHammerheadNovember2024.md)) +- [New features] Added taxon information to proteins ([#349](https://github.com/NCATSTranslator/Babel/pull/349)) - [Updates] Upgraded RxNorm to 09032024. - [Updates] Changed NCBIGene download from FTP to HTTP. -- [Updates] Increased DRUG_CHEMICAL_SMALLER_MAX_LABEL_LENGTH (introduced in [#330](https://github.com/TranslatorSRI/Babel/pull/330)) from 30 to 40. +- [Updates] Increased DRUG_CHEMICAL_SMALLER_MAX_LABEL_LENGTH (introduced in [#330](https://github.com/NCATSTranslator/Babel/pull/330)) from 30 to 40. ## Releases since [Translator "Guppy" August 2024](./TranslatorGuppyAugust2024.md) -* [2.3.17](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.17) - * Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/TranslatorSRI/NodeNormalization/pull/268)). - * Added Guppy release notes ([#294](https://github.com/TranslatorSRI/NodeNormalization/pull/294)). -* [2.3.18](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.18) - * Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/TranslatorSRI/NodeNormalization/pull/298)). - * Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/TranslatorSRI/NodeNormalization/pull/300)). +* [2.3.17](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.17) + * Add a status endpoint with database name, key count and memory usage for each database ([#268](https://github.com/NCATSTranslator/NodeNormalization/pull/268)). + * Added Guppy release notes ([#294](https://github.com/NCATSTranslator/NodeNormalization/pull/294)). +* [2.3.18](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.18) + * Upgraded OTEL implementation to use gRPC by @EvanDietzMorris ([#298](https://github.com/NCATSTranslator/NodeNormalization/pull/298)). + * Updated NodeNorm preferred label to match Babel's ([#300](https://github.com/NCATSTranslator/NodeNormalization/pull/300)). diff --git a/releases/TranslatorMay2024.md b/releases/TranslatorMay2024.md index f989cb65..8cfed7ab 100644 --- a/releases/TranslatorMay2024.md +++ b/releases/TranslatorMay2024.md @@ -2,7 +2,7 @@ - Babel: [2024mar24](https://stars.renci.org/var/babel_outputs/2024mar24/) (Babel Translator May 2024 Release) -- NodeNorm: [v2.3.11](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.11) +- NodeNorm: [v2.3.11](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.11) Next release: [Translator "Fugu" July 2024](TranslatorFuguJuly2024.md) @@ -12,26 +12,26 @@ Next release: [Translator "Fugu" July 2024](TranslatorFuguJuly2024.md) ## Babel updates (from Babel Translator May 2024 Release) * [New identifiers] 36.9 million PubMed IDs (e.g. `PMID:25061375`) have been added as `biolink:JournalArticle`, as well as the mappings to PMC (e.g. `PMC:PMC4109484`) and DOIs (e.g. `doi:10.3897/zookeys.420.7089`) that are included in PubMed. - Details in [TranslatorSRI/Babel#227](https://github.com/TranslatorSRI/Babel/pull/227). + Details in [NCATSTranslator/Babel#227](https://github.com/NCATSTranslator/Babel/pull/227). * Fixed type determination for DrugChemical conflation. Details in - [TranslatorSRI/Babel#266](https://github.com/TranslatorSRI/Babel/pull/266). + [NCATSTranslator/Babel#266](https://github.com/NCATSTranslator/Babel/pull/266). * Synonym files now include the clique identifier count (the number of identifiers in each clique) in synonyms file. * Minor fixes. ## Releases since [Translator December 2023 release](TranslatorDecember2023.md) -* [NodeNorm v2.3.6](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.6) +* [NodeNorm v2.3.6](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.6) * Reverted NodeNorm loader to Redis v6 -* [NodeNorm v2.3.7](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.7) +* [NodeNorm v2.3.7](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.7) * Added Translator release information * Fixed examples -* [NodeNorm v2.3.8](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.8): Forgot to increment the +* [NodeNorm v2.3.8](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.8): Forgot to increment the version in the previous release. -* [NodeNorm v2.3.9](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.9) +* [NodeNorm v2.3.9](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.9) * Bump gunicorn from 20.1.0 to 22.0.0 by @dependabot in #255 * Bump orjson from 3.8.10 to 3.9.15 by @dependabot in #253 * Added NodeNorm May 2024 release to the release notes by @gaurav in #258 * Sort chemical labels in the configured prefix order by @gaurav in #260 -* [NodeNorm v2.3.10](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.10) +* [NodeNorm v2.3.10](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.10) * Updated TRAPI version to 1.5.0 by @gaurav in #269 -* [NodeNorm v2.3.11](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.11) +* [NodeNorm v2.3.11](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.11) * Change default TRAPI version for URL to 1.5 by @gaurav in #270 \ No newline at end of file diff --git a/releases/TranslatorOctober2023.md b/releases/TranslatorOctober2023.md index fc7d67a6..fcecf5e2 100644 --- a/releases/TranslatorOctober2023.md +++ b/releases/TranslatorOctober2023.md @@ -1,6 +1,6 @@ # NodeNorm Translator October 2023 Release -- Babel: [2022dec2-2](https://stars.renci.org/var/babel_outputs/2022dec2-2/) (roughly Babel [v1.2.0](https://github.com/TranslatorSRI/Babel/releases/tag/v1.2.0)) -- NodeNorm: [v2.1.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.1.1) +- Babel: [2022dec2-2](https://stars.renci.org/var/babel_outputs/2022dec2-2/) (roughly Babel [v1.2.0](https://github.com/NCATSTranslator/Babel/releases/tag/v1.2.0)) +- NodeNorm: [v2.1.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.1.1) Next release: [December 2023](TranslatorDecember2023.md) \ No newline at end of file diff --git a/releases/v2.3.19.md b/releases/v2.3.19.md index fb4b6f53..48afc248 100644 --- a/releases/v2.3.19.md +++ b/releases/v2.3.19.md @@ -1,7 +1,7 @@ # NodeNorm v2.3.19 - Babel: [2025jan23](https://stars.renci.org/var/babel_outputs/2025jan23/) - ([Babel 2025jan23](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025jan23.md)) -- NodeNorm: [v2.3.19](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.19) + ([Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) +- NodeNorm: [v2.3.19](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.19) Next release: [v2.3.26](./v2.3.26.md) Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) @@ -12,13 +12,13 @@ Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerhead ## Updates * _None_ -## Babel updates (from [Babel 2025jan23](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025jan23.md)) -- [New feature] Added a check for duplicate CURIEs [#342](https://github.com/TranslatorSRI/Babel/pull/342). +## Babel updates (from [Babel 2025jan23](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025jan23.md)) +- [New feature] Added a check for duplicate CURIEs [#342](https://github.com/NCATSTranslator/Babel/pull/342). - [New feature] Added some additional manual concords for Disease/Phenotype cliques and DrugChemical - conflation [#360](https://github.com/TranslatorSRI/Babel/pull/360). -- [New feature] Replace use of `has_tradename` with `tradename_of` in RxNorm ([#377](https://github.com/TranslatorSRI/Babel/pull/377)). -- [New feature] Added processes from UMLS ([#395](https://github.com/TranslatorSRI/Babel/pull/395)). -- [New feature] Improved EFO relationships ([#396](https://github.com/TranslatorSRI/Babel/pull/396)). + conflation [#360](https://github.com/NCATSTranslator/Babel/pull/360). +- [New feature] Replace use of `has_tradename` with `tradename_of` in RxNorm ([#377](https://github.com/NCATSTranslator/Babel/pull/377)). +- [New feature] Added processes from UMLS ([#395](https://github.com/NCATSTranslator/Babel/pull/395)). +- [New feature] Improved EFO relationships ([#396](https://github.com/NCATSTranslator/Babel/pull/396)). - [Updates] Various updates - [Bugfix] Fixed a bug in choosing the best label shorter than a particular size in src/babel_utils.py:write_compendium() - [Bugfix] Cleaned up src/createcompendia/chemicals.py:parse_smifile() so that includes the ChEMBL ID and calculates the column index by name, with ValueErrors thrown if the column name is missing. @@ -26,5 +26,5 @@ Previous release: [Translator "Hammerhead" November 2024](./TranslatorHammerhead - [Bugfix] Other minor fixes. ## Releases since [Translator "Hammerhead" November 2024](./TranslatorHammerheadNovember2024.md) -* [2.3.19](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.19) +* [2.3.19](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.19) * No changes: incrementing version solely to distinguish between Babel 2024oct24 and 2025jan23. diff --git a/releases/v2.3.26.md b/releases/v2.3.26.md index a68d577e..1b28725c 100644 --- a/releases/v2.3.26.md +++ b/releases/v2.3.26.md @@ -1,43 +1,43 @@ # NodeNorm v2.3.26 - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) - ([Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) -- NodeNorm: [v2.3.26](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.26) + ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) +- NodeNorm: [v2.3.26](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.26) Next release: [v2.4.1](./v2.4.1.md) Previous release: [v2.3.19](./v2.3.19.md) ## New features -* The Babel version is now included in the `/status` endpoint ([PR #315](https://github.com/TranslatorSRI/NodeNormalization/pull/315)). -* Added logging of normalization so we can benchmark NodeNorm ([PR #312](https://github.com/TranslatorSRI/NodeNormalization/pull/312)). +* The Babel version is now included in the `/status` endpoint ([PR #315](https://github.com/NCATSTranslator/NodeNormalization/pull/315)). +* Added logging of normalization so we can benchmark NodeNorm ([PR #312](https://github.com/NCATSTranslator/NodeNormalization/pull/312)). ## Updates -* The `/query` and `/asyncquery` endpoints are now deprecated ([PR #323](https://github.com/TranslatorSRI/NodeNormalization/pull/323)). +* The `/query` and `/asyncquery` endpoints are now deprecated ([PR #323](https://github.com/NCATSTranslator/NodeNormalization/pull/323)). * Improved documentation. * Added release notes for November 2024 and January 2025. * Updated renci-python-image to GitHub Packages. -## Babel updates (from [Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) +## Babel updates (from [Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) - [MAJOR] Information content values were incorrectly calculated as `100` for some cliques when they were in fact lower. - [MAJOR] UMLS proteins have now been removed from chemicals so that they only appear in proteins. - [MAJOR] Improved DrugChemical conflation order. - [MAJOR] Added secondary identifiers for ChEBI. ## Releases since [NodeNorm v2.3.19](./v2.3.19.md) -* [NodeNorm v2.3.20](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.20) +* [NodeNorm v2.3.20](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.20) * Add release info for November 2024 and January 2025 by @gaurav in #308 * Bump gunicorn from 22.0.0 to 23.0.0 by @dependabot in #310 -* [NodeNorm v2.3.21](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.21) +* [NodeNorm v2.3.21](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.21) * Incremented version (forgot to do that in the previous release). -* [NodeNorm v2.3.22](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.22) +* [NodeNorm v2.3.22](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.22) * Added normalization logging by @gaurav in #312 -* [NodeNorm v2.3.23](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.23) +* [NodeNorm v2.3.23](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.23) * Add Babel version to /status by @gaurav in #315 -* [NodeNorm v2.3.24](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.24) +* [NodeNorm v2.3.24](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.24) * Deprecated TRAPI methods (/query and /asyncquery) by @gaurav in #323 * Added API documentation by @gaurav in #322 * Updated RENCI Python Image in Dockerfile. -* [NodeNorm v2.3.25](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.25) +* [NodeNorm v2.3.25](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.25) * Removed unused/invalid imports. -* [NodeNorm v2.3.26](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.25) +* [NodeNorm v2.3.26](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.25) * Bump deepdiff from 5.8.1 to 8.6.1 by @dependabot in #328 * Bump requests from 2.32.0 to 2.32.4 by @dependabot in #319 diff --git a/releases/v2.4.1.md b/releases/v2.4.1.md index 6a2ae19d..d7039307 100644 --- a/releases/v2.4.1.md +++ b/releases/v2.4.1.md @@ -1,7 +1,7 @@ # NodeNorm v2.4.1 - Babel: [2025sep1](https://stars.renci.org/var/babel_outputs/2025sep1/) - ([Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) -- NodeNorm: [v2.4.1](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.4.1) + ([Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) +- NodeNorm: [v2.4.1](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.4.1) Next release: _None as yet_ Previous release: [v2.3.26](./v2.3.26.md) @@ -23,17 +23,17 @@ Previous release: [v2.3.26](./v2.3.26.md) * (v2.3.27) NEW BUG: `description` accidentally moved from `result['id']['description']` to `result['description']` ([#345](https://github.com/NCATSTranslator/NodeNormalization/pull/345)). * (v2.3.28) Fixed description bug introduced in NodeNorm v2.3.27 by @gaurav in [#346](https://github.com/NCATSTranslator/NodeNormalization/pull/346) -## Babel updates (from [Babel 2025sep1](https://github.com/TranslatorSRI/Babel/blob/master/releases/2025sep1.md)) +## Babel updates (from [Babel 2025sep1](https://github.com/NCATSTranslator/Babel/blob/main/releases/2025sep1.md)) Babel was not updated in this release. ## Releases since [NodeNorm v2.3.26](./v2.3.26.md) -* [NodeNorm v2.3.27](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.27) +* [NodeNorm v2.3.27](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.27) * Added release notes for NodeNorm v2.3.26 by @gaurav in [#329](https://github.com/NCATSTranslator/NodeNormalization/pull/329) * Add an `include_taxa` field (default: True) that will return the list of taxa for a clique and equivalent identifiers by @gaurav in [#339](https://github.com/NCATSTranslator/NodeNormalization/pull/339) * NEW BUG: `description` accidentally moved from `result['id']['description']` to `result['description']` ([#345](https://github.com/NCATSTranslator/NodeNormalization/pull/345)). -* [NodeNorm v2.3.28](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.3.27) +* [NodeNorm v2.3.28](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.3.27) * Fixed description bug introduced in NodeNorm v2.3.27 by @gaurav in [#346](https://github.com/NCATSTranslator/NodeNormalization/pull/346) -* [NodeNorm v2.4.0](https://github.com/TranslatorSRI/NodeNormalization/releases/tag/v2.4.0) +* [NodeNorm v2.4.0](https://github.com/NCATSTranslator/NodeNormalization/releases/tag/v2.4.0) * Replace RDB reader by @gaurav in [#349](https://github.com/NCATSTranslator/NodeNormalization/pull/349) * Update NodeNorm Jupyter Notebook by @gaurav in [#265](https://github.com/NCATSTranslator/NodeNormalization/pull/265) * Load a particular version of Biolink Model as per BIOLINK_MODEL_TAG by @gaurav in [#351](https://github.com/NCATSTranslator/NodeNormalization/pull/351) diff --git a/tests/test_docs_links.py b/tests/test_docs_links.py new file mode 100644 index 00000000..86d1fb8b --- /dev/null +++ b/tests/test_docs_links.py @@ -0,0 +1,127 @@ +"""Checks on the links in our documentation and in the endpoint descriptions. + +Ported from NameResolution's tests/test_docs_links.py, so the two services enforce the same +rules. Keep them in sync when either changes. + +Everything here is offline: it resolves relative paths and heading anchors on disk and +greps for a banned URL form. Nothing fetches a URL, because a test that fails when +GitHub is slow is a test people learn to ignore. +""" + +import re +from pathlib import Path + +REPO_ROOT = Path(__file__).parent.parent + +# Markdown we own. The notebook and anything vendored is deliberately excluded. +MARKDOWN_FILES = sorted( + p for p in REPO_ROOT.rglob("*.md") + if not any(part in {"venv", ".venv", "node_modules", ".git", ".pytest_cache", "data"} + for part in p.parts) +) + +# Files that carry links out to GitHub in code rather than in prose. +SOURCE_WITH_LINKS = [REPO_ROOT / "node_normalizer" / "server.py", REPO_ROOT / "node_normalizer" / "resources" / "openapi.yml"] + +INLINE_LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)") +HEADING = re.compile(r"^#+\s+(.*)$", re.MULTILINE) + + +def anchors_for(markdown_text): + """The anchor slugs GitHub generates for a document's headings.""" + slugs = set() + for heading in HEADING.findall(markdown_text): + slug = re.sub(r"[`*]", "", heading.strip().lower()) + slug = re.sub(r"[^a-z0-9 _-]", "", slug) + slugs.add(slug.replace(" ", "-")) + return slugs + + +def test_relative_links_resolve(): + """A relative link is written relative to the file it sits in, which is easy to get + wrong when a document moves into a subdirectory.""" + broken = [] + for path in MARKDOWN_FILES: + for target in INLINE_LINK.findall(path.read_text()): + if target.startswith(("http://", "https://", "#", "mailto:")): + continue + resolved = (path.parent / target.split("#")[0]).resolve() + if not resolved.exists(): + broken.append(f"{path.relative_to(REPO_ROOT)} -> {target}") + assert not broken, "Relative links that do not resolve:\n " + "\n ".join(broken) + + +def test_anchors_resolve(): + """Anchors are case-sensitive and GitHub lower-cases them, so `#Conflation` silently + lands at the top of the page rather than at the heading.""" + broken = [] + for path in MARKDOWN_FILES: + for target in INLINE_LINK.findall(path.read_text()): + if "#" not in target or target.startswith(("http://", "https://", "mailto:")): + continue + rel, _, anchor = target.partition("#") + if not anchor: + continue + target_path = path if rel == "" else (path.parent / rel) + if target_path.suffix != ".md" or not target_path.exists(): + continue + if anchor not in anchors_for(target_path.read_text()): + broken.append(f"{path.relative_to(REPO_ROOT)} -> {target}") + assert not broken, "Anchors with no matching heading:\n " + "\n ".join(broken) + + +def test_in_repo_anchors_in_source_resolve(): + """The endpoint descriptions link into documentation/API.md by absolute URL, so the + check above cannot see them.""" + api_md = REPO_ROOT / "documentation" / "API.md" + valid = anchors_for(api_md.read_text()) + broken = [] + for path in SOURCE_WITH_LINKS: + for anchor in re.findall(r"documentation/API\.md#([A-Za-z0-9_-]+)", path.read_text()): + if anchor not in valid: + broken.append(f"{path.relative_to(REPO_ROOT)} -> API.md#{anchor}") + assert not broken, "Anchors with no matching heading in API.md:\n " + "\n ".join(broken) + + +#: A GitHub link pinned to `master`, in either the blob or the raw form. The raw half is scoped to +#: NCATSTranslator on purpose: api/server.py legitimately builds +#: raw.githubusercontent.com/biolink/biolink-model/master/... and biolink-model really does default +#: to master. +MASTER_LINK = re.compile( + r"github\.com/[^/\s]+/[^/\s]+/blob/master/" + r"|raw\.githubusercontent\.com/NCATSTranslator/[^/\s]+/master/", + re.IGNORECASE, +) + + +def test_no_master_branch_links(): + """NodeNormalization, Babel and NodeNormalization all default to `main`. A master URL resolves + only through GitHub's post-rename redirect, so it looks fine right up until that redirect goes + away. See CLAUDE.md.""" + offenders = [] + for path in MARKDOWN_FILES + SOURCE_WITH_LINKS: + for line_no, line in enumerate(path.read_text().splitlines(), start=1): + if MASTER_LINK.search(line): + offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") + assert not offenders, "Links pinned to `master` instead of `main`:\n " + "\n ".join(offenders) + + +#: The three repositories that moved from the TranslatorSRI org to NCATSTranslator. Scoped to +#: these by name on purpose -- TranslatorSRI/r3 (referenced from README.md) really does still +#: live under that org, so a blanket ban on the string would push someone to "fix" a correct +#: link into a 404. +MOVED_TO_NCATSTRANSLATOR = re.compile(r"TranslatorSRI/(Babel|NameResolution|NodeNormalization)\b") + + +def test_no_stale_org_links(): + """These three repositories are under NCATSTranslator now. The old URLs resolve through + GitHub's org-rename redirect, which is more durable than the branch-rename one, but they + still name an org that no longer owns the code.""" + offenders = [] + for path in MARKDOWN_FILES + SOURCE_WITH_LINKS: + for line_no, line in enumerate(path.read_text().splitlines(), start=1): + if MOVED_TO_NCATSTRANSLATOR.search(line): + offenders.append(f"{path.relative_to(REPO_ROOT)}:{line_no}") + assert not offenders, ( + "Links naming the pre-rename TranslatorSRI org:\n " + "\n ".join(offenders) + ) diff --git a/uv.lock b/uv.lock new file mode 100644 index 00000000..9431a635 --- /dev/null +++ b/uv.lock @@ -0,0 +1,3 @@ +version = 1 +revision = 3 +requires-python = ">=3.11"