Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ Babel Compendia Files → loader.py → Redis (7 DBs) → FastAPI (server.py)
### Key Modules

- **`node_normalizer/server.py`** — FastAPI app with all REST endpoints. Uses lifespan events for Redis connection setup/teardown. Root path is `/`.
- **`node_normalizer/apidocs.py`** — builds the served OpenAPI schema. `resources/openapi.yml` contributes **only** document-level metadata (title, version, description, contact, license, x-translator/x-trapi, servers, tags); `paths` is always regenerated from the route decorators, so per-endpoint documentation belongs on the decorator (`summary`, `description`, `response_description`), never in the YAML. Guarded by `tests/frontend/test_apidocs.py`.
- **`skills/nodenorm/SKILL.md`** — agent-facing usage instructions, also served at `GET /llms.txt`. The Dockerfile must `COPY ./skills`, or that route 404s in deployed images while working locally.
- **`node_normalizer/normalizer.py`** — Core logic: `get_normalized_nodes()`, `normalize_message()` (for TRAPI), and equivalent CURIE discovery. Traverses Biolink Model ancestors for semantic type expansion.
- **`node_normalizer/loader/`** — the data loader: synchronous module-level functions (`load_all`, `load_compendium`, `load_conflation`, `merge_semantic_meta_data`) that read flat compendia files and populate Redis. Validates input against `resources/valid_data_format.json`. Batch size: 100,000. Invoked via the root `load.py`. `load_all` disables Redis periodic saves (`CONFIG SET save ""`) for the duration of the load — the backend DBs are write-once and persisted by a manual BGSAVE, so snapshotting during the load is wasted work. See `documentation/Loader.md`.
- **`node_normalizer/config.py`** — shared repo-relative paths (`config.json`, `redis_config.yaml`, `resources/`) and `get_config()`. Used by both frontend and loader.
Expand Down
2 changes: 2 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ COPY ./node_normalizer node_normalizer
COPY ./config.json config.json
COPY ./redis_config.yaml redis_config.yaml
COPY ./load.py load.py
# Agent instructions served at /llms.txt (node_normalizer/config.py: SKILL_PATH).
COPY ./skills skills

# install requirements (frontend + loader; the loader Helm chart runs load.py
# from this image)
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ which lists the number of times each prefix is used for a semantic type.

For examples of service usage, see the example [notebook](documentation/NodeNormalization.ipynb).

To use NodeNorm from an AI coding agent, see [Using with AI Agents / LLMs](documentation/LLMs.md);
any running instance also serves agent instructions at `/llms.txt`.

The Node normalization website leverages the [R3 (Redis-REST with referencing)](https://github.com/TranslatorSRI/r3) Redis data design and configuration.

Users can find the publicly available website at [service](https://nodenormalization-sri.renci.org/docs).
Expand Down
92 changes: 92 additions & 0 deletions documentation/LLMs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Using NodeNorm with AI Agents and LLMs

NodeNorm can be used directly from an AI coding agent (such as Claude Code) or any LLM-based tool
that can make HTTP requests. A skill file provides the instructions an agent needs to call NodeNorm
correctly — not just which endpoints exist, but which conflation settings to choose, how to batch a
large set of identifiers, and which behaviours will otherwise catch it out.

## The skill file

The skill lives at [`skills/nodenorm/SKILL.md`](../skills/nodenorm/SKILL.md). It covers:

- When NodeNorm is the right service and when you want [NameRes](https://github.com/NCATSTranslator/NameResolution) instead
- Normalizing a single identifier with `GET /get_normalized_nodes`, and how to read the response
- Normalizing many identifiers at once with `POST /get_normalized_nodes`
- Choosing GeneProtein and DrugChemical conflation deliberately, including the fact that the
defaults differ between GET and POST ([#398](https://github.com/NCATSTranslator/NodeNormalization/issues/398))
- Recipes for the common tasks: connecting an identifier to another database, normalizing a column
in a file, and testing whether two identifiers mean the same thing
- The non-obvious behaviours: unnormalizable CURIEs come back as `null` rather than being omitted,
the preferred label is not necessarily the preferred identifier's label, and the data is a fixed
Babel snapshot rather than a live database

Apart from the YAML frontmatter, which is packaging metadata for Claude Code, the file is plain
Markdown with no agent-specific syntax.

## Installing it in Claude Code

Claude Code discovers skills as `<skill-name>/SKILL.md` inside a `skills` directory — copy the whole
`skills/nodenorm/` directory, not just the file.

**For one project:**

```bash
mkdir -p .claude/skills
git clone --depth 1 --filter=blob:none --sparse \
https://github.com/NCATSTranslator/NodeNormalization.git /tmp/nn \
&& (cd /tmp/nn && git sparse-checkout set skills/nodenorm) \
&& cp -r /tmp/nn/skills/nodenorm .claude/skills/
```

**For every session**, use `~/.claude/skills` instead of `.claude/skills`.

Either way you can also just create the directory and save the file into it by hand:

```bash
mkdir -p ~/.claude/skills/nodenorm
curl -o ~/.claude/skills/nodenorm/SKILL.md \
https://raw.githubusercontent.com/NCATSTranslator/NodeNormalization/main/skills/nodenorm/SKILL.md
```

Claude Code will then offer it as `/nodenorm`, and will also load it automatically when a task
involves normalizing identifiers.

## Fetching it from a running instance

Every NodeNorm instance serves the same instructions at `/llms.txt`, with the frontmatter stripped:

```bash
curl https://nodenormalization-sri.renci.org/llms.txt
```

This is the quickest route for an agent that has been given a NodeNorm URL and nothing else, and it
guarantees the instructions match the deployed version. It is also linked from the API description,
so an agent reading `/openapi.json` will find it.

## Using it with other agents

For any agent that accepts a system prompt or context document, paste the contents of
[`skills/nodenorm/SKILL.md`](../skills/nodenorm/SKILL.md) (or the output of `/llms.txt`) into its
instructions. The skill is self-contained and depends on no external tooling.

## Example tasks

Once installed, these should work without further explanation:

```
Normalize MESH:D014867 and tell me what it is.

I have a UniProtKB accession, UniProtKB:P11532. What is the corresponding NCBI Gene ID?

Here's a CSV with an "identifier" column of mixed CURIEs. Normalize them all,
add a "preferred_id" column, and tell me which ones failed.

Are CHEBI:15377 and PUBCHEM.COMPOUND:962 the same thing?
```

## Keeping it accurate

The skill states specific facts about the API — parameter defaults, response shape, and worked
examples with real identifier counts. If you change any of those, update
[`skills/nodenorm/SKILL.md`](../skills/nodenorm/SKILL.md) in the same PR. `/llms.txt` is served
directly from that file, so there is only one copy to maintain.
2 changes: 1 addition & 1 deletion documentation/NodeNormalization.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
"{\n",
" \"status\": \"running\",\n",
" \"babel_version\": \"2025sep1\",\n",
" \"babel_version_url\": \"https://github.com/ncatstranslator/Babel/blob/master/releases/2025sep1.md\",\n",
" \"babel_version_url\": \"https://github.com/ncatstranslator/Babel/blob/main/releases/2025sep1.md\",\n",
" \"databases\": {\n",
" \"eq_id_to_id_db\": {\n",
" \"dbname\": \"id-id\",\n",
Expand Down
4 changes: 4 additions & 0 deletions node_normalizer/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
REDIS_CONFIG_PATH = REPO_ROOT / "redis_config.yaml"
RESOURCES_DIR = Path(__file__).parent / "resources"

# Agent-facing instructions for using this API, served at /llms.txt. Note that the
# Dockerfile has to COPY ./skills for this to exist in a deployed image.
SKILL_PATH = REPO_ROOT / "skills" / "nodenorm" / "SKILL.md"


def get_config() -> dict:
"""Load and return the parsed config.json."""
Expand Down
3 changes: 2 additions & 1 deletion node_normalizer/resources/openapi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ info:
to be useful, check /get_semantic_types, which lists the BioLink semantic types
for which normalization has been attempted, and /get_curie_prefixes, which lists
the number of times each prefix is used for a semantic type. You can find out more about these API methods at the
<a href="https://github.com/NCATSTranslator/NodeNormalization/blob/main/documentation/API.md">Node Normalization API documentation</a>.'
<a href="https://github.com/NCATSTranslator/NodeNormalization/blob/main/documentation/API.md">Node Normalization API documentation</a>.
If you are an AI agent, GET /llms.txt for instructions on using this service.'
license:
name: MIT
url: https://opensource.org/licenses/MIT
Expand Down
38 changes: 37 additions & 1 deletion node_normalizer/server.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""FastAPI server."""
import asyncio
import os
import re
import logging, warnings

from pathlib import Path
Expand All @@ -15,9 +16,10 @@
import yaml
from pydantic import BaseModel
from bmt import Toolkit
from starlette.responses import JSONResponse
from starlette.responses import JSONResponse, PlainTextResponse

from .apidocs import get_app_info, construct_open_api_schema
from .config import SKILL_PATH
from .model import (
SemanticTypes,
CuriePivot,
Expand Down Expand Up @@ -186,6 +188,40 @@ async def status() -> Dict:
}


#: Matches the YAML frontmatter block at the very top of a SKILL.md.
_FRONTMATTER = re.compile(r"\A---\n.*?\n---\n", re.DOTALL)


@app.get(
"/llms.txt",
summary="Instructions for using this API from an AI agent or LLM",
description="Returns a Markdown document explaining how to use this service to normalize "
"identifiers: which endpoint to call, how to read the response, how to batch, and "
"which conflation settings to choose. Intended to be fetched by an agent that has "
"been pointed at this instance and needs to work out how to use it. The same "
"document is maintained as a "
"<a href=\"https://github.com/NCATSTranslator/NodeNormalization/tree/master/skills/nodenorm\">"
"skill</a> in the NodeNorm repository.",
response_class=PlainTextResponse,
response_description="Markdown instructions for using this API.",
)
async def llms_txt() -> PlainTextResponse:
"""Serve the agent instructions from skills/nodenorm/SKILL.md."""
try:
skill = SKILL_PATH.read_text(encoding="utf-8")
except OSError:
# Most likely an image built without `COPY ./skills`; say so rather than 500ing.
logger.warning(f"Could not read agent instructions from {SKILL_PATH}.")
raise HTTPException(
status_code=404,
detail="Agent instructions are not available on this instance. They can be read at "
"https://github.com/NCATSTranslator/NodeNormalization/tree/master/skills/nodenorm",
)

# The frontmatter is Claude Code packaging metadata; it is noise in an llms.txt.
return PlainTextResponse(_FRONTMATTER.sub("", skill, count=1).lstrip("\n"))


@app.post(
"/query",
summary="Normalizes every identifier in a TRAPI message",
Expand Down
Loading
Loading