Align nvMolKit skill with BioNeMo expected representation and add evals. #232
Align nvMolKit skill with BioNeMo expected representation and add evals. #232scal444 wants to merge 1 commit into
Conversation
Signed-off-by: Kevin Boyd <kboyd@nvidia.com>
|
| Filename | Overview |
|---|---|
| nvmolkit/tests/test_skill.py | Adds two new test functions for evals.json and trigger_evals.json; contains a hardcoded eval count (11) and asymmetric schema validation for trigger_evals entries |
| skills/nvmolkit-usage/evals/evals.json | New file with 11 eval definitions covering installation, porting, and from-scratch workflows; schema is consistent and all required fields are present |
| skills/nvmolkit-usage/evals/trigger_evals.json | New file with 5 trigger eval cases (3 positive, 2 negative); straightforward schema and good coverage of both trigger and no-trigger scenarios |
| skills/nvmolkit-usage/SKILL.md | Renamed from agent-skills/ to skills/; no content changes |
| README.md | Path reference updated from agent-skills/ to skills/ and description updated to mention evaluation definitions |
| docs/agent_skill.rst | Path reference updated from agent-skills/ to skills/ to match the directory rename |
Reviews (1): Last reviewed commit: "Port nvMolKit skill evaluations" | Re-trigger Greptile
| assert data["skill_name"] == "nvmolkit-usage" | ||
| assert len(data["evals"]) == 11 | ||
| assert len({entry["id"] for entry in data["evals"]}) == len(data["evals"]) |
There was a problem hiding this comment.
The exact count
11 is hardcoded here, so every new eval added to evals.json will break this test even though nothing is wrong. The uniqueness check on the same line already guards against duplicates and accidental deletions; the count assertion adds friction without catching real schema issues. Dropping it (or replacing with a lower-bound check) would make the test resilient to future additions.
| assert data["skill_name"] == "nvmolkit-usage" | |
| assert len(data["evals"]) == 11 | |
| assert len({entry["id"] for entry in data["evals"]}) == len(data["evals"]) | |
| assert data["skill_name"] == "nvmolkit-usage" | |
| assert len(data["evals"]) > 0 | |
| assert len({entry["id"] for entry in data["evals"]}) == len(data["evals"]) |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| def test_skill_trigger_evals_cover_positive_and_negative_cases() -> None: | ||
| data = json.loads(TRIGGER_EVALS_PATH.read_text()) | ||
|
|
||
| assert data["skill_name"] == "nvmolkit-usage" | ||
| outcomes = {entry["expected"] for entry in data["trigger_evals"]} | ||
| assert outcomes == {"trigger", "no_trigger"} |
There was a problem hiding this comment.
test_skill_trigger_evals_cover_positive_and_negative_cases only checks that both outcome labels are present, but does no per-entry schema validation. In contrast, the evals test verifies unique IDs, non-empty prompts, required field sets, and correct prefixes. A malformed trigger entry (missing id, empty prompt, or a typo in expected) would go undetected. Adding field-presence and non-empty checks for each trigger_evals entry would bring parity with the full eval test.
This will be consumed by https://github.com/NVIDIA-BioNeMo/bionemo-agent-toolkit