Skip to content

Align nvMolKit skill with BioNeMo expected representation and add evals. #232

Open
scal444 wants to merge 1 commit into
NVIDIA-BioNeMo:mainfrom
scal444:port-nvmolkit-skill-evals
Open

Align nvMolKit skill with BioNeMo expected representation and add evals. #232
scal444 wants to merge 1 commit into
NVIDIA-BioNeMo:mainfrom
scal444:port-nvmolkit-skill-evals

Conversation

@scal444

@scal444 scal444 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Signed-off-by: Kevin Boyd <kboyd@nvidia.com>
@scal444
scal444 requested a review from evasnow1992 July 24, 2026 20:42
@greptile-apps

greptile-apps Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR renames the skill directory from agent-skills/nvmolkit-usage/ to skills/nvmolkit-usage/, updates all references in docs and tests, and adds two new eval definition files (evals.json and trigger_evals.json) along with corresponding test coverage.

  • SKILL.md is a pure rename with no content changes; README.md and docs/agent_skill.rst are updated to reflect the new path.
  • skills/nvmolkit-usage/evals/evals.json introduces 11 well-structured eval cases covering installation, porting, from-scratch workflows, and performance debugging; trigger_evals.json adds 5 trigger/no-trigger cases.
  • test_skill.py gains two new tests that validate the eval schemas, though one hardcodes the eval count to 11 and the trigger-evals test lacks the per-entry schema checks present in the full evals test.

Confidence Score: 4/5

Safe to merge; all functional changes are a directory rename and new data files with no behavioural impact on library code.

The hardcoded count assert len(data["evals"]) == 11 in the new test will require a manual update each time an eval is added, creating unnecessary churn. The trigger-evals test also skips per-entry field validation. Neither issue affects runtime behaviour, but they leave gaps in the test suite's robustness.

nvmolkit/tests/test_skill.py — hardcoded eval count and incomplete trigger-evals schema validation.

Important Files Changed

Filename Overview
nvmolkit/tests/test_skill.py Adds two new test functions for evals.json and trigger_evals.json; contains a hardcoded eval count (11) and asymmetric schema validation for trigger_evals entries
skills/nvmolkit-usage/evals/evals.json New file with 11 eval definitions covering installation, porting, and from-scratch workflows; schema is consistent and all required fields are present
skills/nvmolkit-usage/evals/trigger_evals.json New file with 5 trigger eval cases (3 positive, 2 negative); straightforward schema and good coverage of both trigger and no-trigger scenarios
skills/nvmolkit-usage/SKILL.md Renamed from agent-skills/ to skills/; no content changes
README.md Path reference updated from agent-skills/ to skills/ and description updated to mention evaluation definitions
docs/agent_skill.rst Path reference updated from agent-skills/ to skills/ to match the directory rename

Reviews (1): Last reviewed commit: "Port nvMolKit skill evaluations" | Re-trigger Greptile

Comment on lines +71 to +73
assert data["skill_name"] == "nvmolkit-usage"
assert len(data["evals"]) == 11
assert len({entry["id"] for entry in data["evals"]}) == len(data["evals"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The exact count 11 is hardcoded here, so every new eval added to evals.json will break this test even though nothing is wrong. The uniqueness check on the same line already guards against duplicates and accidental deletions; the count assertion adds friction without catching real schema issues. Dropping it (or replacing with a lower-bound check) would make the test resilient to future additions.

Suggested change
assert data["skill_name"] == "nvmolkit-usage"
assert len(data["evals"]) == 11
assert len({entry["id"] for entry in data["evals"]}) == len(data["evals"])
assert data["skill_name"] == "nvmolkit-usage"
assert len(data["evals"]) > 0
assert len({entry["id"] for entry in data["evals"]}) == len(data["evals"])

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +92 to +97
def test_skill_trigger_evals_cover_positive_and_negative_cases() -> None:
data = json.loads(TRIGGER_EVALS_PATH.read_text())

assert data["skill_name"] == "nvmolkit-usage"
outcomes = {entry["expected"] for entry in data["trigger_evals"]}
assert outcomes == {"trigger", "no_trigger"}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 test_skill_trigger_evals_cover_positive_and_negative_cases only checks that both outcome labels are present, but does no per-entry schema validation. In contrast, the evals test verifies unique IDs, non-empty prompts, required field sets, and correct prefixes. A malformed trigger entry (missing id, empty prompt, or a typo in expected) would go undetected. Adding field-presence and non-empty checks for each trigger_evals entry would bring parity with the full eval test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant