Skip to content

[codex] Preserve provider usage metadata - #1322

Open
Tanisha-Katara wants to merge 1 commit into
huggingface:mainfrom
Tanisha-Katara:codex/preserve-provider-usage-metadata
Open

[codex] Preserve provider usage metadata#1322
Tanisha-Katara wants to merge 1 commit into
huggingface:mainfrom
Tanisha-Katara:codex/preserve-provider-usage-metadata

Conversation

@Tanisha-Katara

Copy link
Copy Markdown

Summary

  • add optional provider usage metadata to ModelResponse
  • preserve that metadata through ModelResponse slicing and cache serialization
  • populate it from Hugging Face inference-provider chat completion usage fields

Why

Inference providers can return billing/token usage details such as prompt tokens, completion tokens, and nested cache counters. LightEval currently keeps token ids for evaluation, but drops provider-reported usage metadata from inference-provider responses. Preserving the raw usage payload makes saved details and cached samples more useful for downstream cost and cache analysis without adding pricing logic or a dependency on any external cost tool.

Tests

  • .venv/bin/python -m pytest tests/unit/models/test_model_output.py tests/unit/models/endpoints/test_inference_providers_model.py -q
  • .venv/bin/python -m compileall -q src/lighteval/models/model_output.py src/lighteval/models/endpoints/inference_providers_model.py tests/unit/models/test_model_output.py tests/unit/models/endpoints/test_inference_providers_model.py
  • .venv/bin/ruff check src/lighteval/models/model_output.py src/lighteval/models/endpoints/inference_providers_model.py tests/unit/models/test_model_output.py tests/unit/models/endpoints/test_inference_providers_model.py
  • .venv/bin/ruff format --check src/lighteval/models/model_output.py src/lighteval/models/endpoints/inference_providers_model.py tests/unit/models/test_model_output.py tests/unit/models/endpoints/test_inference_providers_model.py
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant