[LEADS-557] Document MLflow storage backend setup and behavior

[bsatapat-jpg]

Description

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up service version
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Unit tests improvement

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

• Assisted-by: Grok 4.5
• Generated by: Cursor

Related Tickets & Documents

• Related Issue # https://redhat.atlassian.net/browse/LEADS-557
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

• Documentation
  • Expanded storage configuration docs to cover optional Langfuse and MLflow backends.
  • Added MLflow backend setup instructions, required environment variables, and YAML configuration examples (including “File + MLflow”).
  • Clarified scoring/export semantics for score=None outcomes and described MLflow/Langfuse error-handling behavior.
  • Updated observability/export guidance and generated report documentation to reference the new backends.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: af5899c7-d0cd-4da3-99f6-95c78e5c4075

📥 Commits

Reviewing files that changed from the base of the PR and between ce3d4b3 and 5dc7fe9.

📒 Files selected for processing (4)

• README.md
• config/system.yaml
• docs/EVALUATION_GUIDE.md
• docs/configuration.md

🚧 Files skipped from review as they are similar to previous changes (4)

• docs/EVALUATION_GUIDE.md
• config/system.yaml
• README.md
• docs/configuration.md

---

Walkthrough

MLflow is documented as an optional storage backend, with configuration examples, logging behavior, error handling, and UI guidance. README and evaluation documentation now reference Langfuse and MLflow storage and export options.

Changes

MLflow storage documentation

Layer / File(s)	Summary
MLflow configuration and behavior reference config/system.yaml, docs/configuration.md	Adds a commented MLflow configuration and documents installation, settings, logged data, error handling, scoring behavior, UI usage, and File + MLflow configuration.
Backend documentation references README.md, docs/EVALUATION_GUIDE.md	Lists Langfuse and MLflow as optional storage and export integrations with installation and configuration links.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• lightspeed-core/lightspeed-evaluation#261: Documents related Langfuse storage backend behavior.
• lightspeed-core/lightspeed-evaluation#269: Also expands the storage backends documentation in docs/EVALUATION_GUIDE.md.
• lightspeed-core/lightspeed-evaluation#284: Adds related MLflow backend configuration and behavior that this documentation describes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: documenting MLflow storage backend setup and behavior.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/configuration.md`:
- Around line 476-483: Update the MLflow configuration documentation near the
tracing and results logging table to add a prominent warning that queries,
responses, expected values, reasons, and metadata may contain PII or secrets and
are sent to the external MLflow backend. Document retention considerations and
provide guidance to redact sensitive fields or disable tracing/logging before
recommending this setup.
- Around line 468-470: Update the MLflow documentation blockquote near the
“Tracking URI” and “Error handling” paragraphs to keep it contiguous: either
prefix the blank line with the blockquote marker or remove the blank line,
ensuring MD028 passes.
- Around line 489-493: Update the MLflow UI documentation example to remove the
remote MLFLOW_TRACKING_URI value from --backend-store-uri. Clarify that an
existing remote tracking server’s UI should be opened directly, while locally
starting mlflow ui requires a local backend/store URI.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d940ea6a-83a9-46ca-8f70-02bb4f455b39

📥 Commits

Reviewing files that changed from the base of the PR and between d5fcbc0 and ce3d4b3.

📒 Files selected for processing (4)

• README.md
• config/system.yaml
• docs/EVALUATION_GUIDE.md
• docs/configuration.md

[xmican10]

Thanks!! LGTM, noted potentional typos..

[asamal4]

Thanks, minor comment