Skip to content

fix(litellm): provision Vertex sandbox runtime - #985

Open
kywch wants to merge 2 commits into
mainfrom
fix/vertex-litellm-sandbox
Open

fix(litellm): provision Vertex sandbox runtime#985
kywch wants to merge 2 commits into
mainfrom
fix/vertex-litellm-sandbox

Conversation

@kywch

@kywch kywch commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Provision Vertex dependencies and ADC credentials inside sandboxed LiteLLM proxies.

Root cause

Vertex LiteLLM runs in a separate Daytona sandbox venv. #978 added google-cloud-aiplatform to BenchFlow host dependencies, but host packages are not visible in that proxy venv. The proxy also needed a local ADC credential-file path and explicit Vertex project/location.

Changes

  • Install google-cloud-aiplatform only for sandbox Vertex routes.
  • Stage ADC JSON as a sandbox-local credential file.
  • Remove raw ADC JSON before proxy launch config serialization.
  • Forward Vertex project/location to LiteLLM.

Scope

Complements #978. Does not change host provisioning or non-Vertex routes.

Validation

  • 74 focused tests passed.
  • git diff --check passed.

@kywch
kywch deployed to pypi-internal-preview August 14, 2026 02:01 — with GitHub Actions Active
@kywch
kywch deployed to pypi-internal-preview August 14, 2026 02:05 — with GitHub Actions Active
@kywch
kywch marked this pull request as ready for review August 14, 2026 02:11
@kywch
kywch deployed to pypi-internal-preview August 14, 2026 02:11 — with GitHub Actions Active
@Galius5136

Copy link
Copy Markdown

Took an independent look at fc842f14 and also ran the sandbox path against a real GCP project.

Local checks were clean: 74 touched tests, 261 wider provider tests, plus ruff, ruff format, and ty all passing.

Live E2E with Docker + google-vertex/gemini-2.5-flash also worked:

  • sandbox-local LiteLLM proxy installed google-cloud-aiplatform
  • ADC staged as a 0600 file
  • raw GOOGLE_APPLICATION_CREDENTIALS_JSON removed from launch config
  • correct Vertex project/location reached the proxy config
  • real model calls completed successfully through the proxy

Two small non-blocking notes:

  • This changes the effective Vertex location from LiteLLM's implicit default to BenchFlow's GOOGLE_CLOUD_LOCATION default (global). I verified global works for gemini-2.5-flash, but not live for anthropic-vertex/claude-sonnet-4-6, which is what the new tests use.
  • If the sandbox LiteLLM install fails after ADC staging, the credential file remains in the runtime dir until sandbox teardown. I reproduced this with the fake sandbox; moving the install inside the existing cleanup scope would close that window.

From my side this looks good to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants