feat(benchmark): Add vllm_performance benchmark for prithvi by mgazz · Pull Request #155 · IBM/algorithm-nexus

mgazz · 2026-06-29T12:17:16Z

No description provided.

DRL-NextGen · 2026-06-29T12:26:28Z

Checks Summary

Last run: 2026-08-06T15:13:04.567Z

Mend Unified Agent vulnerability scan found 7 vulnerabilities:

Severity	Identifier	Package	Details	Fix
🔺 High	CVE-2026-65918	torchvision-0.26.0-cp312-cp312-manylinux_2_28_x86_64.whl	PyTorch torchvision through 0.28.0, fixed in commit 4e05dc2, contains an out-of-bounds heap read vul... PyTorch torchvision through 0.28.0, fixed in commit 4e05dc2, contains an out-of-bounds heap read vulnerability in the GIF decoder's read_from_tensor callback that passes unclamped length to memcpy. Attackers can supply malicious or truncated GIF files to cause denial of service via segmentation fault or disclose adjacent heap memory contents.	Not Available
🔺 High	CVE-2026-58659	lightning-2.6.5-py3-none-any.whl	PyTorch Lightning through 2.6.5, fixed in commit d710d68, contains a remote code execution vulnerabi... PyTorch Lightning through 2.6.5, fixed in commit d710d68, contains a remote code execution vulnerability in the _load_state function that imports and executes attacker-controlled module names from checkpoint _instantiator hyperparameters. Attackers can craft malicious checkpoint files that bypass weights_only=True protections to execute arbitrary code when LightningModule.load_from_checkpoint is called.	Not Available
🔺 High	CVE-2025-14920	transformers-5.14.1-py3-none-any.whl	Hugging Face Transformers Perceiver Model Deserialization of Untrusted Data Remote Code Execution Vu... Hugging Face Transformers Perceiver Model Deserialization of Untrusted Data Remote Code Execution Vulnerability. This vulnerability allows remote attackers to execute arbitrary code on affected installations of Hugging Face Transformers. User interaction is required to exploit this vulnerability in that the target must visit a malicious page or open a malicious file. The specific flaw exists within the parsing of model files. The issue results from the lack of proper validation of user-supplied data, which can result in deserialization of untrusted data. An attacker can leverage this vulnerability to execute code in the context of the current user. Was ZDI-CAN-25423.	Not Available
🔷 Medium	CVE-2026-65920	diffusers-0.39.0-py3-none-any.whl	Diffusers through 0.39.0, fixed in commit cee298c, contains a path traversal vulnerability in the _g... Diffusers through 0.39.0, fixed in commit cee298c, contains a path traversal vulnerability in the _get_checkpoint_shard_files function that allows attackers to read arbitrary files by supplying malicious weight_map values in model index JSON. Attackers can use ../ sequences or absolute paths in weight_map entries to escape the model directory and read safetensors files outside the intended location during model loading.	Not Available
🔷 Medium	CVE-2025-3000	torch-2.11.0-cp312-cp312-manylinux_2_28_x86_64.whl	A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function to... A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function torch.jit.script. The manipulation leads to memory corruption. It is possible to launch the attack on the local host. The exploit has been disclosed to the public and may be used.	torch - 2.13.0
🔷 Medium	CVE-2026-59890	setuptools-80.10.2-py3-none-any.whl	setuptools is a package that allows users to download, build, install, upgrade, and uninstall Python... setuptools is a package that allows users to download, build, install, upgrade, and uninstall Python packages. Prior to 83.0.0, FileList applied MANIFEST.in exclude, global-exclude, recursive-exclude, and prune directives by matching compiled glob patterns against on-disk file names without Unicode normalization, so on macOS APFS or HFS+ an NFD file name could bypass an NFC exclusion rule and be packed into a source distribution. This issue is fixed in version 83.0.0.	Upgrade to version setuptools - 83.0.0,setuptools - 83.0.0,https://github.com/pypa/setuptools.git - 83.0.0
🔸 Low	CVE-2025-63396	torch-2.11.0-cp312-cp312-manylinux_2_28_x86_64.whl	An issue was discovered in PyTorch v2.5 and v2.7.1. Omission of profiler.stop() can cause torch.prof... An issue was discovered in PyTorch v2.5 and v2.7.1. Omission of profiler.stop() can cause torch.profiler.profile (PythonTracer) to crash or hang during finalization, leading to a Denial of Service (DoS).	Not Available

christian-pinto · 2026-06-30T15:00:30Z

@mgazz We have an issue currently blocking our CI. I will have tests and benchmarks test runs done as soon as we get the CI going again.

christian-pinto · 2026-07-03T12:10:49Z

@mgazz please merge with the latest master and then run

git restore --source origin/main -- uv.lock
uv lock

Then add the new uv.lock and commit. Make sure the pre-commit hooks are installed becuase they will re-generate the requirement files.

christian-pinto · 2026-07-03T12:59:05Z

/run_benchmarks

DRL-NextGen · 2026-07-03T13:01:59Z

✅ Benchmark Run (2026-07-03 13:01 UTC)

Status	Duration	Ray Job ID	Instance
✅	45s	raysubmit_eBX6vxZcWXP7rCCq	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

DRL-NextGen · 2026-07-03T13:07:06Z

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

christian-pinto · 2026-07-08T08:30:23Z

/run_benchmarks

DRL-NextGen · 2026-07-08T08:33:27Z

✅ Benchmark Run (2026-07-08 08:33 UTC)

Status	Duration	Ray Job ID	Instance
✅	-	raysubmit_kCSfr6uEANHdn6qW	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

DRL-NextGen · 2026-07-08T08:39:15Z

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

christian-pinto · 2026-07-08T08:46:20Z

/run_benchmarks

DRL-NextGen · 2026-07-08T08:47:55Z

🔁 Benchmark Run (2026-07-08 08:47 UTC)

Status	Ray Job ID	Instance
🔄	raysubmit_7U4393QH8DGxgrWh	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

christian-pinto · 2026-07-08T09:04:47Z

/run_benchmarks

DRL-NextGen · 2026-07-08T09:07:25Z

🔁 Benchmark Run (2026-07-08 09:07 UTC)

Status	Ray Job ID	Instance
❌	-	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

christian-pinto · 2026-07-08T09:45:36Z

/run_benchmarks

DRL-NextGen · 2026-07-08T09:48:13Z

✅ Benchmark Run (2026-07-08 09:48 UTC)

Status	Duration	Ray Job ID	Instance
✅	175s	raysubmit_h8hjk6QypreGPMdE	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

DRL-NextGen · 2026-07-08T09:53:46Z

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

christian-pinto · 2026-07-15T09:50:23Z

/run_benchmarks

DRL-NextGen · 2026-07-15T09:53:02Z

🔁 Benchmark Run (2026-07-15 09:53 UTC)

Status	Ray Job ID	Instance
❌	-	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

christian-pinto · 2026-07-15T10:11:20Z

/run_benchmarks

DRL-NextGen · 2026-07-15T10:13:17Z

✅ Benchmark Run (2026-07-15 10:13 UTC)

Status	Duration	Ray Job ID	Instance
✅	-	raysubmit_xiPY8v5tG6iww75x	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

DRL-NextGen · 2026-07-15T10:13:48Z

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

DRL-NextGen · 2026-07-15T10:14:32Z

🔁 Benchmark Run (2026-07-15 10:14 UTC)

Status	Ray Job ID	Instance
🔄	raysubmit_Z9NN62AP4aewAPgR	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

christian-pinto · 2026-07-15T10:37:34Z

/run_benchmarks

DRL-NextGen · 2026-07-15T10:40:30Z

🔁 Benchmark Run (2026-07-15 10:40 UTC)

Status	Ray Job ID	Instance
🔄	raysubmit_P2A2J3DyeAnUAii3	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

christian-pinto · 2026-07-15T10:47:28Z

/run_benchmarks

DRL-NextGen · 2026-07-15T10:50:00Z

✅ Benchmark Run (2026-07-15 10:49 UTC)

Status	Duration	Ray Job ID	Instance
✅	-	raysubmit_PWj6xWHrqk2HCMTJ	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

DRL-NextGen · 2026-07-15T10:58:10Z

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

mgazz · 2026-07-20T10:21:02Z

Currently blocked on this vLLM PR to restore ability to run vllm benchmarks without tokeniser

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

- bump ado-vllm-performance version to 1.13.2 - update experiment name to geospatial-vllm-bench-deployment-custom-dataset Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

mgazz · 2026-08-06T16:01:34Z

/run_benchmarks

mgazz force-pushed the prithvi branch from 94c5bbd to 64cf1ec Compare June 29, 2026 12:19

michael-johnston requested a review from christian-pinto June 30, 2026 10:06

christian-pinto added the ci Enable CI integration label Jun 30, 2026

mgazz added 3 commits August 5, 2026 14:06

feat(benchmark): Add vllm_performance benchmark for prithvi

fe626e6

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

build(benchmark): bump terratorch benchmark package version to >=1.9.6

90759c3

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

build(benchmark): update Prithvi benchmark package

d7c66e7

- bump ado-vllm-performance version to 1.13.2 - update experiment name to geospatial-vllm-bench-deployment-custom-dataset Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

mgazz force-pushed the prithvi branch from f4b6db7 to d7c66e7 Compare August 6, 2026 12:05

mgazz added 2 commits August 6, 2026 13:38

fix(benchmark): update Prithvi experiment indentifier

0b1dbb5

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

fix(benchmark): add experimentVersion to the Prithvi experiment

c64418e

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>

Uh oh!

Conversation

mgazz commented Jun 29, 2026

Uh oh!

DRL-NextGen commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christian-pinto commented Jun 30, 2026

Uh oh!

christian-pinto commented Jul 3, 2026

Uh oh!

christian-pinto commented Jul 3, 2026

Uh oh!

DRL-NextGen commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Benchmark Run (2026-07-03 13:01 UTC)

Uh oh!

DRL-NextGen commented Jul 3, 2026

Uh oh!

christian-pinto commented Jul 8, 2026

Uh oh!

DRL-NextGen commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Benchmark Run (2026-07-08 08:33 UTC)

Uh oh!

DRL-NextGen commented Jul 8, 2026

Uh oh!

christian-pinto commented Jul 8, 2026

Uh oh!

DRL-NextGen commented Jul 8, 2026

🔁 Benchmark Run (2026-07-08 08:47 UTC)

Uh oh!

christian-pinto commented Jul 8, 2026

Uh oh!

DRL-NextGen commented Jul 8, 2026

🔁 Benchmark Run (2026-07-08 09:07 UTC)

Uh oh!

christian-pinto commented Jul 8, 2026

Uh oh!

DRL-NextGen commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Benchmark Run (2026-07-08 09:48 UTC)

Uh oh!

DRL-NextGen commented Jul 8, 2026

Uh oh!

christian-pinto commented Jul 15, 2026

Uh oh!

DRL-NextGen commented Jul 15, 2026

🔁 Benchmark Run (2026-07-15 09:53 UTC)

Uh oh!

christian-pinto commented Jul 15, 2026

Uh oh!

DRL-NextGen commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Benchmark Run (2026-07-15 10:13 UTC)

Uh oh!

DRL-NextGen commented Jul 15, 2026

Uh oh!

DRL-NextGen commented Jul 15, 2026

🔁 Benchmark Run (2026-07-15 10:14 UTC)

Uh oh!

christian-pinto commented Jul 15, 2026

Uh oh!

DRL-NextGen commented Jul 15, 2026

🔁 Benchmark Run (2026-07-15 10:40 UTC)

Uh oh!

christian-pinto commented Jul 15, 2026

Uh oh!

DRL-NextGen commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Benchmark Run (2026-07-15 10:49 UTC)

Uh oh!

DRL-NextGen commented Jul 15, 2026

Uh oh!

mgazz commented Jul 20, 2026

Uh oh!

mgazz commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

DRL-NextGen commented Jun 29, 2026 •

edited

Loading

DRL-NextGen commented Jul 3, 2026 •

edited

Loading

DRL-NextGen commented Jul 8, 2026 •

edited

Loading

DRL-NextGen commented Jul 8, 2026 •

edited

Loading

DRL-NextGen commented Jul 15, 2026 •

edited

Loading

DRL-NextGen commented Jul 15, 2026 •

edited

Loading