Fix vision captioning on OpenAI-compatible backends: media parsing + image format

[LegalInspiration]

NOTE: An AI model, specifically Claude Code, was used to investigate this bug, test fixes, and generate the first draft of this report. A human being read every single line in it, so if there's something wrong with it, blame the meatsack.

ALSO NOTE: CodeRabbit found two issues related to the code I touched. While I didn't cause any new issues, I didn't want to leave them hanging, especially one flagged as major, so there is a second commit which addresses them.

Summary

After several attempts and different configuration pass, I could not get captioning or vision mode to work while using my local llama.cpp instance, a local llama.cpp llama-server exposing an OpenAI-style endpoint. Symptoms: either the model insisted that no image was uploaded, or it confidently returned a detailed description that doesn't match the actual uploaded image, differently wrong on every attempt. In my limited experience, this usually indicates that the vision model doesn't have an actual/parsable image to work from, so it basically hallucinates one from its weights.

Investigation, for which I also used Claude Code, found two independent, confirmed bugs, both reproduced directly against /API/MagicPromptPhoneHome (bypassing the browser) and fixes for both are presented here.

Please note: the README specifically already flagged "If you are using any LLM service other than Ollama, I cannot guarantee that it will work." This PR is exactly that unsupported path — filed with a full diagnosis and working patch in case it's useful, not a complaint.

Bug 1 — media parsing was case-sensitive, but the client always sent lowercase keys

WebAPI/LLMAPICalls.cs, MagicPromptPhoneHome:

messageContent.Media = mediaToken.ToObject<List<MediaContent>>();

MediaContent is PascalCase (Type/Data/MediaType). The actual browser JS (Assets/magicprompt.js, createRequestPayload) always sends lowercase:

media: image ? [{ type: "base64", data: image, mediaType: ... }] : null,

Confirmed via direct testing: lowercase keys always produced an empty/unbound Media list — no exception, request just proceeded as if no image was ever attached. Sending an Identical payload with PascalCase keys parsed fine. Since the real client never sent PascalCase, every real vision request from the UI hit this — no image ever actually reached the vision backend in normal use, regardless of anything downstream.

Fix: explicit case-insensitive property lookup instead of the case-sensitive ToObject<T>() call.

Bug 2 — WEBP is mishandled by at least one real vision model's decode pipeline, independent of quality/resolution

Once bug 1 is fixed and an image actually attaches, captioning becomes unreliable rather than absent — confident, detailed, differently-wrong descriptions each time. Isolated via systematic A/B testing, holding every variable constant except one at a time:

• Ruled out: data corruption (image verified byte-identical at every processing stage via file dumps), prompt-cache pollution (still failed after a full model-server restart), request structure/prompt wording (held constant across tests), Image complexity (tested an image of nothing but a white circle on a black background and still got exactly the same errors), and plain low quality/resolution (still failed at WEBP quality 95, near-lossless).
• Decisive test: the exact same image, same model, only the format changed — WEBP fails, JPEG succeeds, repeatedly.

This lines up with something already in the codebase — OllamaRequestBody already hardcodes JPEG:

images = content.Media.Select(m => CompressImageForVision(m, "JPG")).ToArray()

— while OpenAICompatibleRequestBody defaulted to WEBP for openai/openaiapi/openrouter. My read: Ollama's choice wasn't a deliberate fix for this, it just never touched WEBP in the first place, so it never hit whatever's going on in that decode path (plausibly a llama.cpp clip.cpp/mtmd-side WEBP handling gap, not something fixable in this extension beyond avoiding the format).

Fix: OpenAI-compatible path now defaults to JPEG, matching Ollama's already-working choice.

Also tuned: two settings already flagged with your own TODOs

CompressImageForVision: maxDimension 256→1024, quality 40/60→90. 256px/q40 is likely fine for older CLIP-ViT-era encoders (224–336px native) but throws away real detail for newer higher-resolution-capable models.

OpenAICompatibleRequestBody, vision branch only: temperature 1.0→0.2. Factual captioning wants low/deterministic sampling; high temperature turned a marginal image into confidently-wrong, differently-wrong-each-time answers. Text-only chat and Ollama's own path are untouched, deliberately out of scope.

Testing

Unit-level (isolated, no SwarmUI involved): built a synthetic PNG (blue background, alpha=0 circular region with hidden RGB=(0,255,0)) and ran it through the exact same ImageSharp SaveAsJpeg call this fix uses, in isolation. Confirmed the alpha channel is silently dropped and the hidden green shows through (see caveat below) — this is a genuine ImageSharp/JPEG behavior, not something introduced by this patch, but worth knowing about given the format change.

Integration-level (real, live SwarmUI, actual vision model): applied the fix to a running install, rebuilt the extension, and tested through the real UI in all three MagicPrompt modes — chat, vision-prompt, and auto-caption — against a local OpenAIAPI-compatible backend. Then re-verified directly via /API/MagicPromptPhoneHome using the exact lowercase-key JSON shape the real browser sends: 3/3 clean, accurate, consistent descriptions of a real test image, versus repeated failures/wrong descriptions before the fix, across multiple structural variations.

Known caveat — not fixed here, flagging for visibility

JPEG has no alpha channel. Once bug 2's fix lands, any image with real transparency will have its alpha silently dropped by the encoder, and whatever RGB happens to sit under fully-transparent pixels can "leak through" into the encoded JPEG — potentially producing a confidently-described object that was never actually visible in the source image. Confirmed both in isolation (unit test above) and end-to-end: sending that same synthetic image through the live vision pipeline produced a detailed, confident description of "a bright green circle on a solid blue background" — an object that doesn't exist in the actual image; the green was invisible authoring data hidden under alpha=0.

This isn't something this PR introduces (WEBP has the same limitation for lossy mode, and was already broken for an unrelated reason), and it wasn't hit in my own use case, so I left it out of scope. A real fix looks straightforward, though: detect an alpha channel and explicitly composite it onto a solid background (or fall back to PNG) before the JPEG encode, rather than leaving the dropped-alpha behavior undefined. Happy to take a pass at that too if it's wanted — didn't want to bundle an untested fix for a problem I didn't personally hit into the same PR as the confirmed, tested one.

Summary by CodeRabbit

• Bug Fixes
  • Improved compatibility with browser-submitted media messages, ensuring image and media content are recognized correctly.
  • Updated vision image processing to preserve more detail in larger images.
  • Standardized JPEG image handling for compatible vision requests.
  • Improved vision response consistency by using more focused sampling settings.

[coderabbitai]

📝 Walkthrough

Walkthrough

Vision requests now use larger JPEG-compressed images and lower temperature values. MagicPromptPhoneHome manually maps media payload fields with case-insensitive property matching.

Changes

Vision request updates

Layer / File(s)	Summary
Vision encoding and sampling BackendSchema.cs	Vision image downscaling increases to 1024px with quality 90; non-PNG images use JPEG data URLs; vision request temperature changes to 0.2 for both seed paths.

Media payload parsing

Layer / File(s)	Summary
Case-insensitive media mapping WebAPI/LLMAPICalls.cs	MagicPromptPhoneHome iterates media objects and maps type, data, and mediaType using case-insensitive property lookups.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: kalebbroo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main fix for vision captioning on OpenAI-compatible backends, covering media parsing and image format changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@BackendSchema.cs`:
- Around line 166-171: Update the image URL construction in the media content
flow around CompressImageForVision so fallback data retains its actual MIME type
instead of always being labeled according to preferPngForBase64. Return the
compressed/unchanged encoded data together with the resolved MIME type, or use
media.MediaType for fallback URLs, while preserving the current preferred
PNG/JPEG behavior for successful conversions.
- Around line 77-91: Update the XML documentation and return comment for the
image compression method containing maxDimension and quality so they document
the new JPG format alongside the supported PNG and WEBP formats, and remove the
stale statement that the method returns WEBP specifically. Keep the
documentation aligned with the method’s actual format-selection behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ab7eaef7-5cc6-4a10-842d-58da84ae5df6

📥 Commits

Reviewing files that changed from the base of the PR and between 8f3f842 and 6e44f93.

📒 Files selected for processing (2)

• BackendSchema.cs
• WebAPI/LLMAPICalls.cs