---

Robomous VisionSet

VisionSet is an open-source, local-first, SDK-first tool by Robomous for creating, curating, and versioning computer-vision training datasets. Today it targets 2D image annotation; the domain model is built for a Physical AI roadmap — 3D point clouds, lane labeling, and multimodal data land on the same foundations. Your data stays on your machines, every surface (UI, CLI, MCP) is a thin client of the same SDK, and the release artifact is a plain pip package.

What it does

Point it at a folder of images or a clip of video, label them, and hand a trainer a dataset — without a server, an account, or your pixels leaving the machine.

Ingest	folders and video. Frames are cut, hashed and stored by content, so the same file twice is one asset and a re-run costs nothing.
Annotate	boxes, polygons and classification tags in the browser, with undo/redo, keyboard-first tools, and a headless engine underneath that the UI is only one renderer of.
Version	schema versions are immutable and every label records the one it was judged against. A release freezes the whole thing into a manifest; publish twice from unchanged data and the bytes are identical.
Split	a stored recipe rather than a materialised assignment, keyed on content hash — so two copies of one image cannot straddle a train/test boundary.
Export	addressed to the model you will train — the Ultralytics YOLO line, YOLOv7, COCO, Pascal VOC, classification and the lane family — each format declaring what it can carry. VisionSet works out exactly what a target would drop before writing anything, and refuses to drop it silently.
Pre-process	a named recipe applied at export: resize every image, and write augmented variants of the training images beside their sources. The release stays untouched, and the export report records the recipe it ran with.
Auto-label	a model you configure and fetch yourself, never one that arrives on its own. Click a point and SAM 2 proposes the shape under it; type words and Grounding DINO finds what they name. Every suggestion is a proposal until you accept it, and an accepted one records which model produced it.
Automate	one SDK under everything, reachable as a Python API, a REST API, a CLI, and an MCP server an agent can drive.

Quickstart

uv tool install "git+https://github.com/Robomous/VisionSet"   # PyPI lands with the beta

visionset init ~/datasets/road-signs     # a workspace, here and nowhere else
cd ~/datasets/road-signs
visionset server                         # API at http://127.0.0.1:8000, app at /app

Then follow the tutorial: a clip of video to a YOLO dataset in about half an hour. Full prerequisites — Python 3.12, and ffmpeg only if you are starting from video — are in docs/content/install.md.

init is the only command that creates a workspace, and it refuses a directory that already holds something. visionset server run outside one refuses with one sentence and exit 1; it never creates one, because a command that silently made a workspace out of whatever directory you were standing in is how data ends up somewhere nobody chose.

Or hand the workspace to an agent — the same cycle, over MCP, with the tools an agent needs to look at what it is labelling:

{ "mcpServers": { "visionset": {
    "command": "visionset", "args": ["mcp"],
    "env": { "VISIONSET_WORKSPACE": "/path/to/workspace" } } } }

The whole cycle as tools, plus the four deletions that are offered only when the server is started with --allow-destructive — because a confirm parameter is documented in the same listing an agent reads before choosing, and four of four measured runs sent it on the first call. See docs/content/mcp.md for how a client is configured and why each tool exists, docs/content/mcp-tools.md for the generated reference, or docs/content/mcp-walkthrough.md for a session start to finish — including what twelve real agent runs actually did with it.

Or drive the whole cycle from the terminal, without a server:

visionset project create road-signs
visionset schema apply schema.json --project road-signs
BATCH=$(visionset ingest ./incoming --project road-signs)
visionset batch approve "$BATCH" --jobs-of 100 --start
# …annotate, then…
visionset batch complete "$BATCH" --promote
visionset release publish --tag v1.0 --project road-signs --split 0.7,0.15,0.15
visionset export --project road-signs --release v1.0 --target yolo11 --out ./out --allow-lossy

Every command takes --json for scripting, and the shapes are the REST API's. See docs/content/cli.md, or examples/cli_end_to_end.sh for that walk with its assertions still in it.

Prefer to see the SDK first? examples/sdk_end_to_end.py drives an empty directory to a hash-verified release in one pass, generating its own images — no server, no CLI, nothing to download. Run it with uv run python examples/sdk_end_to_end.py; the walkthrough is in docs/content/examples.md.

For where the assets themselves come from, examples/ingest_end_to_end.py turns a generated ten-second clip into 50 deduplicated assets in an approved batch, then shows a re-run creating nothing. It needs ffmpeg.

The same cycle runs over each of the other two surfaces, and both start the shipped command for real: examples/http_end_to_end.py starts visionset server on a free port and drives the API with urllib and a bearer token — multipart upload, 202-and-poll ingest, hash-checked manifest and a 401 it asserts — while examples/mcp_end_to_end.py spawns visionset mcp and talks JSON-RPC down its pipe, scaling every box out of the preview it saw and into the asset's own pixels.

Monorepo map

src/visionset/          Single Python distribution (one wheel, one import namespace)
  kernel/               Hexagonal core: domain + ports + default adapters (framework-free)
  wire/                 The JSON shapes the CLI and MCP publish (gated against the REST models)
  server/               FastAPI — exposes the SDK via REST; openapi.json is a committed contract
  cli/                  Typer CLI (`visionset` console script)
  mcp/                  MCP server (stdio) — 56 agent tools over the same SDK, four more on request
  formats/              Exporter plugins: ultralytics, yolov5-yaml, coco, voc, classification and
                        the five lane formats (entry-point group `visionset.formats`)
  preprocessing/        Pre-processing drivers: Pillow resize and augmentation behind the
                        `PreprocessingDriver` port (entry-point group `visionset.preprocessing`)
  jobs/                 Handlers for work that outlives a request: ingest, export, weights
  inference/            Where a model connection becomes a running model (optional runtime)
  _static/              Compiled UI bundle lands here at build time (ships in the wheel)
frontend/
  annotator/            @visionset/annotator — headless annotation engine (no React in core/)
  ui-core/              @visionset/ui-core — domain components, tokens, generated API client
  app/                  @visionset/app — OSS product shell (Vite + React, never published)
tests/                  Python tests, incl. machine-enforced architecture contracts
examples/               Six runnable end-to-end scripts, all exercised in CI
docs/                   Documentation: content/ is the Markdown (user and contributor docs, one page per
                        subsystem); the rest is the Astro + Starlight site that renders it — a view, never a second copy
docker/                 Dev-only compose environment (never the release artifact)
scripts/                Repo automation (OpenAPI export, version sync, bundling, dist build)
.agents/skills/         Coding-agent skills, tool-agnostic (see AGENTS.md)

Documentation

Start with docs/content/install.md and docs/content/tutorial.md. docs/content/README.md indexes the rest — one page per subsystem, each written to explain the decisions rather than restate the code.

docs/content/ is the source of truth, and it is plain Markdown so that it reads on GitHub with nothing installed. docs/ renders the same files as a searchable website; it adds no content of its own. To read it locally:

docker compose -f docker/compose.yaml up docs    # http://localhost:4321

or, without Docker, pnpm --dir docs install && pnpm --dir docs dev. Editing anything under docs/content/ reloads the page.

Development setup

uv sync         # Python env + dev tools
pnpm install    # frontend workspace

Then uv run visionset server and pnpm --filter @visionset/app dev. Or run the whole thing in containers instead, with nothing installed on the host and nothing built.

Run it with Docker, and sign in with nothing

docker compose -f docker/compose.yaml up

Open http://localhost:8080. There is no token to find and nothing to paste — the app opens on the project list. The server signs in the browser it served itself, over an HttpOnly cookie it sets on the first request the page makes; docs/content/auth.md has the mechanism and the reasoning.

One port, nginx in front of both services. The first run builds two images, every later one just starts them; dependencies are installed at build time, so starting the stack downloads nothing.

A token is still minted on first boot and printed in the api logs, because curl, the SDK and MCP clients have no session and never will:

docker compose -f docker/compose.yaml logs api | grep vst_    # if you scrolled past it
docker compose -f docker/compose.yaml exec api \
  visionset token create --name <name>                        # or mint another

The browser never needs either. If the page does ask for a token, the stack is not the one this README describes — check that VISIONSET_UI_SESSION: always is set on the api service and that you are reaching it through port 8080.

Why always here, and what it costs. The default, VISIONSET_UI_SESSION=auto, issues a session only to a client on this machine — and behind a proxy no request ever looks like one, because the peer is nginx. So the compose stack says always. The front door on 8080 is published on every interface, so another device on your network — a phone, a tablet — can open the dev stack at http://<your address>:8080; the api and vite ports stay on loopback. The consequence is that whoever reaches 8080 is signed in, so the stack trusts the network it runs on. On one that is not yours, set VISIONSET_UI_SESSION: never to go back to typing a token, or bind the nginx port to 127.0.0.1 in docker/compose.yaml.

Everything it stores lands in workspace-data/ (git-ignored): SQLite for metadata, a local directory for the files, one workspace holding both — the shape MLflow's default mode has, and the only shape VisionSet has. Put it elsewhere with VISIONSET_DATA=/path docker compose …; it is a bind mount, so down -v does not take your data with it.

The containers run as you rather than as root, so everything they write there — and every __pycache__ and generated page elsewhere in the checkout — belongs to the account that started them, and a later pnpm -r build or git worktree remove is not blocked by a file it cannot touch. That is the uid the stack finds at 1000, which is the first account on a Linux workstation. If yours is another number, tell it once and rebuild:

printf 'VISIONSET_UID=%s\nVISIONSET_GID=%s\n' "$(id -u)" "$(id -g)" > docker/.env
docker compose -f docker/compose.yaml up --build

On macOS and Windows the defaults are already right — Docker Desktop translates ownership itself. Under rootless Docker, set both to 0: there the daemon maps the container's root onto you already.

Dev only — the release artifact is always the pip package.

The inner loop while iterating: uv run pytest <path>, uv run lint-imports, uv run mypy src/visionset/kernel (the kernel only — CI checks all of src/visionset), pnpm -r build, pnpm test (no browser — the two Playwright suites sit outside it). Each is a subset of what CI runs on every pull request; the full list, with what each command does not cover, is in CONTRIBUTING.md.

Releases

CHANGELOG.md — what each version added, and the six milestones that got here. docs/content/releasing.md is the runbook for cutting one.

License

Apache-2.0 — copyright Robomous Inc. See LICENSE.