From 5a14096230e7d3af374cd1dc044aa6c1ab866df9 Mon Sep 17 00:00:00 2001 From: M2Night Date: Wed, 12 Aug 2026 10:49:12 +0800 Subject: [PATCH] Self-hosting guides for the enterprise delivery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the open-source self-hosting pages with guides for the enterprise product: what the three delivery forms are and how to choose between them, requirements, registry access, the Kubernetes and All-in-One installs, air-gapped operation, day-two operations, and how versions are identified. Corrections from three rounds of customer-view testing, all reproduced: - **Air-gapped mirroring.** The chart now takes `global.imageRegistry`, so pointing it at your own registry is one value plus the two Redis digests it cannot unset itself, rather than editing every component. The image list and the push list come from a single render through that same flag, which is what keeps them aligned — a default render resolves Redis to `repository@digest` with no tag, so a hand-written retag invents one the chart never asks for. The pull secret has to move too: it holds a login for one host and a kubelet matches by host, so a private mirror answers every pull with a 401. - **The values snippet was a paste-able top-level `global:` block.** Every shipped profile already has one, and a YAML document with two is not an error — the last wins silently, taking `offlineMode` with it and deploying the online-billing build. Shown in place now, and the verification compares the whole image set against what was pushed rather than looking for an external host, which cannot see that substitution. - **Zero egress.** Every check cut the network and looked for a failure, which proves the deployment does not need the internet, not that it makes no calls when there is some. A packet capture answers that, filtered on the workload's own address rather than by excluding private ranges — a cloud host's own address usually sits in one, so the exclusions emptied the file whatever the software did. - **The ledger.** These pages told customers to verify signatures with a public key Fish Audio supplies, and gave a runnable command whose first line names a file that does not exist. There is no key to ship. What a customer can do — prove the file has not been edited — is now separated from what only we can do. - **Housekeeping.** The pages printed the registry host they promise to write as a placeholder, once inside a `sed` that would have sent a customer to a registry they were not granted. The air-gap checks started two containers that each hold both GPUs and never removed either. Appliance figures are ranges by card, and the disk figure distinguishes pulling the image from carrying it as a file. --- developer-guide/self-hosting/air-gapped.mdx | 63 +++ developer-guide/self-hosting/all-in-one.mdx | 77 ++++ .../self-hosting/docker-deployment.mdx | 343 -------------- .../self-hosting/enterprise-releases.mdx | 41 ++ developer-guide/self-hosting/introduction.mdx | 158 +++++++ developer-guide/self-hosting/kubernetes.mdx | 78 ++++ developer-guide/self-hosting/local-setup.mdx | 155 ------- developer-guide/self-hosting/operations.mdx | 101 +++++ .../self-hosting/registry-access.mdx | 79 ++++ developer-guide/self-hosting/requirements.mdx | 109 +++++ .../self-hosting/running-inference.mdx | 419 ------------------ docs.json | 27 +- 12 files changed, 730 insertions(+), 920 deletions(-) create mode 100644 developer-guide/self-hosting/air-gapped.mdx create mode 100644 developer-guide/self-hosting/all-in-one.mdx delete mode 100644 developer-guide/self-hosting/docker-deployment.mdx create mode 100644 developer-guide/self-hosting/enterprise-releases.mdx create mode 100644 developer-guide/self-hosting/introduction.mdx create mode 100644 developer-guide/self-hosting/kubernetes.mdx delete mode 100644 developer-guide/self-hosting/local-setup.mdx create mode 100644 developer-guide/self-hosting/operations.mdx create mode 100644 developer-guide/self-hosting/registry-access.mdx create mode 100644 developer-guide/self-hosting/requirements.mdx delete mode 100644 developer-guide/self-hosting/running-inference.mdx diff --git a/developer-guide/self-hosting/air-gapped.mdx b/developer-guide/self-hosting/air-gapped.mdx new file mode 100644 index 0000000..136cb06 --- /dev/null +++ b/developer-guide/self-hosting/air-gapped.mdx @@ -0,0 +1,63 @@ +--- +title: "Air-gapped deployments" +description: "Running Fish Audio Enterprise on a network that cannot reach Fish Audio at all" +icon: "shield-halved" +--- + +Both offline delivery forms make no outbound calls at runtime. Neither one, however, +installs itself out of thin air: a Helm install pulls container images and the chart +from a registry, and the appliance needs its image on the host. An air-gapped +deployment is about getting those artifacts across the boundary. + + + Fish Audio does this with you as part of an air-gapped delivery, and the step-by-step + procedure is in the deployment runbook for the version you install — see + [Releases](/developer-guide/self-hosting/enterprise-releases). This page is what to + expect and what to plan for. + + +## Which form to choose + +**The All-in-One appliance** is the straightforward answer to a strict air gap: one +image with every weight baked in, moved to the disconnected host as a file and loaded +there. There is nothing else to mirror. + +**The Helm chart** is more work, because the release is many images rather than one. +Every image has to be mirrored into a registry the cluster can reach and the release +pointed at it, component by component — there is no single switch. Plan for the whole +set, and expect to do it with your account team rather than alone. + +**What you give up either way:** images are pinned to the exact content Fish Audio +published, so a deployment cannot quietly run anything else. Copying an image into +another registry drops that pin, so check what you mirrored while both sides are still +there to compare. + +## Offline usage accounting + +An offline deployment cannot call a billing service, so it records usage locally +instead, in a signed ledger on persistent storage. It is durable data rather than +cache: include it in your backups, and agree a reconciliation cadence with your account +team. Usage is settled from the ledger files themselves, not from totals compiled off +them. + +Because there is no service to validate a token against, any non-empty bearer token is +accepted and recorded as the billing identity. Use a stable, distinct token per tenant. + +## Proving there is no egress + +Regulated deployments usually need evidence rather than a configuration review, and the +runbook carries the exercise. Two things are worth knowing before you plan it. + +It answers two questions, and they need different methods: whether the deployment +*needs* the internet, and whether it *calls out* when allowed to — the second is what a +telemetry or data-residency review actually asks. + +It is also a cluster-level exercise rather than a namespace one. Confirm early that +whoever runs your cluster can take part, because a namespace-scoped account cannot +complete it. + +## Next steps + +- [Requirements](/developer-guide/self-hosting/requirements) — hardware, platform, and network baselines +- [All-in-One container](/developer-guide/self-hosting/all-in-one) — the single-container form +- [Kubernetes deployment](/developer-guide/self-hosting/kubernetes) — the Helm forms diff --git a/developer-guide/self-hosting/all-in-one.mdx b/developer-guide/self-hosting/all-in-one.mdx new file mode 100644 index 0000000..b3520dd --- /dev/null +++ b/developer-guide/self-hosting/all-in-one.mdx @@ -0,0 +1,77 @@ +--- +title: "All-in-One container" +description: "The single-container appliance: what it is, what it cannot do, and what running it involves" +icon: "box" +--- + +The All-in-One image packages the whole speech stack — edge API, model API layer, +inference router and worker, vocoder, text normalizer, and Redis — into one container, +with every model weight baked in. Once the image is on the host it runs with no +Kubernetes and no internet access, which makes it the turnkey option for single-node +appliances and strict air gaps. + + + This page covers what the appliance is and what to plan for. The commands, tuning + options, and troubleshooting are in the **All-in-One guide**, which ships in the + documentation bundle for the image version you run — see + [Releases](/developer-guide/self-hosting/enterprise-releases). + + +## What it cannot do + +The appliance runs one inference worker and one vocoder, a GPU each. It does not +autoscale, does not shard across more GPUs or nodes, and does not ship the forced +aligner, so it returns no word or segment timings. It is offline-only — there is no +hosted-billing variant. For elastic or higher-throughput deployments, use the +[Kubernetes chart](/developer-guide/self-hosting/kubernetes), which scales replicas +across all GPUs and nodes. + +## What running it involves + +One `docker run` on a host that meets the +[All-in-One host requirements](/developer-guide/self-hosting/requirements#all-in-one-container-host). +It needs: + +- **Two GPUs.** The first runs the inference worker, the second the vocoder. +- **One exposed port** for the API. +- **One persistent volume.** Compile caches, the vocoder's built engine, reference + voice archives, and the usage ledger all live there. Model weights are in the image, + not on the volume. + +Everything inside the container runs as a non-root user, so a reused volume or a host +bind mount has to be writable by it. + +**Plan for a slow first start.** The worker compiles its inference graphs and the +vocoder builds its engine before either serves, and the health endpoint verifies the +speech backend end to end rather than reporting immediate liveness. Both artifacts are +cached on the volume, so later starts take minutes. The vocoder engine is specific to +the GPU model, so moving to different cards rebuilds it once. + +## Usage accounting and tenancy + +This build records usage to a local, signed, append-only ledger on the volume instead +of calling a billing service. Two consequences worth designing around: + +- **Any non-empty bearer token is accepted**; a missing or empty one is rejected. The + appliance has nothing to validate a token against. +- **The token is recorded verbatim as the billing identity.** Use a stable, distinct + token per tenant — two tenants sharing a token are indistinguishable in the ledger. + +See [Offline usage accounting](/developer-guide/self-hosting/air-gapped#offline-usage-accounting). + +## Capacity + +The single worker admits a bounded number of in-flight requests; beyond that, requests +queue and time-to-first-audio climbs. The ceiling is set by the worker's key-value +cache VRAM, so larger cards support a higher cap. It is adjustable at launch without +rebuilding the image. Validate latency and error rate at any new value before +committing to it. + +Reference-id requests resolve only from local archives placed on the volume, one zip +per voice. + +## Next steps + +- [Requirements](/developer-guide/self-hosting/requirements#all-in-one-container-host) — host baseline +- [Registry access](/developer-guide/self-hosting/registry-access) — how your team gets the image +- [Air-gapped deployments](/developer-guide/self-hosting/air-gapped) — moving the image to a disconnected host diff --git a/developer-guide/self-hosting/docker-deployment.mdx b/developer-guide/self-hosting/docker-deployment.mdx deleted file mode 100644 index ed8646a..0000000 --- a/developer-guide/self-hosting/docker-deployment.mdx +++ /dev/null @@ -1,343 +0,0 @@ ---- -title: "Docker Deployment" -description: "Deploy Fish Audio models using Docker containers" -icon: "docker" ---- -import { AudioTranscript } from '/snippets/audio-transcript.jsx'; - -{/* speak-mintlify-hash: 7519d2e10765e9a8af2318d43268e2c1f0fe2db4478dadeec9afa3068c0dcb98 */} - - - - - -Fish Audio provides Docker images for both WebUI and API server deployments. You can use pre-built images from Docker Hub or build custom images locally. - -## Prerequisites - -Before deploying with Docker, ensure you have: - -- **Docker** and **Docker Compose** installed -- **NVIDIA Docker runtime** (for GPU support) -- At least **12GB GPU memory** for CUDA inference -- Downloaded model weights (see [Running Inference](/developer-guide/self-hosting/running-inference#download-weights)) - -## Pre-built Images - -Fish Audio provides ready-to-use Docker images on Docker Hub: - -| Image | Description | Best For | -|-------|-------------|----------| -| `fishaudio/fish-speech:latest-webui-cuda` | WebUI with CUDA support | Interactive development with GPU | -| `fishaudio/fish-speech:latest-webui-cpu` | WebUI CPU-only | Testing without GPU | -| `fishaudio/fish-speech:latest-server-cuda` | API server with CUDA | Production deployments with GPU | -| `fishaudio/fish-speech:latest-server-cpu` | API server CPU-only | Low-traffic CPU deployments | - - -For production use, we recommend using specific version tags instead of `latest` to ensure consistency across deployments. - - -## Quick Start with Docker Run - -The fastest way to get started is using `docker run`: - -### WebUI Deployment - -```bash -# Create directories for model weights and reference audio -mkdir -p checkpoints references - -# Start WebUI with CUDA support (recommended) -docker run -d \ - --name fish-speech-webui \ - --gpus all \ - -p 7860:7860 \ - -v ./checkpoints:/app/checkpoints \ - -v ./references:/app/references \ - -e COMPILE=1 \ - fishaudio/fish-speech:latest-webui-cuda - -# For CPU-only deployment -docker run -d \ - --name fish-speech-webui-cpu \ - -p 7860:7860 \ - -v ./checkpoints:/app/checkpoints \ - -v ./references:/app/references \ - fishaudio/fish-speech:latest-webui-cpu -``` - -Access the WebUI at `http://localhost:7860` - -### API Server Deployment - -```bash -# Start API server with CUDA support -docker run -d \ - --name fish-speech-server \ - --gpus all \ - -p 8080:8080 \ - -v ./checkpoints:/app/checkpoints \ - -v ./references:/app/references \ - -e COMPILE=1 \ - fishaudio/fish-speech:latest-server-cuda - -# For CPU-only deployment -docker run -d \ - --name fish-speech-server-cpu \ - -p 8080:8080 \ - -v ./checkpoints:/app/checkpoints \ - -v ./references:/app/references \ - fishaudio/fish-speech:latest-server-cpu -``` - -Access the API documentation at `http://localhost:8080` - - -Enable the `COMPILE=1` environment variable for ~10x faster inference on CUDA deployments. This uses `torch.compile` to optimize the model. - - -## Docker Compose Deployment - -For development or customization, Docker Compose provides easier configuration management: - -### Setup - -```bash -# Clone the repository -git clone https://github.com/fishaudio/fish-speech.git -cd fish-speech -``` - -### Start Services - -```bash -# Start WebUI with CUDA -docker compose --profile webui up - -# Start WebUI with compile optimization -COMPILE=1 docker compose --profile webui up - -# Start API server -docker compose --profile server up - -# Start API server with compile optimization -COMPILE=1 docker compose --profile server up - -# For CPU-only deployment -BACKEND=cpu docker compose --profile webui up -``` - - -Run containers in detached mode by adding the `-d` flag: `docker compose --profile webui up -d` - - -### Environment Variables - -Customize deployment using environment variables or a `.env` file: - -```bash -# .env file example -BACKEND=cuda # or cpu -COMPILE=1 # Enable compile optimization -GRADIO_PORT=7860 # WebUI port -API_PORT=8080 # API server port -UV_VERSION=0.8.15 # UV package manager version -``` - -## Manual Docker Build - -For advanced users who need custom configurations: - -### Build WebUI Image - -```bash -# Build with CUDA support -docker build \ - --platform linux/amd64 \ - -f docker/Dockerfile \ - --build-arg BACKEND=cuda \ - --build-arg CUDA_VER=12.6.0 \ - --build-arg UV_EXTRA=cu126 \ - --target webui \ - -t fish-speech-webui:cuda . - -# Build CPU-only (supports multi-platform) -docker build \ - --platform linux/amd64,linux/arm64 \ - -f docker/Dockerfile \ - --build-arg BACKEND=cpu \ - --target webui \ - -t fish-speech-webui:cpu . -``` - -### Build API Server Image - -```bash -# Build with CUDA support -docker build \ - --platform linux/amd64 \ - -f docker/Dockerfile \ - --build-arg BACKEND=cuda \ - --build-arg CUDA_VER=12.6.0 \ - --build-arg UV_EXTRA=cu126 \ - --target server \ - -t fish-speech-server:cuda . -``` - -### Build Development Image - -```bash -# Build development image with all tools -docker build \ - --platform linux/amd64 \ - -f docker/Dockerfile \ - --build-arg BACKEND=cuda \ - --target dev \ - -t fish-speech-dev:cuda . -``` - -### Build Arguments - -| Argument | Options | Default | Description | -|----------|---------|---------|-------------| -| `BACKEND` | `cuda`, `cpu` | `cuda` | Compute backend | -| `CUDA_VER` | `12.6.0`, etc. | `12.6.0` | CUDA version | -| `UV_EXTRA` | `cu126`, `cu128`, `cu129` | `cu126` | UV extra for CUDA | -| `UBUNTU_VER` | `24.04`, etc. | `24.04` | Ubuntu base version | -| `PY_VER` | `3.12`, etc. | `3.12` | Python version | - -## Volume Mounts - -Both Docker run and Compose methods require these volume mounts: - -| Host Path | Container Path | Purpose | -|-----------|----------------|---------| -| `./checkpoints` | `/app/checkpoints` | Model weights directory | -| `./references` | `/app/references` | Reference audio files for voice cloning | - - -Ensure model weights are downloaded and placed in the `./checkpoints` directory before starting containers. See [Running Inference](/developer-guide/self-hosting/running-inference#download-weights) for download instructions. - - -## Environment Variables Reference - -### WebUI Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `GRADIO_SERVER_NAME` | `0.0.0.0` | WebUI server host | -| `GRADIO_SERVER_PORT` | `7860` | WebUI server port | -| `GRADIO_SHARE` | `false` | Enable Gradio public sharing | - -### API Server Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `API_SERVER_NAME` | `0.0.0.0` | API server host | -| `API_SERVER_PORT` | `8080` | API server port | - -### Model Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `LLAMA_CHECKPOINT_PATH` | `checkpoints/openaudio-s1-mini` | Path to model weights | -| `DECODER_CHECKPOINT_PATH` | `checkpoints/openaudio-s1-mini/codec.pth` | Path to decoder weights | -| `DECODER_CONFIG_NAME` | `modded_dac_vq` | Decoder configuration name | - -### Performance Optimization - -| Variable | Default | Description | -|----------|---------|-------------| -| `COMPILE` | `0` | Enable torch.compile for ~10x speedup (CUDA only) | - -## Container Management - -### View Logs - -```bash -# Docker run -docker logs fish-speech-webui - -# Docker Compose -docker compose logs webui -``` - -### Stop Containers - -```bash -# Docker run -docker stop fish-speech-webui - -# Docker Compose -docker compose down -``` - -### Update Images - -```bash -# Pull latest images -docker pull fishaudio/fish-speech:latest-webui-cuda - -# Restart containers with new image -docker compose --profile webui up -d -``` - -## GPU Support - -### Prerequisites - -Install NVIDIA Container Toolkit: - -```bash -# Ubuntu/Debian -distribution=$(. /etc/os-release;echo $ID$VERSION_ID) -curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - -curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \ - sudo tee /etc/apt/sources.list.d/nvidia-docker.list - -sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit -sudo systemctl restart docker -``` - -### Verify GPU Access - -```bash -docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi -``` - - -GPU support requires NVIDIA Docker runtime. For CPU-only deployment, remove the `--gpus all` flag and use CPU images. - - -## Troubleshooting - -### Container Won't Start - -Check logs for errors: -```bash -docker logs fish-speech-webui -``` - -Common issues: -- Missing model weights in `./checkpoints` -- Port already in use (change port mapping) -- Insufficient GPU memory - -### GPU Not Detected - -Verify NVIDIA Docker runtime is installed: -```bash -docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi -``` - -### Performance Issues - -1. Enable compile optimization: `COMPILE=1` -2. Ensure GPU is being used (check with `nvidia-smi`) -3. Verify sufficient GPU memory is available - -## Next Steps - -- **[Run inference](/developer-guide/self-hosting/running-inference)** - Learn how to generate speech -- **[Download models](https://huggingface.co/fishaudio)** - Get pre-trained weights -- **[API documentation](/api-reference/introduction)** - Integrate with your applications diff --git a/developer-guide/self-hosting/enterprise-releases.mdx b/developer-guide/self-hosting/enterprise-releases.mdx new file mode 100644 index 0000000..29c9583 --- /dev/null +++ b/developer-guide/self-hosting/enterprise-releases.mdx @@ -0,0 +1,41 @@ +--- +title: "Releases" +description: "How Fish Audio Enterprise versions are identified and how to move between them" +icon: "code-branch" +--- + +Fish Audio publishes a version of each delivery form once it has been tested. +**Developer → Self Host** lists the versions available to your team and fills the +one you pick into the install commands on that page. + +## Version identifiers + +| Delivery form | Identified by | Applied with | +| ---------------------------- | --------------- | ------------------------------------ | +| Online Helm and Offline Helm | A chart version | `--version` on `helm pull`/`install` | +| All-in-One | An image tag | The tag in the image reference | + +Both Helm forms are one chart, so they share a version. Which form you deploy is +decided by the values file, not by the version. + +## Upgrading + +A deployment stays on its version until you change it. Take the new version from +**Developer → Self Host**, then follow the upgrade +procedure in the deployment runbook for the Helm forms, or pull the new tag and +recreate the container against the same volume for +[All-in-One](/developer-guide/self-hosting/all-in-one). Air-gapped deployments +mirror the new version first — see +[Air-gapped deployments](/developer-guide/self-hosting/air-gapped). + + + Talk to your account team before skipping versions, and before upgrading a + production deployment, so the change can be checked against how yours is + configured. + + +## Next step + +Return to [Kubernetes deployment](/developer-guide/self-hosting/kubernetes) or +[All-in-One container](/developer-guide/self-hosting/all-in-one) to apply the +version you picked. diff --git a/developer-guide/self-hosting/introduction.mdx b/developer-guide/self-hosting/introduction.mdx new file mode 100644 index 0000000..7e3df1c --- /dev/null +++ b/developer-guide/self-hosting/introduction.mdx @@ -0,0 +1,158 @@ +--- +title: "Introduction" +description: "Run the Fish Audio Enterprise speech stack inside your own infrastructure" +icon: "server" +--- + +Fish Audio Enterprise can be deployed into infrastructure you control: your own +cloud account, an on-premise data center, or a network with no internet access at +all. Fish Audio delivers container images and a Helm chart through a private +registry; you own the cluster, the network boundary, and the data. + +The self-hosted stack runs the same speech engine as the hosted API, so requests, +audio formats, and voice behavior match what you already build against. + +## Why self-host + +| Reason | What it gives you | +| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| Colocation | Run inference in the same region, VPC, or rack as your application and remove public-internet round trips from time-to-first-audio. | +| Single-tenant isolation | Dedicated GPUs and queues. Capacity is not shared with other tenants, and you decide when the deployment is upgraded. | +| Security posture | Voice traffic never leaves your network. The offline delivery forms make no outbound calls at runtime and run on disconnected networks. | +| Data sovereignty | Input text, generated audio, and reference voices stay inside your boundary and under your own retention policy. | + +## Delivery forms + +The same engine ships in three forms. Your enterprise agreement determines which +ones your team is granted. + +| | Online Helm | Offline Helm | Offline All-in-One | +| ---------------- | --------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------- | +| Platform | Kubernetes | Kubernetes | A single container, `docker run` | +| Model assets | Served in-cluster from a bundled model warehouse | Same | Baked into the image | +| Usage accounting | Validated and billed against the Fish Audio service | Local signed usage ledger | Local signed usage ledger | +| Runtime egress | The billing endpoint, plus one more host if you enable timestamps | None | None | +| Air-gap capable | No | Yes, after mirroring images and charts | Yes | +| Scaling | Scales replicas across all GPUs and nodes | Same | Single node, single inference worker | +| Best for | Managed clusters with outbound access | Isolated or regulated production clusters | Evaluation, single-node appliances, strict air gaps | + +Both Helm forms share one deployment procedure and differ only in a few values. +See [Kubernetes deployment](/developer-guide/self-hosting/kubernetes) for the +Helm forms and [All-in-One container](/developer-guide/self-hosting/all-in-one) +for the single-container form. + +## What is included + +| Capability | Online Helm | Offline Helm | All-in-One | +| ----------------------------------------- | ---------------------------- | ------------------------ | ------------------------ | +| Text to speech over `POST /v1/tts` | Included | Included | Included | +| WebSocket streaming | Included | Included | Included | +| Reference-voice requests (`reference_id`) | From pre-staged archives | From pre-staged archives | From pre-staged archives | +| Word and segment timestamps | Optional, needs an extra GPU | Not included | Not included | +| Horizontal scaling and autoscaling | Included | Included | Not included | +| Prometheus metrics | Included | Included | Container logs only | + + + Timestamp alignment is served by a separate forced-aligner component. It is + disabled by default, is not part of the offline model bundle, and is not built + into the All-in-One image, so `/v1/tts/stream/with-timestamp` returns audio + without alignment data on those forms. Contact Fish Audio if your deployment + needs timestamps. + + +Reference voices are resolved from archives you stage yourself. A self-hosted +deployment does not read from the hosted voice library, and voice models created +on fish.audio are not automatically available to it. Products other than text to +speech are hosted-only unless your agreement says otherwise. + +## Architecture at a glance + +| Component | Role | +| ------------------ | ---------------------------------------------------------------------------------------------- | +| Edge API | Entry point. Validates requests, applies product logic, and routes work to the model services. | +| Model API layer | Coordinates normalization, inference, and audio decoding. | +| Inference router | Distributes work across the GPU workers. | +| Inference workers | GPU-backed generation. | +| Vocoder | GPU-backed audio decoding. | +| Text normalizer | Text normalization ahead of inference. | +| Redis | Runtime state and cache. Installed by the chart into the release namespace. | +| Shared storage | Model cache, reference archives, and the usage ledger. Mounted on every node. | + +Fish Audio delivers the deployment at the Kubernetes service level. Ingress, DNS, +TLS, external load balancing, and network allowlists are yours to choose and +configure. + +## Getting access + + + + Self-hosting is enabled per team under an enterprise agreement. Reach out + through [fish.audio/enterprise](https://fish.audio/enterprise) with your GPU + target, expected concurrency, and whether you need air-gapped operation. + + + Once your agreement is in place, Fish Audio grants your team the artifacts + it is entitled to: the Helm chart, the component images, and the All-in-One + image where applicable. + + + Sign in to fish.audio and open **Developer → Self Host** to see the delivery + forms your team is granted and to create the token that authenticates + against the registry. See [Registry + access](/developer-guide/self-hosting/registry-access). + + + +## Next steps + + + + GPU, CPU, memory, storage, and platform baselines. + + + Create a deploy token and authenticate Docker and Helm. + + + Install, upgrade, roll back, and validate the Helm release. + + + Run the whole stack from a single `docker run`. + + + Mirror artifacts, account for usage offline, and prove zero egress. + + + Monitoring signals, scaling, backups, and troubleshooting. + + + How versions are published, and which one to deploy. + + diff --git a/developer-guide/self-hosting/kubernetes.mdx b/developer-guide/self-hosting/kubernetes.mdx new file mode 100644 index 0000000..a2efea8 --- /dev/null +++ b/developer-guide/self-hosting/kubernetes.mdx @@ -0,0 +1,78 @@ +--- +title: "Kubernetes deployment" +description: "What the Helm delivery installs, what you decide before installing, and what to expect" +icon: "dharmachakra" +--- + +Fish Audio Enterprise ships as a Helm chart that installs the whole speech stack into +a cluster you run. Both Helm delivery forms use the same chart and the same procedure; +they differ only in how usage is accounted. + + + This page covers what the deployment involves and what you need to decide. The + commands, values, and troubleshooting are in the **deployment runbook**, which ships + in the documentation bundle alongside the chart version you install — see + [Releases](/developer-guide/self-hosting/enterprise-releases). The runbook is + versioned with the chart; this page is not, so follow the runbook when they differ. + + +## Choose a delivery form + +| | Offline | Online | +| ---------------- | ------------------------------------------------ | --------------------------------------------------- | +| Usage accounting | Local signed ledger on shared storage | Validated and billed against the Fish Audio service | +| Runtime egress | None | The billing endpoint on 443 | +| Model assets | Served in-cluster by the bundled model warehouse | The same | + +Both forms serve model weights from inside the cluster, so neither reaches an external +object store at runtime. "Offline" means runtime-offline: installation still pulls +images and the chart from a registry. For a cluster with no network access at all, see +[Air-gapped deployments](/developer-guide/self-hosting/air-gapped). + +## What gets installed + +The chart deploys the edge API, the model API layer, an inference router and worker, +a vocoder, a text normalizer, the model warehouse that serves the weights, and Redis. +Everything lands in one namespace, in the single-worker shape the delivery was sized +against. + +Ingress and TLS are off by default. Put your own ingress controller or load balancer +in front of the edge API service, or enable the chart's if you want Kubernetes to +manage that layer. + +## What you decide before installing + +| Decision | Notes | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| Namespace | `fish-audio` is the supported default. A different one has to be set in two places; the runbook says where. | +| Shared storage path | Mounted at the same path on **every** node. See [Requirements](/developer-guide/self-hosting/requirements#shared-storage). | +| GPU scheduling | Tolerations and node selectors, if your GPU nodes are tainted or you run more than one GPU model. | +| Metrics | The chart exposes Prometheus annotations and creates no ServiceMonitors, so an annotation-scraping Prometheus works as-is and kube-prometheus-stack needs a scrape config. | +| Reference voices | Requests carrying a `reference_id` resolve from a local archive, never from the network. One zip per voice, staged on shared storage before that traffic starts. | +| Replica counts | Talk to your account team first. The shipped shape is what the delivery was sized and validated against. | + +Three Kubernetes Secrets have to exist before the install: registry credentials, a JWT +secret for the edge API, and one shared between the in-cluster model store and the +workers that read from it — that one authenticates nothing outside the cluster, and the +value is yours to choose. Generate the JWT secret once +and keep it stable — changing it invalidates issued tokens. For production, prefer +External Secrets, Sealed Secrets, or your cloud secret manager over plain Secrets. + +## What to expect + +Image references are pinned in the chart, so you select a version with a Helm argument +rather than by editing tags. + +The first install is slow: the inference worker compiles its GPU graphs and the +vocoder builds its inference engine before either becomes ready, which takes far longer +than any later rollout. Both artifacts are cached on shared storage, so subsequent +starts are quick — as long as that storage persists. + +Upgrades, rollback, and uninstall are all standard Helm operations. The runbook covers +the order to do them in and what to check at each step. + +## Next steps + +- [Requirements](/developer-guide/self-hosting/requirements) — hardware, platform, and network baselines +- [Registry access](/developer-guide/self-hosting/registry-access) — how your team gets the chart +- [Operations](/developer-guide/self-hosting/operations) — running it once it is live diff --git a/developer-guide/self-hosting/local-setup.mdx b/developer-guide/self-hosting/local-setup.mdx deleted file mode 100644 index c9ffa44..0000000 --- a/developer-guide/self-hosting/local-setup.mdx +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: "Local Model Setup" -description: "Install and configure Fish Audio models for local inference" -icon: "server" ---- -import { AudioTranscript } from '/snippets/audio-transcript.jsx'; - -{/* speak-mintlify-hash: f1038df54a82e77cfe449b4182e5a43b9739fce33ecc1347496c5db4efd98e09 */} - - - - - - -This guide is for advanced users who want to self-host Fish Audio models. For most users, we recommend using the [Fish Audio API](https://fish.audio) for easier integration and automatic updates. - - -## Prerequisites - -Before you begin, ensure you have: - -- **GPU**: 12GB VRAM minimum (for inference) -- **OS**: Linux or WSL (Windows Subsystem for Linux) -- **System dependencies**: Audio processing libraries - -Install required system packages: - -```bash -apt install portaudio19-dev libsox-dev ffmpeg -``` - -## Installation Methods - -Fish Audio supports multiple installation methods. Choose the one that best fits your development environment. - -### Conda Installation - -Conda provides a stable, isolated Python environment: - -```bash -# Create a new environment with Python 3.12 -conda create -n fish-speech python=3.12 -conda activate fish-speech - -# GPU installation (choose your CUDA version: cu126, cu128, cu129) -pip install -e .[cu129] - -# CPU-only installation (slower, not recommended for production) -pip install -e .[cpu] - -# Default installation (uses PyTorch default index) -pip install -e . -``` - - -For best performance, match your CUDA version with your GPU driver. Use `nvidia-smi` to check your CUDA version. - - -### UV Installation - -[UV](https://github.com/astral-sh/uv) provides faster dependency resolution and installation: - -```bash -# GPU installation (choose your CUDA version: cu126, cu128, cu129) -uv sync --python 3.12 --extra cu129 - -# CPU-only installation -uv sync --python 3.12 --extra cpu -``` - - -UV is recommended for faster setup times, especially when working with large dependency trees. - - -### Intel Arc XPU Support - -For Intel Arc GPU users, install with XPU support: - -```bash -# Create environment -conda create -n fish-speech python=3.12 -conda activate fish-speech - -# Install required C++ standard library -conda install libstdcxx -c conda-forge - -# Install PyTorch with Intel XPU support -pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/xpu - -# Install Fish Speech -pip install -e . -``` - - -The `--compile` optimization flag is not supported on Windows and macOS. To use compile acceleration, you need to install Triton manually. - - -## Repository Setup - -Clone the Fish Speech repository to get started: - -```bash -git clone https://github.com/fishaudio/fish-speech.git -cd fish-speech -``` - -Then follow one of the installation methods above. - -## Next Steps - -Once installation is complete, you can: - -- **[Set up Docker deployment](/developer-guide/self-hosting/docker-deployment)** - Use containerized deployment for easier management -- **[Run inference](/developer-guide/self-hosting/running-inference)** - Start generating speech with your local models -- **Download models** - Get pre-trained weights from [Hugging Face](https://huggingface.co/fishaudio) - -## Hardware Recommendations - -For optimal performance: - -| Use Case | Recommended GPU | VRAM | Expected Speed | -|----------|----------------|------|----------------| -| Development | RTX 3060 | 12GB | ~1:15 real-time factor | -| Production | RTX 4090 | 24GB | ~1:7 real-time factor | -| Enterprise | A100 | 40GB+ | ~1:5 real-time factor | - - -Real-time factor indicates how much faster than real-time the model can generate audio. For example, 1:7 means generating 1 minute of audio takes ~8.5 seconds. - - -## Troubleshooting - -### CUDA Out of Memory - -If you encounter CUDA out of memory errors: - -1. Reduce batch size in inference settings -2. Use `--half` flag for FP16 inference -3. Close other GPU-intensive applications - -### Package Installation Errors - -If you encounter dependency conflicts: - -1. Try using UV instead of pip for better dependency resolution -2. Create a fresh conda environment -3. Ensure you're using Python 3.12 (other versions may have compatibility issues) - -## Community Support - -Need help with local setup? - -- Join our [Discord community](https://discord.gg/dF9Db2Tt3Y) for community support -- Check [GitHub Issues](https://github.com/fishaudio/fish-speech/issues) for known problems -- Contact [enterprise support](mailto:support@fish.audio) for commercial deployments \ No newline at end of file diff --git a/developer-guide/self-hosting/operations.mdx b/developer-guide/self-hosting/operations.mdx new file mode 100644 index 0000000..ec7ffc2 --- /dev/null +++ b/developer-guide/self-hosting/operations.mdx @@ -0,0 +1,101 @@ +--- +title: "Operations" +description: "Monitoring, scaling, backups, and troubleshooting for a self-hosted deployment" +icon: "chart-line" +--- + +Day-2 guidance for the Kubernetes delivery forms. For the single-container form, see +[All-in-One container](/developer-guide/self-hosting/all-in-one). + +## Ownership + +| Area | Owner | +| ---------------------------------------------------- | ------------------------------------------------------------- | +| Kubernetes cluster, node lifecycle, platform add-ons | You | +| Fish Audio chart and application configuration | Fish Audio | +| Secrets and credentials | You, with Fish Audio inputs where required | +| Monitoring and alerting | You. Fish Audio can advise on expected signals and thresholds | +| Incident response | Joint during the deployment window, yours after handoff | + +Agree an escalation path before production traffic starts. + +## Monitoring signals + +Prometheus is the metrics baseline. Watch: + +- Pod readiness and restart counts for every service in the release, including Redis. +- Request success rate, error rate, latency, and time-to-first-audio. +- Queue depth, where the application exposes it. +- GPU utilization and GPU memory. +- CPU and memory usage against the configured requests and limits. +- Shared storage usage and latency. +- Redis availability and latency. +- Image pull failures and Kubernetes events in the release namespace. + +Centralize application logs and retain Kubernetes events long enough to +investigate a bad rollout. Redact secrets before sharing any logs outside your +environment. + +## Scaling + +Capacity is changed through the release configuration: API replicas, GPU worker +replicas, CPU and memory requests, GPU resource requests, and per-worker concurrency. +All of it takes effect on an upgrade. Scaling on load instead of by hand is included +and off by default; while it is on, the replica count is the autoscaler's rather than +the file's. Your documentation bundle covers turning it on. + +Add a GPU replica only when a GPU is actually free: a pod that requests one on a full +cluster stays `Pending` indefinitely. + +Validate every scale change with the smoke test and a benchmark run at your +expected concurrency, and remember that the first start of a new GPU worker pays +the compile cost before it becomes ready. + +## Backup and retention + +Back up: + +- Reference voice archives, if your traffic uses reference ids. In a self-hosted + deployment these are durable data, not cache — nothing can re-download them. +- The offline usage ledger directories, before they are pruned by your own + archival process. +- The values file used for the production release. +- Secret manager entries. +- Dashboards and alert rules. + +Compile and model caches on shared storage do not need backing up. Losing them +costs a slow first start, nothing more. + +## Upgrades + +Commands are in the deployment runbook. Before running them, capture the release +history so you know the revision to roll back to, and keep that revision available +until the new one is accepted. + +## Troubleshooting + +The failures a Kubernetes deployment actually produces, and where each one usually +comes from. The commands to diagnose each, and the fixes, are in the troubleshooting +guide that ships in your documentation bundle. + +| Symptom | Likely causes | +| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ImagePullBackOff` or `ErrImagePull` | Missing pull secret, a deploy token that was rotated or deleted, a tag that is not in your mirror, or blocked egress to the registry. | +| Pods stay `Pending` | Not enough CPU, memory, or GPU capacity; GPU node taints without matching tolerations; a node selector that matches nothing. | +| GPU not available, or `nvidia-smi` fails in a pod | Driver missing or unhealthy, device plugin or GPU Operator not ready, container runtime not configured for NVIDIA, or an unsupported GPU for the driver stack. | +| hostPath volume errors, or missing model and cache paths | Shared storage is not mounted on that node, a newly added node never got the mount, the mount target is unreachable, or permissions prevent writes. | +| Requests fail after reaching the edge API, with Redis errors in its logs | The Redis master or replica is not ready, the service is missing, or a network policy blocks it. | +| `Reference not found` | The archive was never staged, the filename does not exactly match the reference id, the zip is malformed, or the reference root was changed without mounting the new path into every edge API pod. | +| Metrics missing from Prometheus | Scrape annotations disabled in values, Prometheus not configured for annotation-based discovery, a port mismatch, or a network policy blocking scrapes. | +| `CrashLoopBackOff` | A missing secret or environment variable, model assets unreachable, or a path the container cannot write. | +| Time-to-first-audio high, GPU utilization low | Concurrency above what the current replicas can serve, cold caches after a restart, storage latency, or CPU saturation on the API nodes. | + +## Escalating to Fish Audio + +Include: + +- Kubernetes version, cloud provider, region, node types, and GPU type. +- Chart version and the values file with secrets removed. +- Pod status, relevant events, and logs with secrets redacted. +- The exact command that failed, with its output. +- Timestamp and time zone. diff --git a/developer-guide/self-hosting/registry-access.mdx b/developer-guide/self-hosting/registry-access.mdx new file mode 100644 index 0000000..ba47398 --- /dev/null +++ b/developer-guide/self-hosting/registry-access.mdx @@ -0,0 +1,79 @@ +--- +title: "Registry access" +description: "The Self Host dashboard, deploy tokens, and how your team reaches the Fish Audio registry" +icon: "key" +--- + +Self-hosted images, charts, and documentation are distributed from a private Fish Audio +registry. Your team authenticates to it with a **deploy token** created in the fish.audio +dashboard. + +## The Self Host page + +Sign in to fish.audio and open **Developer → Self Host**. Everything your team needs to +start is there: + +- **Your deployment** — the delivery forms your team is granted. If it is empty, or the + page reports that self-host deployment is not enabled, contact your account manager. +- **The versions available to you**, and the documentation bundle for each — see + [Releases](/developer-guide/self-hosting/enterprise-releases). +- **Install commands built for your team**, with the registry host, artifact references, + and the version you pick already filled in. + + + The registry host, the artifact references, and the versions available to you are + specific to your team and are shown only in the dashboard — copy them from the Self + Host page. + + +## Prerequisites + +- Self-hosting enabled for your team under an enterprise agreement. +- A fish.audio account that is a member of that team. +- Docker, and Helm 3.8 or newer for the OCI chart commands. + +## Create a deploy token + + + + On **Developer → Self Host**, select **Create Deploy Token**. + + + Use a name that identifies the consumer, such as `prod-cluster` or `ci-mirror`. The + name appears in the token list alongside the creation date and last-used time. + + + The token value is shown once, at creation. Store it in your secret manager before + closing the dialog. If you lose it, rotate the token to issue a new one. + + + +A team can hold up to five deploy tokens at a time. Tokens carry the grants of the team +that owns them, not of the person who created them, and follow those grants as they +change — there is no token to recreate when your entitlement is updated. + +Authenticate with your account email as the username and the deploy token as the +password. Docker, Helm, and the cluster's image pull secret all use the same pair; the +exact commands are on the Self Host page and in the deployment runbook. + +## Managing tokens + +| Action | Effect | +| --------- | ---------------------------------------------------------------------------------------------------------------------------------------- | +| Rotate | Issues a new token value and invalidates the old one immediately. Update every consumer before rotating, or new pods will fail to pull. | +| Delete | Revokes the token immediately. Any deployment still using it stops pulling images. Running pods keep running until they are rescheduled. | +| Last used | Shows when the token last authenticated, which identifies tokens that are safe to retire. | + +Recommended practice: + +- Issue one token per consumer — production cluster, staging cluster, CI mirror — so a + single revocation never takes down more than one of them. +- Store tokens in your secret manager, not in values files or version control. +- Rotate on your normal credential schedule and whenever someone with access to a token + leaves the team. + +## Next step + +With access in place, continue to +[Kubernetes deployment](/developer-guide/self-hosting/kubernetes) or the +[All-in-One container](/developer-guide/self-hosting/all-in-one). diff --git a/developer-guide/self-hosting/requirements.mdx b/developer-guide/self-hosting/requirements.mdx new file mode 100644 index 0000000..74c583f --- /dev/null +++ b/developer-guide/self-hosting/requirements.mdx @@ -0,0 +1,109 @@ +--- +title: "Requirements" +description: "Hardware, platform, and network baselines for a self-hosted deployment" +icon: "microchip" +--- + +These are the baselines Fish Audio deploys against. Confirm final sizing with your +account team once your traffic profile and target GPU are known: time-to-first-audio +and throughput depend on the model, GPU, text length, and concurrency, and should be +measured on your own hardware before you commit to a capacity plan. + +## Kubernetes deployments + +### Cluster topology + +| Node type | Minimum | Purpose | +| -------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| CPU-only control-plane or system nodes | 3 nodes, roughly 4 vCPU and 16 GB RAM each | Control-plane high availability on self-managed Kubernetes, or a system node group for platform add-ons on managed Kubernetes. | +| CPU workload capacity | Roughly 28 vCPU and 125 GiB of memory requests | Non-GPU workloads: Redis, the edge API, the text normalizer, the model API layer, and the inference router. Provide a CPU node group, or leave GPU nodes schedulable so these can land there. | +| GPU capacity | 2 GPUs | One for the inference worker, one for the vocoder. Spread them across two nodes if the deployment also has to survive a node drain or failure. | +| Shared storage | 1 TB usable, mounted on every node | Model cache, reference archives, compile caches, and the offline usage ledger. | +| Network | Low-latency east-west networking | GPU workload stability, service-to-service calls, and shared storage access. | + +Verify these against your own chart version by rendering the release with a deployment +profile applied — the chart defaults alone stop on a required value — and summing the +requests. + +### GPUs + +| Requirement | Detail | +| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Count | 2 GPUs minimum: one inference worker, one vocoder. Enabling timestamp alignment adds a third. | +| Type | NVIDIA H100 or H200. H200 is the preferred target for its larger memory and bandwidth. | +| Memory | The worker pre-allocates most of its card for the KV cache, so a near-full card is expected, not a fault. A larger card raises the concurrency ceiling rather than leaving headroom. | +| Newer architectures | Contact Fish Audio before standardizing on a GPU generation that is not H100 or H200, so the image, CUDA stack, and driver combination can be confirmed. | +| MIG | Disable MIG unless the configuration has been validated with Fish Audio. | + +On AWS, `p5.48xlarge` (H100) is an acceptable baseline and `p5en.48xlarge` (H200) +is the preferred target. Validate GPU instance quota in the target region before +scheduling a deployment window. + +### Shared storage + +Provide at least 1 TB of usable shared storage backed by EFS, NFS, or an +equivalent service, mounted at the same path on every node that runs Fish Audio +workloads (`/mnt/share` by default, and configurable). The chart mounts it into containers with +hostPath volumes and creates no PersistentVolumeClaims. + + + The hostPath mount is created with `DirectoryOrCreate`. A node that is missing + the shared mount silently gets a local directory instead, and the deployment + looks healthy while data splits across nodes. Verify the mount, and that it is + writable, on **every** node before installing. The deployment runbook gives a check + that does this without node access. + + +Confirm the following with your storage team: how the mount is applied to newly +added nodes, the throughput mode, the backup policy, mount target reachability, +and expected growth of the model and cache data. + +Redis is installed by the chart and does not use shared storage. Its append-only +file is backed by node-local ephemeral storage, so provision ephemeral capacity +on the nodes that host it, plus headroom for rewrites. + +### Platform baseline + +| Layer | Recommendation | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Kubernetes | A currently supported minor version. Stay one minor behind the newest release if your GPU add-ons have not been validated on it yet. | +| Helm | A current maintained release supported by your platform. | +| Container runtime | The provider-managed containerd runtime where available. | +| GPU runtime | NVIDIA GPU Operator, or the provider-managed driver and device plugin stack. GPU nodes must expose `nvidia.com/gpu`. | +| GPU telemetry | DCGM exporter or the provider equivalent. | +| Storage | EFS, NFS, or an equivalent shared filesystem mounted on every node. | +| Object storage | Not required. Models are served from inside the cluster in every delivery form. Hosting the weights in your own bucket instead is possible; ask your account team. | +| Redis | Installed by the chart into the release namespace. Do not point the release at a shared Redis without discussing it first. | +| Metrics | Prometheus, scraping `prometheus.io/*` pod annotations. The chart creates no ServiceMonitors, so kube-prometheus-stack users must add an annotation-based scrape config. | +| Access layer | Yours to choose. Ingress controller, DNS, TLS, and load balancing are not part of the delivery. | + +The cluster also needs working in-cluster DNS, a dedicated namespace, node labels +and taints for GPU scheduling, and a Pod Security level in that namespace that +permits `hostPID` and `hostIPC`, which the inference workers require. + +### Network + +| Direction | Requirement | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Install-time egress | Access to the Fish Audio registry to pull images and the chart. Mirror both into your own registry for air-gapped installs. | +| Runtime egress, offline form | None. No model download and no billing call. | +| Runtime egress, online form | HTTPS to the Fish Audio authorization and billing endpoint. Models are served from inside the cluster, so no object-store endpoint has to be reachable. Ask your account team for the billing hostname to allowlist. Enabling the timestamp aligner adds one more host. | +| Ingress | Customer-approved ingress or a private endpoint, with DNS and TLS in place before production traffic. | + +## All-in-One container host + +| Requirement | Detail | +| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| OS | Linux x86-64. | +| GPUs | 2 GPUs, 32 GB of memory or larger. The first runs the inference worker, the second runs the vocoder. No NVLink required. | +| NVIDIA driver | Must support CUDA 13.x and your card's compute capability. This applies to the Kubernetes forms too: a driver capped at CUDA 12 reports a healthy GPU and then fails the workloads. | +| Docker | Docker Engine 24 or newer. | +| Container toolkit | NVIDIA Container Toolkit installed and the `nvidia` runtime registered, so `--gpus all` exposes GPUs. | +| RAM | 128 GiB minimum, 192 GiB recommended. The container runs the whole stack in one process tree. | +| CPU | 32 vCPU minimum, 48 to 64 recommended. | +| Disk | 100 GB free if the host pulls the image itself; 120 GB if you move it as a file, since `docker load` needs the ~30 GB archive and the ~80 GB unpacked image at once. Plus the compile and engine caches. | + +## Next step + +Once the platform checks pass, authenticate to the Fish Audio registry in +[Registry access](/developer-guide/self-hosting/registry-access). diff --git a/developer-guide/self-hosting/running-inference.mdx b/developer-guide/self-hosting/running-inference.mdx deleted file mode 100644 index df0a412..0000000 --- a/developer-guide/self-hosting/running-inference.mdx +++ /dev/null @@ -1,419 +0,0 @@ ---- -title: "Running Inference" -description: "Generate speech using self-hosted Fish Audio models" -icon: "play" ---- -import { AudioTranscript } from '/snippets/audio-transcript.jsx'; - -{/* speak-mintlify-hash: 7698d1a07e9798942356f5040025620002f27a00c185f904d68ca5902892dca4 */} - - - - - -Fish Audio supports multiple inference methods: command line, HTTP API, WebUI, and GUI. Choose the method that best fits your workflow. - - -This guide assumes you have already [installed Fish Audio locally](/developer-guide/self-hosting/local-setup) or [set up Docker deployment](/developer-guide/self-hosting/docker-deployment). - - -## Download Weights - -Before running inference, download the required model weights from Hugging Face: - -```bash -# Install Hugging Face CLI (if not already installed) -pip install huggingface_hub[cli] -# or -uv tool install huggingface_hub[cli] - -# Download Fish Audio S1-mini weights -hf download fishaudio/openaudio-s1-mini --local-dir checkpoints/openaudio-s1-mini -``` - - -**Fish Audio S1-mini** is the open-source distilled version (0.5B parameters) optimized for local deployment. The full **S1** model (4B parameters) is available exclusively on [Fish Audio cloud](https://fish.audio). - - -## Command Line Inference - -Command line inference provides maximum control and is ideal for scripting and batch processing. - -### Step 1: Extract VQ Tokens from Reference Audio - -First, encode your reference audio to get voice characteristics: - -```bash -python fish_speech/models/dac/inference.py \ - -i "reference_audio.wav" \ - --checkpoint-path "checkpoints/openaudio-s1-mini/codec.pth" -``` - -This generates two files: -- `fake.npy` - VQ tokens representing voice characteristics -- `fake.wav` - Reconstructed audio for verification - - -**Skip this step if you want random voice generation** - the model can generate speech without reference audio. - - -### Step 2: Generate Semantic Tokens from Text - -Convert your text to semantic tokens using the language model: - -```bash -python fish_speech/models/text2semantic/inference.py \ - --text "The text you want to convert to speech" \ - --prompt-text "Transcription of your reference audio" \ - --prompt-tokens "fake.npy" \ - --compile -``` - -**Parameters:** -- `--text`: The text to synthesize -- `--prompt-text`: Transcription of the reference audio (for voice cloning) -- `--prompt-tokens`: Path to VQ tokens from Step 1 (for voice cloning) -- `--compile`: Enable kernel fusion for faster inference (~10x speedup on RTX 4090) - - -For random voice generation, omit `--prompt-text` and `--prompt-tokens` parameters. - - -This creates a file named `codes_N.npy` (where N starts from 0) containing semantic tokens. - - -For GPUs that don't support bf16 (bfloat16), add the `--half` flag to use fp16 instead. - - -### Step 3: Generate Audio from Semantic Tokens - -Finally, convert semantic tokens to audio: - -```bash -python fish_speech/models/dac/inference.py \ - -i "codes_0.npy" -``` - -This generates the final audio file. - -### Full Example - -Here's a complete workflow for voice cloning: - -```bash -# 1. Encode reference audio -python fish_speech/models/dac/inference.py \ - -i "my_voice.wav" \ - --checkpoint-path "checkpoints/openaudio-s1-mini/codec.pth" - -# 2. Generate semantic tokens -python fish_speech/models/text2semantic/inference.py \ - --text "Hello, this is a test of voice cloning." \ - --prompt-text "This is my reference voice recording." \ - --prompt-tokens "fake.npy" \ - --compile - -# 3. Generate final audio -python fish_speech/models/dac/inference.py \ - -i "codes_0.npy" -``` - -## HTTP API Inference - -The HTTP API provides a programmatic interface for integrations and production deployments. - -### Start API Server - -```bash -# With local installation -python -m tools.api_server \ - --listen 0.0.0.0:8080 \ - --llama-checkpoint-path "checkpoints/openaudio-s1-mini" \ - --decoder-checkpoint-path "checkpoints/openaudio-s1-mini/codec.pth" \ - --decoder-config-name modded_dac_vq - -# With UV -uv run tools/api_server.py \ - --listen 0.0.0.0:8080 \ - --llama-checkpoint-path "checkpoints/openaudio-s1-mini" \ - --decoder-checkpoint-path "checkpoints/openaudio-s1-mini/codec.pth" \ - --decoder-config-name modded_dac_vq -``` - - -Add the `--compile` flag to enable torch.compile optimization for faster inference. - - -### Access API Documentation - -Once the server is running, access the interactive API documentation at: - -``` -http://localhost:8080/docs -``` - -The API provides endpoints for: -- Text-to-speech synthesis -- Voice cloning with reference audio -- Batch processing -- Model information - -### Example API Request - -```bash -curl -X POST "http://localhost:8080/v1/tts" \ - -H "Content-Type: application/json" \ - -d '{ - "text": "Hello, this is a test", - "reference_audio": "base64_encoded_audio", - "reference_text": "Reference transcription" - }' -``` - -## WebUI Inference - -The WebUI provides an intuitive interface for interactive testing and development. - -### Start WebUI - -```bash -# With all parameters -python -m tools.run_webui \ - --llama-checkpoint-path "checkpoints/openaudio-s1-mini" \ - --decoder-checkpoint-path "checkpoints/openaudio-s1-mini/codec.pth" \ - --decoder-config-name modded_dac_vq - -# Or use defaults (auto-detects models in checkpoints/) -python -m tools.run_webui -``` - - -Add the `--compile` flag for faster inference during interactive sessions. - - -### Access WebUI - -The WebUI starts on port 7860 by default. Access it at: - -``` -http://localhost:7860 -``` - -### Configure with Environment Variables - -Customize the WebUI using Gradio environment variables: - -```bash -# Enable public sharing -GRADIO_SHARE=1 python -m tools.run_webui - -# Change server port -GRADIO_SERVER_PORT=8080 python -m tools.run_webui - -# Change server name -GRADIO_SERVER_NAME=0.0.0.0 python -m tools.run_webui -``` - -### Using Reference Audio Library - -For faster workflow, pre-save reference audio: - -1. Create a `references/` directory in the project root -2. Create subdirectories named by voice ID: `references//` -3. Place files in each subdirectory: - - `sample.wav` - Reference audio file - - `sample.lab` - Text transcription of the audio - -Example structure: -``` -references/ -├── alice/ -│ ├── sample.wav -│ └── sample.lab -└── bob/ - ├── sample.wav - └── sample.lab -``` - -These references will appear as selectable options in the WebUI. - -## GUI Inference - -For users who prefer a native desktop application, a PyQt6-based GUI is available. - -### Download GUI Client - -Download the latest release from the [Fish Speech GUI repository](https://github.com/AnyaCoder/fish-speech-gui/releases). - -**Supported platforms:** -- Linux -- Windows -- macOS - -### Connect to API Server - -The GUI client connects to a running API server (see [HTTP API Inference](#http-api-inference) above). - -1. Start the API server -2. Launch the GUI client -3. Configure the API endpoint (default: `http://localhost:8080`) - -## Docker Inference - -If you're using Docker deployment, refer to the [Docker Deployment guide](/developer-guide/self-hosting/docker-deployment) for detailed instructions on: - -- Running pre-built WebUI containers -- Running pre-built API server containers -- Customizing container configuration -- Volume mounts for models and references - -Quick example: - -```bash -# Start WebUI with Docker -docker run -d \ - --name fish-speech-webui \ - --gpus all \ - -p 7860:7860 \ - -v ./checkpoints:/app/checkpoints \ - -v ./references:/app/references \ - -e COMPILE=1 \ - fishaudio/fish-speech:latest-webui-cuda -``` - -## Performance Optimization - -### Enable Compilation - -Torch compilation provides ~10x speedup on compatible GPUs: - -```bash -# Add --compile flag to any inference command -python -m tools.api_server --compile ... -``` - - -Compilation requires: -- CUDA-compatible GPU -- Triton library (not supported on Windows/macOS) -- First run will be slow due to compilation overhead - - -### Use Mixed Precision - -For GPUs without bf16 support, use fp16: - -```bash -python fish_speech/models/text2semantic/inference.py --half ... -``` - -### Batch Processing - -For multiple audio generations, use batch processing to amortize model loading overhead: - -```python -# Example batch processing script -import fish_speech - -model = fish_speech.load_model("checkpoints/openaudio-s1-mini") - -texts = ["First sentence", "Second sentence", "Third sentence"] -for text in texts: - audio = model.synthesize(text) - audio.save(f"output_{texts.index(text)}.wav") -``` - -## Emotion Control - -Fish Audio S1 supports emotional markers for expressive speech synthesis: - -### Basic Emotions - -``` -(angry) (sad) (excited) (surprised) (satisfied) (delighted) -(scared) (worried) (upset) (nervous) (frustrated) (depressed) -(empathetic) (embarrassed) (disgusted) (moved) (proud) (relaxed) -(grateful) (confident) (interested) (curious) (confused) (joyful) -``` - -### Advanced Emotions - -``` -(disdainful) (unhappy) (anxious) (hysterical) (indifferent) -(impatient) (guilty) (scornful) (panicked) (furious) (reluctant) -(keen) (disapproving) (negative) (denying) (astonished) (serious) -(sarcastic) (conciliative) (comforting) (sincere) (sneering) -(hesitating) (yielding) (painful) (awkward) (amused) -``` - -### Tone Markers - -``` -(in a hurry tone) (shouting) (screaming) (whispering) (soft tone) -``` - -### Special Effects - -``` -(laughing) (chuckling) (sobbing) (crying loudly) (sighing) (panting) -(groaning) (crowd laughing) (background laughter) (audience laughing) -``` - -### Example Usage - -```bash -python fish_speech/models/text2semantic/inference.py \ - --text "(excited)This is amazing! (laughing)Ha ha ha!" \ - --compile -``` - - -Emotion control is currently supported for English, Chinese, and Japanese. More languages coming soon! - - -For more details, see the [Emotion Control guide](/developer-guide/core-features/emotions). - -## Troubleshooting - -### Out of Memory Errors - -If you encounter CUDA out of memory errors: - -1. Reduce input text length -2. Use `--half` flag for fp16 inference -3. Close other GPU applications -4. Use a smaller batch size - -### Slow Inference - -To improve speed: - -1. Enable `--compile` flag -2. Verify GPU is being used (check with `nvidia-smi`) -3. Ensure CUDA version matches PyTorch installation -4. Use fp16 instead of bf16 on older GPUs - -### Poor Audio Quality - -For better quality: - -1. Use high-quality reference audio (clear, no background noise) -2. Ensure reference text accurately matches reference audio -3. Use 10-30 seconds of reference audio -4. See [Voice Cloning Best Practices](/developer-guide/best-practices/voice-cloning) - -### Model Loading Errors - -If models fail to load: - -1. Verify model weights are downloaded completely -2. Check checkpoint paths are correct -3. Ensure sufficient disk space -4. Re-download weights if corrupted - -## Next Steps - -- **[Emotion Control Best Practices](/developer-guide/best-practices/emotion-control)** - Master expressive speech -- **[Voice Cloning Best Practices](/developer-guide/best-practices/voice-cloning)** - Optimize voice cloning quality -- **[API Reference](/api-reference/introduction)** - Integrate with your applications -- **[Cloud API](https://fish.audio)** - Compare with managed service performance diff --git a/docs.json b/docs.json index e41f188..676e6bc 100644 --- a/docs.json +++ b/docs.json @@ -214,9 +214,14 @@ { "group": "Self-Hosting", "pages": [ - "developer-guide/self-hosting/local-setup", - "developer-guide/self-hosting/docker-deployment", - "developer-guide/self-hosting/running-inference" + "developer-guide/self-hosting/introduction", + "developer-guide/self-hosting/requirements", + "developer-guide/self-hosting/registry-access", + "developer-guide/self-hosting/kubernetes", + "developer-guide/self-hosting/all-in-one", + "developer-guide/self-hosting/air-gapped", + "developer-guide/self-hosting/operations", + "developer-guide/self-hosting/enterprise-releases" ] }, { @@ -637,6 +642,22 @@ { "source": "/developer-guide/getting-started/introduction", "destination": "/overview/capabilities" + }, + { + "source": "/self-host", + "destination": "/developer-guide/self-hosting/introduction" + }, + { + "source": "/developer-guide/self-hosting/local-setup", + "destination": "/developer-guide/self-hosting/introduction" + }, + { + "source": "/developer-guide/self-hosting/docker-deployment", + "destination": "/developer-guide/self-hosting/introduction" + }, + { + "source": "/developer-guide/self-hosting/running-inference", + "destination": "/developer-guide/self-hosting/introduction" } ], "footer": {