Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 21 additions & 21 deletions .mintlify/skills/fish-audio-api/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This file condenses those into rules an agent can apply directly.
- Auth (all endpoints): `Authorization: Bearer <FISH_API_KEY>`
- Optional distributed tracing for inference APIs: see `https://docs.fish.audio/api-reference/observability`.
- Get API keys: `https://fish.audio/app/api-keys`
- Never hardcode keys — read from an env var like `FISH_API_KEY`.
- Never hardcode keys. Read from an env var like `FISH_API_KEY`.
- Errors are JSON `{status, message}` for 401 / 402 / 404, and an array of `{loc, type, msg, ctx, in}` for 422 (validation).

## Endpoint map
Expand All @@ -38,7 +38,7 @@ This file condenses those into rules an agent can apply directly.
| GET | `/wallet/{user_id}/api-credit` | API credit balance (`user_id` defaults to `self`) |
| WSS | `/v1/tts/live` | Real-time TTS streaming (MessagePack frames) |

## Text-to-Speech `POST /v1/tts`
## Text-to-Speech: `POST /v1/tts`

Required headers:

Expand All @@ -47,7 +47,7 @@ Required headers:

Optional headers:

- `model`values: `s1`, `s2-pro`, `s2.1-pro`, `s2.1-pro-free`. If omitted or unrecognized, the server falls back to `s2.1-pro` (paid). Default to `s2.1-pro` for production; use `s2.1-pro-free` for free-tier evaluation and prototyping (same model, no TTFA/DPA guarantees).
- `model`: values `s1`, `s2-pro`, `s2.1-pro`, `s2.1-pro-free`. If omitted or unrecognized, the server falls back to `s2.1-pro` (paid). Default to `s2.1-pro` for production; use `s2.1-pro-free` for free-tier evaluation and prototyping (same model, no TTFA/DPA guarantees).

Response: streaming audio bytes (`Transfer-Encoding: chunked`) in the format set by `format`. Write to a file or pipe to a player. There is **no JSON wrapper** on success.

Expand Down Expand Up @@ -197,15 +197,15 @@ if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
await pipeline(Readable.fromWeb(res.body), createWriteStream("out.mp3"));
```

## Speech-to-Text `POST /v1/asr`
## Speech-to-Text: `POST /v1/asr`

Required headers: `Authorization`. Content type: `multipart/form-data` or `application/msgpack`.

Form fields:

- `audio` (binary, required)
- `language` (string, optional; omit to auto-detect)
- `ignore_timestamps` (bool, default `true`; set `false` to get per-segment timestamps adds latency on clips < 30 s)
- `ignore_timestamps` (bool, default `true`; set `false` to get per-segment timestamps, which adds latency on clips < 30 s)

Response (200):

Expand Down Expand Up @@ -244,7 +244,7 @@ r.raise_for_status()
print(r.json()["text"])
```

## Voice Design `POST /v1/voice-design`
## Voice Design: `POST /v1/voice-design`

Required headers:

Expand Down Expand Up @@ -315,11 +315,11 @@ with open("voice.wav", "wb") as f:

Billing: one successful generation request is charged once, even when it returns multiple candidates. Authentication, validation, balance, concurrency, and service errors are not billed.

## Voice models `/model`
## Voice models: `/model`

### List: `GET /model`

Query params: `page_size` (default 10), `page_number` (default 1), `title`, `tag` (string or array), `self` (bool only your models), `author_id`, `language`, `title_language`, `sort_by` (`score` | `task_count` | `created_at`, default `score`).
Query params: `page_size` (default 10), `page_number` (default 1), `title`, `tag` (string or array), `self` (bool; only your models), `author_id`, `language`, `title_language`, `sort_by` (`score` | `task_count` | `created_at`, default `score`).

Returns `{total, items: ModelEntity[]}`.

Expand Down Expand Up @@ -349,7 +349,7 @@ Returns 201 with the full `ModelEntity` including `_id`, `state` (`created` | `t
### Get / Update / Delete

- `GET /model/{id}` → `ModelEntity`
- `PATCH /model/{id}` JSON, form-urlencoded, multipart, or msgpack. Nullable fields: `title`, `description`, `cover_image` (binary), `visibility`, `tags`.
- `PATCH /model/{id}`: JSON, form-urlencoded, multipart, or msgpack. Nullable fields: `title`, `description`, `cover_image` (binary), `visibility`, `tags`.
- `DELETE /model/{id}` → 200 on success.

```bash
Expand All @@ -362,32 +362,32 @@ curl --request PATCH https://api.fish.audio/model/<id> \
## Wallet

- `GET /wallet/self/package` → `{user_id, type, total, balance, created_at, updated_at, finished_at}`
- `GET /wallet/self/api-credit` → `{_id, user_id, credit, created_at, updated_at, has_phone_sha256, has_free_credit}`. Pass `?check_free_credit=true` to also populate `has_free_credit` (default `false` the field is `null` when not checked).
- `GET /wallet/self/api-credit` → `{_id, user_id, credit, created_at, updated_at, has_phone_sha256, has_free_credit}`. Pass `?check_free_credit=true` to also populate `has_free_credit` (default `false`; the field is `null` when not checked).

Replace `self` with a specific `user_id` if you have permission; otherwise always use `self`.

## WebSocket TTS `wss://api.fish.audio/v1/tts/live`
## WebSocket TTS: `wss://api.fish.audio/v1/tts/live`

For low-latency / streaming TTS (e.g. LLM token stream → speech). All frames are **MessagePack-encoded** binary messages.

### Connection headers

- `Authorization: Bearer <FISH_API_KEY>`
- `model` optional; same values and fallback behavior as `POST /v1/tts` (falls back to `s2.1-pro` when omitted or unrecognized)
- `model`: optional; same values and fallback behavior as `POST /v1/tts` (falls back to `s2.1-pro` when omitted or unrecognized)

### Event sequence

Client → server:

1. `StartEvent` once, first message: `{event: "start", request: <TTSRequest>}`. The `request` object is the same schema as `POST /v1/tts` above. Usually `request.text = ""` and the real text streams in `TextEvent`s.
2. `TextEvent` one per text chunk: `{event: "text", text: "..."}`. Send as many as needed.
3. `FlushEvent` optional: `{event: "flush"}`. Forces the server to synthesize buffered text immediately (use for turn-taking / low-latency flushes).
4. `CloseEvent` final: `{event: "stop"}`. **Note the literal is `stop`, not `close`.**
1. `StartEvent` (once, first message): `{event: "start", request: <TTSRequest>}`. The `request` object is the same schema as `POST /v1/tts` above. Usually `request.text = ""` and the real text streams in `TextEvent`s.
2. `TextEvent` (one per text chunk): `{event: "text", text: "..."}`. Send as many as needed.
3. `FlushEvent` (optional): `{event: "flush"}`. Forces the server to synthesize buffered text immediately (use for turn-taking / low-latency flushes).
4. `CloseEvent` (final): `{event: "stop"}`. **Note the literal is `stop`, not `close`.**

Server → client:

- `AudioEvent`: `{event: "audio", audio: <bytes>}` — many of these, concatenate in order to reconstruct the audio stream in the format set by `request.format`.
- `FinishEvent`: `{event: "finish", reason: "stop" | "error"}` — exactly one, then the server closes the socket. Ignore unknown events for forward compatibility.
- `AudioEvent`: `{event: "audio", audio: <bytes>}`. Many of these; concatenate in order to reconstruct the audio stream in the format set by `request.format`.
- `FinishEvent`: `{event: "finish", reason: "stop" | "error"}`. Exactly one, then the server closes the socket. Ignore unknown events for forward compatibility.

### Python example (`websockets>=14` + `msgpack`)

Expand Down Expand Up @@ -489,11 +489,11 @@ ws.on("message", (buf) => {

## Emotion / expression control

The S1 model uses `(parenthesis)` tags inside `text`, e.g. `(happy) What a day!`. S2-Pro uses free-form `[bracket]` natural-language tags, e.g. `[slightly sarcastic, rising tone]`. Either works through `text` no separate parameter. Full list: `https://docs.fish.audio/api-reference/emotion-reference.md`.
The S1 model uses `(parenthesis)` tags inside `text`, e.g. `(happy) What a day!`. S2-Pro uses free-form `[bracket]` natural-language tags, e.g. `[slightly sarcastic, rising tone]`. Either works through `text`; there is no separate parameter. Full list: `https://docs.fish.audio/api-reference/emotion-reference.md`.

## Encoding and content-type rules

- Use `application/json` for normal TTS requests — it's the simplest and works for `reference_id` flows.
- Use `application/json` for normal TTS requests. It's the simplest and works for `reference_id` flows.
- Use `application/msgpack` when you need to send raw audio bytes inline (inline `references`, or the WebSocket protocol).
- Use `multipart/form-data` for `/v1/asr` and `POST /model` because they upload files.
- All WebSocket frames are MessagePack binary, regardless of inner payload.
Expand All @@ -508,7 +508,7 @@ The S1 model uses `(parenthesis)` tags inside `text`, e.g. `(happy) What a day!`
- `references` sent with `Content-Type: application/json` (must be msgpack).
- Numeric param out of range (`temperature`, `top_p`, `chunk_length`, `min_chunk_length`, `prosody.speed`, `early_stop_threshold`).
- `mp3_bitrate` / `opus_bitrate` set without matching `format`.
- WebSocket: a `finish` event with `reason: "error"` means the server failed mid-stream — surface the message and reconnect rather than retrying on the same socket.
- WebSocket: a `finish` event with `reason: "error"` means the server failed mid-stream. Surface the message and reconnect rather than retrying on the same socket.

## Decision shortcuts

Expand Down
22 changes: 11 additions & 11 deletions .mintlify/skills/fish-audio-sdk/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
---
name: fish-audio-sdk
description: Write code with the official Fish Audio SDKs Python (`fishaudio`, PyPI `fish-audio-sdk`) and JavaScript/TypeScript (`fish-audio`). Use when the user wants text-to-speech, speech-to-text, voice cloning / voice-model management, or realtime WebSocket TTS through the installed SDK rather than raw HTTP. Covers install and auth, sync + async Python, the TypeScript client, exact method signatures and defaults, model selection (including the S2.1 typing caveat), the real exception types, and the Python↔JavaScript naming differences. For raw REST/WebSocket calls without an SDK (curl, unsupported languages, edge runtimes), use the `fish-audio-api` skill instead.
description: Write code with the official Fish Audio SDKs, Python (`fishaudio`, PyPI `fish-audio-sdk`) and JavaScript/TypeScript (`fish-audio`). Use when the user wants text-to-speech, speech-to-text, voice cloning / voice-model management, or realtime WebSocket TTS through the installed SDK rather than raw HTTP. Covers install and auth, sync + async Python, the TypeScript client, exact method signatures and defaults, model selection (including the S2.1 typing caveat), the real exception types, and the Python↔JavaScript naming differences. For raw REST/WebSocket calls without an SDK (curl, unsupported languages, edge runtimes), use the `fish-audio-api` skill instead.
---

# Fish Audio SDK Skill

Use this skill to generate correct, runnable code with the **official Fish Audio SDKs**:

- **Python**package `fish-audio-sdk` on PyPI, imported as `fishaudio`. (The same wheel still ships a separate legacy `fish_audio_sdk` package — do **not** mix them; everything here is the modern `fishaudio` package.)
- **JavaScript / TypeScript** package `fish-audio` on npm, imported as `FishAudioClient`.
- **Python**: package `fish-audio-sdk` on PyPI, imported as `fishaudio`. (The same wheel still ships a separate legacy `fish_audio_sdk` package. Do **not** mix them; everything here is the modern `fishaudio` package.)
- **JavaScript / TypeScript**: package `fish-audio` on npm, imported as `FishAudioClient`.

If the user wants raw `curl` / HTTP / WebSocket without installing an SDK, use the **`fish-audio-api`** skill instead.

> This file is the index. Deeper, task-specific rules and full examples live in [`references/`](references/). Read the reference for the task you're doing before writing code.

## Global facts

- **Auth:** both SDKs read the API key from the `FISH_API_KEY` environment variable automatically. Get keys at `https://fish.audio/app/api-keys`. Never hardcode a key — read it from the environment.
- **Auth:** both SDKs read the API key from the `FISH_API_KEY` environment variable automatically. Get keys at `https://fish.audio/app/api-keys`. Never hardcode a key.
- **Base URL:** `https://api.fish.audio` (override with `base_url=` in Python / `baseUrl:` in JS).
- **Models:** the API supports `s1`, `s2-pro`, `s2.1-pro` (recommended for production), and `s2.1-pro-free` (free tier), but the SDK type definitions currently list only `s1` and `s2-pro` (`s2-pro` = SDK default). Both SDKs forward the model value without runtime validation, so `"s2.1-pro"` works over the wire — static type checkers will flag it, so add `# type: ignore` (Python) / an `as` cast (TS), or use the `fish-audio-api` skill for raw calls. `speech-1.5` / `speech-1.6` are **deprecated**. In Python pass `model="s2-pro"` (keyword); in JS pass the **positional** `backend` argument.
- **Models:** the API supports `s1`, `s2-pro`, `s2.1-pro` (recommended for production), and `s2.1-pro-free` (free tier), but the SDK type definitions currently list only `s1` and `s2-pro` (`s2-pro` = SDK default). Both SDKs forward the model value without runtime validation, so `"s2.1-pro"` works over the wire. Static type checkers will flag it, so add `# type: ignore` (Python) / an `as` cast (TS), or use the `fish-audio-api` skill for raw calls. `speech-1.5` / `speech-1.6` are **deprecated**. In Python pass `model="s2-pro"` (keyword); in JS pass the **positional** `backend` argument.
- **Audio formats:** `mp3` (default), `wav`, `pcm`, `opus`.
- **Playback in examples:** `play()` shells out to a system audio tool Python uses **ffmpeg/ffplay** (or `mpv`), JS uses **ffplay**. It is for local/desktop use; in a server, `save()` to a file or stream the bytes instead. See [references/installation.md](references/installation.md).
- **Playback in examples:** `play()` shells out to a system audio tool: Python uses **ffmpeg/ffplay** (or `mpv`), JS uses **ffplay**. It is for local/desktop use; in a server, `save()` to a file or stream the bytes instead. See [references/installation.md](references/installation.md).

## Quick start Python
## Quick start: Python

```python
from fishaudio import FishAudio
Expand All @@ -37,7 +37,7 @@ save(audio, "output.mp3") # write to a file
# play(audio) # or play locally (needs ffmpeg)
```

Async identical resource tree on `AsyncFishAudio`, used as a context manager:
Async: identical resource tree on `AsyncFishAudio`, used as a context manager:

```python
import asyncio
Expand All @@ -52,7 +52,7 @@ async def main():
asyncio.run(main())
```

## Quick start JavaScript / TypeScript
## Quick start: JavaScript / TypeScript

```ts
import { FishAudioClient, play } from "fish-audio";
Expand Down Expand Up @@ -113,7 +113,7 @@ The two SDKs do **not** use the same names. Use this map when porting code betwe

## Gotchas (verified against the SDK source)

- Python `latency` accepts only **`"normal"` or `"balanced"`** (default `"balanced"`) there is no `"low"`.
- Python `latency` accepts only **`"normal"` or `"balanced"`** (default `"balanced"`); there is no `"low"`.
- The Python client has **no `max_retries`** and does **not** auto-retry; the JS client **does** auto-retry (configurable via per-call `requestOptions.maxRetries`). See [errors](references/errors.md).
- Python defines a `ValidationError` class but **never raises it** don't catch it expecting validation failures; a 422 surfaces as `APIError`. The JS SDK throws `UnprocessableEntityError` on 422.
- Python defines a `ValidationError` class but **never raises it**, so don't catch it expecting validation failures; a 422 surfaces as `APIError`. The JS SDK throws `UnprocessableEntityError` on 422.
- ASR segment `start` / `end` are in **seconds**, but `duration` is in **milliseconds**. See [speech-to-text](references/speech-to-text.md).
10 changes: 5 additions & 5 deletions .mintlify/skills/fish-audio-sdk/references/errors.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Errors, Retries & Timeouts

The two SDKs have **different** exception models. The tables below reflect what the SDK source actually raises not every exported class is thrown.
The two SDKs have **different** exception models. The tables below reflect what the SDK source actually raises; not every exported class is thrown.

## Python exceptions

Expand All @@ -9,9 +9,9 @@ Hierarchy (all subclasses of `FishAudioError`):
| Exception | When | Attributes |
| --------------------- | ---------------------------------------------- | --------------------------------- |
| `APIError` | base for HTTP errors | `.status`, `.message`, `.body` |
| `AuthenticationError` | 401 bad/missing key | (APIError) |
| `AuthenticationError` | 401 (bad/missing key) | (APIError) |
| `PermissionError` | 403 | (APIError) |
| `NotFoundError` | 404 voice id not found | (APIError) |
| `NotFoundError` | 404 (voice id not found) | (APIError) |
| `RateLimitError` | 429 | (APIError) |
| `ServerError` | 5xx | (APIError) |
| `WebSocketError` | realtime stream failed | — |
Expand Down Expand Up @@ -48,7 +48,7 @@ except FishAudioError as e:

- **No automatic retries.** The Python client makes a single request and raises on failure. Implement your own retry loop if you need one (e.g. back off on `RateLimitError`).
- **Timeout** is set on the client: `FishAudio(timeout=240.0)` (seconds, default 240).
- `RequestOptions(max_retries=...)` exists but is currently a **no-op** don't rely on it. `RequestOptions(timeout=..., additional_headers=...)` does work per request:
- `RequestOptions(max_retries=...)` exists but is currently a **no-op**, so don't rely on it. `RequestOptions(timeout=..., additional_headers=...)` does work per request:

```python
from fishaudio.core.request_options import RequestOptions
Expand Down Expand Up @@ -92,7 +92,7 @@ What the JS client actually throws:

| Error | When |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `UnprocessableEntityError` (extends `FishAudioError`) | 422 the **only** typed HTTP subclass thrown; `.body` is `{ detail: [{ loc, msg, type }] }` |
| `UnprocessableEntityError` (extends `FishAudioError`) | 422, the **only** typed HTTP subclass thrown; `.body` is `{ detail: [{ loc, msg, type }] }` |
| `FishAudioError` | every other non-2xx response; read `.statusCode`, `.body`, `.rawResponse` |
| `FishAudioTimeoutError` | request exceeded the timeout |

Expand Down
4 changes: 2 additions & 2 deletions .mintlify/skills/fish-audio-sdk/references/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Never hardcode a key in source. If neither the argument nor `FISH_API_KEY` is se
| Request timeout | `timeout=240.0` (seconds) | per-call `requestOptions.timeoutInSeconds` |
| Custom HTTP client | `httpx_client=` | (not exposed) |

> Python caveat: if you pass your own `httpx_client`, the SDK uses it **as-is** — your `base_url`, `timeout`, and the `Authorization` header are **not** applied to it. Pre-configure those on the client you inject.
> Python caveat: if you pass your own `httpx_client`, the SDK uses it **as-is**. Your `base_url`, `timeout`, and the `Authorization` header are **not** applied to it. Pre-configure those on the client you inject.

There is no client-level `max_retries` or `default_headers` option in Python. Per-request headers go through `request_options`. See [errors.md](errors.md) for retry/timeout behavior.

Expand All @@ -72,7 +72,7 @@ brew install ffmpeg
sudo apt-get install ffmpeg
```

In a server or browser context, don't use `play()` — use `save()` (Python) or write/stream the bytes yourself.
In a server or browser context, don't use `play()`. Use `save()` (Python) or write/stream the bytes yourself.

## Verify a key works

Expand Down
Loading
Loading