Compositions documentation

.gitmodules

@@ -16,3 +16,6 @@
 [submodule "packages/python-server-sdk"]
 	path = packages/python-server-sdk
 	url = https://github.com/fishjam-cloud/python-server-sdk.git
+[submodule "api/smelter-cloud"]
+	path = api/composition
+	url = git@github.com:fishjam-cloud/foundry.git

docs/api/reference.md

@@ -6,7 +6,7 @@ type: reference
 
 Describes APIs for direct interaction with Fishjam.
 
-Fishjam publishes documentation for the Sandbox API and Fishjam Server APIs.
+Fishjam publishes documentation for the Sandbox API, Fishjam Server APIs, and the Composition API.
 
 ## Sandbox API
 
@@ -51,3 +51,56 @@ the first message that must be sent is an `AuthRequest`,
 with a valid Management Token.
 
 Next, you can should subscribe to notifications by sending `SubscribeRequest` event with `SERVER_NOTIFICATION` event type.
+
+## Compositions
+
+[Compositions](../explanation/compositions) are managed through the Composition API: a REST API plus a WebSocket stream for engine events. All requests go to `https://rtc.fishjam.io`.
+
+### REST API
+
+[Composition REST API Reference](/api/compositions)
+
+The [OpenAPI document](https://github.com/fishjam-cloud/documentation/blob/main/static/api/composition-openapi.json) is generated from the service's source code and republished together with documentation updates.
+
+### WebSocket event stream
+
+Some engine events (for example, an output finishing) are delivered over a WebSocket rather than HTTP. Connect to:
+
+```
+GET wss://rtc.fishjam.io/api/composition/{composition_id}/ws
+```
+
+Because browsers cannot set an `Authorization` header on a WebSocket, authentication rides on the `Sec-WebSocket-Protocol` header, which must carry **two** subprotocols (order does not matter):
+
+- `json.fishjam.io`: selects the JSON wire format.
+- `bearer.auth.fishjam.io.<token>`: your token, appended to the literal prefix.
+
+```js
+const ws = new WebSocket(
+  `wss://rtc.fishjam.io/api/composition/${compositionId}/ws`,
+  ["json.fishjam.io", `bearer.auth.fishjam.io.${token}`],
+);
+```
+
+Messages are JSON text frames, each with a `type` field identifying the event.
+
+### Composition authentication
+
+| What you're calling                                                         | How it authenticates                                                                                                                                    |
+| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Control-plane calls (composition, inputs, outputs, renderers, events, room) | `Authorization: Bearer <token>`: your Fishjam **Management Token**, the same token used across Fishjam.                                                 |
+| Publishing to an input (`/whip/{input_id}`)                                 | The **input's** own bearer token, returned when you register a `whip_server` input (or the one you supplied). This is distinct from your account token. |
+| Playing back an output (`/whep/{output_id}`)                                | Optional. Public playback is allowed; add a Bearer token only if you want to require one.                                                               |
+| The WebSocket event stream                                                  | The subprotocol scheme described above.                                                                                                                 |
+
+Get your Management Token from the [**Fishjam developer panel**](https://fishjam.io/app).
+
+### Errors
+
+Every non-2xx response of the Composition API is a JSON object:
+
+```json
+{ "message": "Composition not found", "http_status_code": 404 }
+```
+
+Common statuses are `400` (bad request), `401` (unauthorized), `404` (not found), `500` (server error), and `503` (no capacity, returned by composition creation).

docs/explanation/compositions.mdx

@@ -0,0 +1,123 @@
+---
+type: explanation
+title: Compositions
+sidebar_position: 4.5
+description: Understand compositions, the Fishjam feature that mixes multiple live media streams into a single composed output in real time.
+---
+
+# Compositions
+
+_Understanding real-time stream composition in Fishjam_
+
+A **composition** mixes multiple live media streams into a single output stream in real time. You send inputs (WebRTC, RTMP, HLS, or MP4), describe how they should be laid out, and it produces one composed output that you can publish anywhere, all without running any rendering infrastructure yourself.
+
+Compositions are built on [Smelter](https://smelter.dev), the open-source real-time video compositing engine by [Software Mansion](https://swmansion.com), and are a native part of Fishjam: they authenticate with the same Fishjam credentials and can compose the peers of a [Fishjam room](./rooms) directly.
+
+## What you can build
+
+- **Multi-party layouts**: arrange the cameras of a conference or livestream into grids, side-by-sides, or picture-in-picture.
+- **Branded streams**: overlay logos, captions, lower-thirds, and backgrounds on top of live video.
+- **React templates**: describe your layout as a React component that reacts to who is speaking, who joined, and custom application events.
+- **Cross-protocol bridging**: take WebRTC inputs and republish the composed result to an RTMP destination, or vice versa.
+
+## Core concepts
+
+A **composition** is a single running compositing session. Into it you register:
+
+- **Inputs**: the live media sources being composed (WebRTC via WHIP/WHEP, RTMP, HLS, or MP4).
+- **Outputs**: where the composed result is sent (WebRTC via WHIP, or RTMP). Each output carries a **scene** that describes the layout.
+- **Renderers**: shared assets such as images and fonts you can place in a scene.
+
+An output's scene can either be described directly in the API or rendered by a **template**: a React component, written with the layout components and the [`@fishjam-cloud/composition`](./../how-to/compositions/write-and-deploy-a-template) hooks, that updates the layout live as the room changes.
+
+## Scenes
+
+Every video output carries a **scene**: a tree of components that describes how inputs, text, and images are arranged into the composed frame. Audio outputs carry an **audio scene** that describes which inputs are mixed together.
+
+### The video scene tree
+
+A video scene has a single `root` component. Each component has a `type` that determines how it lays out its children.
+
+```json
+{
+  "root": {
+    "type": "tiles",
+    "children": [
+      { "type": "input_stream", "input_id": "camera_1" },
+      { "type": "input_stream", "input_id": "camera_2" }
+    ]
+  }
+}
+```
+
+The available component types are:
+
+| `type`         | Purpose                                                                          |
+| -------------- | -------------------------------------------------------------------------------- |
+| `input_stream` | Renders one registered input. Identified by `input_id`.                          |
+| `view`         | A container you position and style; the basic building block for custom layouts. |
+| `tiles`        | Automatically arranges its children into a grid.                                 |
+| `rescaler`     | Fits a single child into a target area, preserving aspect ratio.                 |
+| `text`         | Renders a text string.                                                           |
+| `image`        | Renders a registered image. Identified by `image_id`.                            |
+
+Components nest freely: a `tiles` of `rescaler`s wrapping `input_stream`s, a `view` with a `text` caption over an `input_stream`, and so on. The styling and full property set of each component come from Smelter itself; see the [Smelter component reference](https://smelter.dev) for every available property.
+
+### The audio scene
+
+An audio scene lists the inputs to mix and, optionally, their relative volume:
+
+```json
+{
+  "inputs": [
+    { "input_id": "camera_1" },
+    { "input_id": "camera_2", "volume": 0.5 }
+  ]
+}
+```
+
+`volume` defaults to `1.0`. Only the inputs you list are audible in the output.
+
+### Setting a scene
+
+You provide the initial scene when you register an output, under `video.initial` (a video scene) and `audio.initial` (an audio scene). See [Choose inputs and outputs](./../how-to/compositions/inputs-and-outputs) for the full output shape.
+
+### Updating a scene live
+
+To change the layout while a composition is running, call the output's update endpoint with the new `video` and/or `audio` scene:
+
+```bash
+curl -X POST "$COMPOSITION_URL/api/composition/$COMPOSITION/output/main/update" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "video": {
+      "root": {
+        "type": "rescaler",
+        "child": { "type": "input_stream", "input_id": "camera_1" }
+      }
+    }
+  }'
+```
+
+Only the sides you include are changed; omitting `audio` leaves the audio scene untouched, and vice versa.
+
+An update can be scheduled instead of applied immediately by adding `schedule_time_ms`, the composition timeline offset in milliseconds at which the change should take effect:
+
+```json
+{
+  "video": { "root": { "type": "tiles", "children": [] } },
+  "schedule_time_ms": 5000
+}
+```
+
+### What if I don't want to update the scene on every change?
+
+Pushing a scene update every time a participant joins, leaves, or mutes gets tedious fast. Instead, you can describe the layout once as a **template**: a React component that receives the live room state and re-renders automatically as it changes, cutting manual scene management down to a minimum. See [Write and deploy a template](./../how-to/compositions/write-and-deploy-a-template) to build one.
+
+## Where to go next
+
+- [Compositions tutorial](./../tutorials/compositions): create your first composition end to end.
+- [Write and deploy a template](./../how-to/compositions/write-and-deploy-a-template): build a React layout with the composition SDK.
+- [Compose a Fishjam room](./../how-to/compositions/compose-a-fishjam-room): turn a room's peers into one composed stream.
+- [Composition API](./../api/reference#compositions): the full REST surface.

docs/how-to/compositions/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Compositions",
+  "position": 5,
+  "link": {
+    "type": "generated-index",
+    "description": "Compose rooms and live streams into new outputs with templates, events, and multiple transports."
+  }
+}

docs/how-to/compositions/compose-a-fishjam-room.mdx

@@ -0,0 +1,137 @@
+---
+type: how-to
+sidebar_position: 2
+description: Forward a Fishjam room into a composition and render its peers with a live React template.
+---
+
+# Compose a Fishjam room
+
+A composition can include the peers of a [Fishjam room](./../../explanation/rooms). You forward the room's tracks to the composition with a single Fishjam API call: Fishjam then pushes each participant's media into the composition as inputs, and hooks from `@fishjam-cloud/composition` let a [template](./write-and-deploy-a-template) render one tile per participant and push the result to a [livestream](./../../explanation/livestreams).
+
+```
+Fishjam room (peers) ──forwarded──▶ composition (template) ──WHIP──▶ Fishjam livestream ──WHEP──▶ [viewers]
+```
+
+## Prerequisites
+
+- A Fishjam room created with `"videoCodec": "h264"` (compositions consume h264 video), with peers publishing into it.
+- A [livestream](./../../explanation/livestreams) (or any other WHIP/RTMP destination) to publish the composed stream to.
+- A template project scaffolded with the composition CLI (see [Write and deploy a template](./write-and-deploy-a-template)).
+
+## Step 1: Write a room-aware template
+
+Hooks from `@fishjam-cloud/composition` give the template live room state; it re-renders automatically as participants join, leave, mute, or speak.
+
+```tsx
+// @jsx: react-jsx
+// ---cut-before---
+import { InputStream, Rescaler, Text, Tiles, View } from "@swmansion/smelter";
+import { usePeers, useSpeakingState } from "@fishjam-cloud/composition";
+import type { PeerWithStreams } from "@fishjam-cloud/composition";
+
+type PeerMetadata = { displayName?: string };
+
+function PeerTile({ peer }: { peer: PeerWithStreams<PeerMetadata> }) {
+  const camera = peer.cameraStream;
+  const cameraOn = camera?.video && !camera.video.paused;
+  const speaking = useSpeakingState(peer.id) === "speech";
+  const name = peer.metadata.peer?.displayName ?? peer.id;
+
+  return (
+    <View style={{ borderWidth: speaking ? 4 : 0, borderColor: "#00cc66ff" }}>
+      {cameraOn ? (
+        <Rescaler>
+          <InputStream inputId={camera.inputId} />
+        </Rescaler>
+      ) : (
+        <View>
+          <Text>{name}</Text>
+        </View>
+      )}
+    </View>
+  );
+}
+
+export default function App() {
+  const peers = usePeers<PeerMetadata>();
+  const connected = peers.filter((peer) => peer.streams.length > 0);
+
+  return (
+    <View style={{ backgroundColor: "#0b1020ff" }}>
+      <Tiles style={{ padding: 8 }}>
+        {connected.map((peer) => (
+          <PeerTile peer={peer} key={peer.id} />
+        ))}
+      </Tiles>
+    </View>
+  );
+}
+```
+
+| Hook                       | Returns                                                                                                        |
+| -------------------------- | -------------------------------------------------------------------------------------------------------------- |
+| `usePeers()`               | All peers in the forwarded room, each with its streams (`cameraStream`, `screenShareStream`, `customStreams`). |
+| `usePeer(peerId)`          | A single peer, or `undefined`.                                                                                 |
+| `useRoom()`                | The forwarded room `{ id }`, or `undefined` before a room is forwarded.                                        |
+| `useSpeakingState(peerId)` | `"speech"` or `"silence"` for active-speaker highlighting.                                                     |
+
+The key link is `stream.inputId`: you pass it to `<InputStream inputId={…} />` to render that participant's forwarded track. A peer's streams fill in once its media actually starts flowing into the composition.
+
+Build the bundle as usual with `npm run build`.
+
+## Step 2: Create the composition and register the templated output
+
+Create the composition with auto-start and auto-cleanup off, since the room's inputs only appear once forwarding starts:
+
+```bash
+curl -X POST "$COMPOSITION_URL/api/composition" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{ "autostart": false, "cleanup_without_inputs": false }'
+```
+
+Register a templated `whip_client` output that pushes to your livestream's WHIP endpoint. The output configuration and the template bundle go together in one multipart request:
+
+```bash
+curl -X POST "$COMPOSITION_URL/api/composition/$COMPOSITION/output/main/template" \
+  -H "Authorization: Bearer $TOKEN" \
+  -F 'config={
+        "type": "whip_client",
+        "endpoint_url": "<LIVESTREAM_WHIP_URL>",
+        "bearer_token": "<LIVESTREAM_STREAMER_TOKEN>",
+        "video": { "resolution": { "width": 1280, "height": 720 }, "initial": { "root": { "type": "view" } } },
+        "audio": { "initial": { "inputs": [] } }
+      };type=application/json' \
+  -F "template=@dist/App.js"
+```
+
+## Step 3: Forward the room into the composition
+
+One call to the [Fishjam Server API](/api/rest) wires everything up:
+
+```bash
+curl -X POST "https://fishjam.io/api/v1/connect/$FISHJAM_ID/room/$ROOM_ID/track_forwardings" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d "{ \"compositionURL\": \"$COMPOSITION_URL/api/composition/$COMPOSITION\", \"selector\": \"all\" }"
+```
+
+Everything else happens automatically: Fishjam links the room to the composition, registers an input for every forwarded track, and streams the media in. Your template's `usePeers()` fills with the room's peers as their media starts flowing. You never register room inputs by hand.
+
+## Step 4: Start the composition
+
+```bash
+curl -X POST "$COMPOSITION_URL/api/composition/$COMPOSITION/start" \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+Viewers can now watch the composed grid through the livestream's WHEP endpoint.
+
+## Step 5: Clean up
+
+Delete the composition when you are done; deleting the room (or removing its track forwarding) stops the media on the Fishjam side.
+
+```bash
+curl -X DELETE "$COMPOSITION_URL/api/composition/$COMPOSITION" \
+  -H "Authorization: Bearer $TOKEN"
+```