diff --git a/README.md b/README.md index 395e7da..8c474bd 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ Clone this repository, or open the directory of the example you want. Each examp ## Examples +- [bulk-csv-videos](examples/bulk-csv-videos) renders one video per row of a CSV from a single template with merge fields, tracked in a resumable manifest, with an optional AI step where Claude writes each row's headline and image prompt. Companion code for [Generate videos in bulk with an API and an AI agent](https://shotstack.io/learn/bulk-create-videos-from-csv-and-ai/). - [instagram-ai-video](examples/instagram-ai-video) generates a script, voiceover and background image with AI, renders a 1080x1920 video, and publishes it as an Instagram Reel. Companion code for [How to automate Instagram posts with AI video](https://shotstack.io/learn/automate-instagram-posts-with-ai-video/). - [rapidreels](examples/rapidreels) creates faceless short-form videos using generative AI. [View demo](https://shotstack.io/demos/social-media-video-maker/). - [reelestate](examples/reelestate) turns static real estate images into fully edited video slideshows. [View demo](https://shotstack.io/demos/real-estate-video-listing-maker/). diff --git a/examples/bulk-csv-videos/.env.example b/examples/bulk-csv-videos/.env.example new file mode 100644 index 0000000..464cc81 --- /dev/null +++ b/examples/bulk-csv-videos/.env.example @@ -0,0 +1,4 @@ +SHOTSTACK_API_KEY=your_sandbox_api_key +SHOTSTACK_ENV=stage +SHOTSTACK_TEMPLATE_ID=your_stage_template_id +ANTHROPIC_API_KEY=your_anthropic_api_key diff --git a/examples/bulk-csv-videos/.gitignore b/examples/bulk-csv-videos/.gitignore new file mode 100644 index 0000000..bbecc9f --- /dev/null +++ b/examples/bulk-csv-videos/.gitignore @@ -0,0 +1,4 @@ +node_modules/ +.env +products*.csv +batch-results*.json diff --git a/examples/bulk-csv-videos/README.md b/examples/bulk-csv-videos/README.md new file mode 100644 index 0000000..ab39989 --- /dev/null +++ b/examples/bulk-csv-videos/README.md @@ -0,0 +1,100 @@ +# Bulk videos from a CSV + +Render one video per row of a CSV from a single reusable template: create the template with merge +fields, generate and validate a 100-row dataset, submit one template render per row at a safe pace, +and track every render in a resumable manifest. The submit loop is implemented twice, in Node.js and +in Python. An optional AI step has Claude write each row's headline and text-to-image prompt, which +the same pipeline validates like any other input. + +Companion code for [Generate videos in bulk with an API and an AI agent](https://shotstack.io/learn/bulk-create-videos-from-csv-and-ai/). + +## Requirements + +- A [Shotstack account](https://dashboard.shotstack.io/register) with your **sandbox** API key + (dashboard menu under your account name, top right, under **API Keys**) +- Node.js 20 or later, or Python 3 for the submit loop +- Optional, for the AI step: an [Anthropic API key](https://platform.claude.com/) + +Sandbox renders are watermarked, and your account needs at least one credit to use the environment. + +## Setup + +```bash +git clone https://github.com/shotstack/shotstack-cookbook.git +cd shotstack-cookbook/examples/bulk-csv-videos +npm install +``` + +Set your sandbox key and environment: + +```bash +export SHOTSTACK_API_KEY="your_sandbox_api_key" +export SHOTSTACK_ENV="stage" +``` + +## Run + +1. Create the template and keep the returned template id: + +```bash +curl --fail-with-body \ + --request POST \ + "https://api.shotstack.io/edit/stage/templates" \ + --header "Accept: application/json" \ + --header "Content-Type: application/json" \ + --header "x-api-key: ${SHOTSTACK_API_KEY}" \ + --data-binary @template.json +``` + +2. Generate the 100-row dataset: + +```bash +node generate-data.mjs +``` + +3. Preflight three rows, then submit and poll (Node or Python): + +```bash +export SHOTSTACK_TEMPLATE_ID="your_stage_template_id" +SHOTSTACK_ROW_LIMIT=3 node bulk-render.mjs submit +node bulk-render.mjs status +``` + +```bash +SHOTSTACK_ROW_LIMIT=3 python3 bulk_render.py submit +python3 bulk_render.py status +``` + +Remove `SHOTSTACK_ROW_LIMIT` for the full batch. `node bulk-render.mjs summary` (or the Python +equivalent) reads the local manifest without calling any API. + +4. Optional AI step — Claude writes each row's headline and image prompt, the script validates them, + and the render loop runs unchanged on the new file. Use a separate manifest, because the AI-written + rows differ from rows already tracked in `batch-results.json`: + +```bash +export ANTHROPIC_API_KEY="your_anthropic_api_key" +node generate-data-ai.mjs +CSV_PATH=products-ai.csv MANIFEST_PATH=batch-results-ai.json node bulk-render.mjs submit +``` + +If your template has a `text-to-image` asset with an `{{IMAGE_PROMPT}}` placeholder, the AI-written +prompt drives it. Templates without the placeholder ignore the extra field. + +## What happens + +`submit` validates the CSV, queues one template render per row, and records every render in the +manifest. `status` polls each render until it is `done` or `failed` and retrieves the hosted video +URL for finished renders. Each command ends with a status table and the count of hosted videos. The +first three rows finish in under a minute; the full batch takes several minutes at the default +one-request-per-second pace. + +## Notes + +- Templates belong to the environment they were created in. Re-create the template with your + production key and `SHOTSTACK_ENV=v1` before a production run; the ids will differ. +- The manifest (`batch-results.json`) makes reruns safe: rows with a render id are skipped, and + only confirmed failures are retried with `SHOTSTACK_RETRY_FAILED=true`. +- A row marked `unknown` means the process stopped before the API confirmed the request. Check the + render dashboard first. To retry the row, delete its entry from the manifest and run `submit` again. +- Do not run the Node and Python submitters against the same manifest at the same time. diff --git a/examples/bulk-csv-videos/__pycache__/bulk_render.cpython-313.pyc b/examples/bulk-csv-videos/__pycache__/bulk_render.cpython-313.pyc new file mode 100644 index 0000000..05c0864 Binary files /dev/null and b/examples/bulk-csv-videos/__pycache__/bulk_render.cpython-313.pyc differ diff --git a/examples/bulk-csv-videos/bulk-render.mjs b/examples/bulk-csv-videos/bulk-render.mjs new file mode 100644 index 0000000..bf96ae0 --- /dev/null +++ b/examples/bulk-csv-videos/bulk-render.mjs @@ -0,0 +1,546 @@ +import { createHash } from 'node:crypto'; +import { readFile, rename, writeFile } from 'node:fs/promises'; +import { parse } from 'csv-parse/sync'; + +const command = process.argv[2] ?? 'submit'; +const validCommands = new Set(['submit', 'status', 'summary']); + +if (!validCommands.has(command)) { + console.error('Usage: node bulk-render.mjs [submit|status|summary]'); + process.exit(1); +} + +const API_KEY = process.env.SHOTSTACK_API_KEY; +const ENVIRONMENT = process.env.SHOTSTACK_ENV ?? 'stage'; +const TEMPLATE_ID = process.env.SHOTSTACK_TEMPLATE_ID; +const CSV_PATH = process.env.CSV_PATH ?? 'products.csv'; +const MANIFEST_PATH = process.env.MANIFEST_PATH ?? 'batch-results.json'; +const REQUEST_INTERVAL_MS = Number( + process.env.SHOTSTACK_REQUEST_INTERVAL_MS ?? '1000' +); +const ROW_LIMIT = Number(process.env.SHOTSTACK_ROW_LIMIT ?? '0'); +const RETRY_FAILED = process.env.SHOTSTACK_RETRY_FAILED === 'true'; +const MAX_RATE_LIMIT_RETRIES = 3; +const EDIT_BASE = ( + process.env.SHOTSTACK_EDIT_BASE ?? + `https://api.shotstack.io/edit/${ENVIRONMENT}` +).replace(/\/$/, ''); +const SERVE_BASE = ( + process.env.SHOTSTACK_SERVE_BASE ?? + `https://api.shotstack.io/serve/${ENVIRONMENT}` +).replace(/\/$/, ''); + +const fail = message => { + console.error(message); + process.exit(1); +}; + +if (!['stage', 'v1'].includes(ENVIRONMENT)) { + fail('SHOTSTACK_ENV must be stage or v1.'); +} + +if (!Number.isFinite(REQUEST_INTERVAL_MS) || REQUEST_INTERVAL_MS < 0) { + fail('SHOTSTACK_REQUEST_INTERVAL_MS must be zero or greater.'); +} + +if (!Number.isInteger(ROW_LIMIT) || ROW_LIMIT < 0) { + fail('SHOTSTACK_ROW_LIMIT must be a non-negative integer.'); +} + +if (command !== 'summary' && !API_KEY) { + fail('Set SHOTSTACK_API_KEY before running this command.'); +} + +if (command === 'submit' && !TEMPLATE_ID) { + fail('Set SHOTSTACK_TEMPLATE_ID before submitting renders.'); +} + +const sleep = milliseconds => + milliseconds > 0 + ? new Promise(resolve => setTimeout(resolve, milliseconds)) + : Promise.resolve(); + +const hashRow = row => + createHash('sha256') + .update(JSON.stringify(row, Object.keys(row).sort())) + .digest('hex'); + +const responseMessage = body => + body?.response?.error ?? + body?.response?.message ?? + body?.message ?? + JSON.stringify(body); + +async function parseResponse(response) { + const text = await response.text(); + + if (!text) { + return {}; + } + + try { + return JSON.parse(text); + } catch { + return { message: text }; + } +} + +async function saveManifest(manifest) { + manifest.updatedAt = new Date().toISOString(); + const temporaryPath = `${MANIFEST_PATH}.tmp`; + await writeFile(temporaryPath, `${JSON.stringify(manifest, null, 2)}\n`); + await rename(temporaryPath, MANIFEST_PATH); +} + +async function loadManifest({ create = false } = {}) { + try { + const manifest = JSON.parse(await readFile(MANIFEST_PATH, 'utf8')); + + if (manifest.environment !== ENVIRONMENT) { + throw new Error( + `${MANIFEST_PATH} belongs to ${manifest.environment}, not ${ENVIRONMENT}.` + ); + } + + if ( + TEMPLATE_ID && + manifest.templateId && + manifest.templateId !== TEMPLATE_ID + ) { + throw new Error(`${MANIFEST_PATH} belongs to a different template.`); + } + + return manifest; + } catch (error) { + if (error.code !== 'ENOENT' || !create) { + throw error; + } + + return { + version: 1, + environment: ENVIRONMENT, + templateId: TEMPLATE_ID, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + rows: [] + }; + } +} + +async function loadRows() { + let csvText; + + try { + csvText = await readFile(CSV_PATH, 'utf8'); + } catch (error) { + if (error.code === 'ENOENT') { + throw new Error( + `Cannot find ${CSV_PATH}. Run node generate-data.mjs first, or set CSV_PATH.` + ); + } + throw error; + } + + const rows = parse(csvText, { + bom: true, + columns: true, + skip_empty_lines: true, + trim: true + }); + + const requiredColumns = [ + 'row_id', + 'product_name', + 'headline', + 'price', + 'image_url', + 'brand_color' + ]; + const seenIds = new Set(); + const errors = []; + + rows.forEach((row, index) => { + const line = index + 2; + + for (const column of requiredColumns) { + if (!row[column]) { + errors.push(`Line ${line}: ${column} is required.`); + } + } + + if (!/^[A-Za-z0-9_-]+$/.test(row.row_id ?? '')) { + errors.push( + `Line ${line}: row_id may contain only letters, numbers, _ and -.` + ); + } + + if (seenIds.has(row.row_id)) { + errors.push(`Line ${line}: duplicate row_id ${row.row_id}.`); + } + seenIds.add(row.row_id); + + if ((row.product_name ?? '').length > 40) { + errors.push(`Line ${line}: product_name must be 40 characters or fewer.`); + } + + if ((row.headline ?? '').length > 55) { + errors.push(`Line ${line}: headline must be 55 characters or fewer.`); + } + + if ((row.price ?? '').length > 20) { + errors.push(`Line ${line}: price must be 20 characters or fewer.`); + } + + try { + const imageUrl = new URL(row.image_url); + if (imageUrl.protocol !== 'https:') { + throw new Error('not HTTPS'); + } + } catch { + errors.push(`Line ${line}: image_url must be a valid HTTPS URL.`); + } + + if (!/^#[0-9A-Fa-f]{6}$/.test(row.brand_color ?? '')) { + errors.push(`Line ${line}: brand_color must be a six-digit hex color.`); + } + }); + + if (errors.length > 0) { + throw new Error(`CSV validation failed:\n${errors.join('\n')}`); + } + + return ROW_LIMIT > 0 ? rows.slice(0, ROW_LIMIT) : rows; +} + +function mergeFields(row) { + const fields = [ + { find: 'PRODUCT_NAME', replace: row.product_name }, + { find: 'HEADLINE', replace: row.headline }, + { find: 'PRICE', replace: row.price }, + { find: 'IMAGE_URL', replace: row.image_url }, + { find: 'BRAND_COLOR', replace: row.brand_color } + ]; + + // The AI data step adds an image_prompt column for templates that use a + // text-to-image asset with an {{IMAGE_PROMPT}} placeholder. Templates + // without the placeholder ignore the extra merge field. + if (row.image_prompt) { + fields.push({ find: 'IMAGE_PROMPT', replace: row.image_prompt }); + } + + return fields; +} + +function retryDelay(response, retryNumber) { + const retryAfter = response.headers.get('retry-after'); + + if (retryAfter && /^\d+$/.test(retryAfter)) { + return Number(retryAfter) * 1000; + } + + if (retryAfter) { + const dateDelay = Date.parse(retryAfter) - Date.now(); + if (Number.isFinite(dateDelay) && dateDelay > 0) { + return dateDelay; + } + } + + return 60_000 * 2 ** retryNumber; +} + +async function submitTemplate(row) { + const payload = { + id: TEMPLATE_ID, + merge: mergeFields(row) + }; + + for (let retry = 0; retry <= MAX_RATE_LIMIT_RETRIES; retry += 1) { + let response; + + try { + response = await fetch(`${EDIT_BASE}/templates/render`, { + method: 'POST', + headers: { + Accept: 'application/json', + 'Content-Type': 'application/json', + 'x-api-key': API_KEY + }, + body: JSON.stringify(payload), + signal: AbortSignal.timeout(30_000) + }); + } catch (error) { + return { + kind: 'unknown', + error: `No definitive API response: ${error.message}` + }; + } + + const body = await parseResponse(response); + + if (response.status === 429) { + if (retry === MAX_RATE_LIMIT_RETRIES) { + return { + kind: 'rejected', + statusCode: 429, + error: responseMessage(body) + }; + } + + const delay = retryDelay(response, retry); + console.warn(`Rate limited. Waiting ${Math.ceil(delay / 1000)} seconds.`); + await sleep(delay); + continue; + } + + if (response.status === 201 && body?.response?.id) { + return { kind: 'accepted', renderId: body.response.id }; + } + + if (response.status >= 400 && response.status < 500) { + return { + kind: 'rejected', + statusCode: response.status, + error: responseMessage(body) + }; + } + + return { + kind: 'unknown', + statusCode: response.status, + error: `Unexpected response: ${responseMessage(body)}` + }; + } +} + +async function submitRows() { + const rows = await loadRows(); + const manifest = await loadManifest({ create: true }); + let failures = 0; + + console.log(`Validated ${rows.length} rows from ${CSV_PATH}.`); + + for (const [index, row] of rows.entries()) { + let entry = manifest.rows.find(item => item.rowId === row.row_id); + + if (entry?.status === 'submitting') { + entry.status = 'unknown'; + entry.error = + 'The previous process stopped during submission. Check the dashboard before retrying.'; + await saveManifest(manifest); + } + + if (entry?.status === 'unknown') { + console.warn(`[${row.row_id}] skipped: previous outcome is unknown.`); + continue; + } + + const canRetry = + RETRY_FAILED && + (entry?.status === 'failed' || entry?.status === 'submission_failed'); + const currentHash = hashRow(row); + + if (entry?.inputHash && entry.inputHash !== currentHash && !canRetry) { + throw new Error( + `Row ${row.row_id} changed after its first submission. ` + + 'Use a new row_id, or retry it only after confirming the previous request failed.' + ); + } + + if (entry?.renderId && !canRetry) { + console.log(`[${row.row_id}] skipped: already has a render ID.`); + continue; + } + + if (entry?.status === 'submission_failed' && !canRetry) { + console.log(`[${row.row_id}] skipped: set SHOTSTACK_RETRY_FAILED=true.`); + continue; + } + + if (!entry) { + entry = { rowId: row.row_id, attempts: 0 }; + manifest.rows.push(entry); + } + + if (canRetry) { + if (entry.renderId) { + entry.previousRenderIds = [ + ...(entry.previousRenderIds ?? []), + entry.renderId + ]; + } + delete entry.renderId; + delete entry.temporaryUrl; + delete entry.hostedUrl; + delete entry.hostingStatus; + delete entry.statusUpdatedAt; + delete entry.completedAt; + delete entry.submittedAt; + delete entry.statusCode; + } + + entry.inputHash = currentHash; + entry.status = 'submitting'; + entry.error = null; + entry.attempts += 1; + entry.lastAttemptAt = new Date().toISOString(); + await saveManifest(manifest); + + const result = await submitTemplate(row); + + if (result.kind === 'accepted') { + entry.renderId = result.renderId; + entry.status = 'queued'; + entry.submittedAt = new Date().toISOString(); + console.log( + `[${index + 1}/${rows.length}] ${row.row_id} -> ${result.renderId}` + ); + } else if (result.kind === 'rejected') { + entry.status = 'submission_failed'; + entry.statusCode = result.statusCode; + entry.error = result.error; + failures += 1; + console.error(`[${row.row_id}] rejected: ${result.error}`); + } else { + entry.status = 'unknown'; + entry.statusCode = result.statusCode; + entry.error = result.error; + failures += 1; + console.error(`[${row.row_id}] unknown outcome: ${result.error}`); + } + + await saveManifest(manifest); + + if (result.kind === 'rejected' && [401, 403].includes(result.statusCode)) { + console.error( + 'The API key was rejected. Check SHOTSTACK_API_KEY and SHOTSTACK_ENV, then run submit again.' + ); + break; + } + + await sleep(REQUEST_INTERVAL_MS); + } + + printSummary(manifest); + + if (failures > 0) { + process.exitCode = 1; + } +} + +async function getJson(url) { + const response = await fetch(url, { + headers: { + Accept: 'application/json', + 'x-api-key': API_KEY + }, + signal: AbortSignal.timeout(30_000) + }); + const body = await parseResponse(response); + return { response, body }; +} + +async function updateStatuses() { + const manifest = await loadManifest(); + + for (const entry of manifest.rows) { + if ( + !entry.renderId || + entry.status === 'failed' || + (entry.status === 'done' && entry.hostedUrl) + ) { + continue; + } + + if (entry.status !== 'done') { + try { + const { response, body } = await getJson( + `${EDIT_BASE}/render/${entry.renderId}?data=false` + ); + + if (response.ok && body?.response?.status) { + entry.status = body.response.status; + entry.error = body.response.error || null; + entry.temporaryUrl = body.response.url || null; + entry.statusUpdatedAt = body.response.updated || null; + + if (entry.status === 'done' || entry.status === 'failed') { + entry.completedAt = body.response.updated || null; + } + } else { + console.warn( + `[${entry.rowId}] status lookup failed: ${response.status} ${responseMessage(body)}` + ); + } + } catch (error) { + console.warn(`[${entry.rowId}] status lookup failed: ${error.message}`); + } + } + + if (entry.status === 'done' && !entry.hostedUrl) { + try { + const { response, body } = await getJson( + `${SERVE_BASE}/assets/render/${entry.renderId}` + ); + + if (response.ok && Array.isArray(body.data)) { + const video = body.data.find( + asset => + asset?.attributes?.status === 'ready' && + asset?.attributes?.filename?.endsWith('.mp4') + ); + const firstAsset = body.data[0]?.attributes; + + entry.hostingStatus = video + ? 'ready' + : (firstAsset?.status ?? 'pending'); + entry.hostedUrl = video?.attributes?.url ?? null; + } else { + entry.hostingStatus = 'pending'; + } + } catch { + entry.hostingStatus = 'pending'; + } + } + + await saveManifest(manifest); + await sleep(REQUEST_INTERVAL_MS); + } + + printSummary(manifest); +} + +function printSummary(manifest) { + const counts = {}; + + for (const entry of manifest.rows) { + counts[entry.status] = (counts[entry.status] ?? 0) + 1; + } + + console.table( + Object.entries(counts) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([status, count]) => ({ status, count })) + ); + console.log( + `Hosted videos ready: ${manifest.rows.filter(row => row.hostedUrl).length}/${manifest.rows.length}` + ); +} + +try { + if (command === 'submit') { + await submitRows(); + } else if (command === 'status') { + await updateStatuses(); + } else { + printSummary(await loadManifest()); + } +} catch (error) { + if (error.code === 'ENOENT') { + console.error( + `Cannot find ${MANIFEST_PATH}. Run node bulk-render.mjs submit first.` + ); + } else { + console.error(error instanceof Error ? error.message : error); + } + process.exitCode = 1; +} diff --git a/examples/bulk-csv-videos/bulk_render.py b/examples/bulk-csv-videos/bulk_render.py new file mode 100644 index 0000000..fd25094 --- /dev/null +++ b/examples/bulk-csv-videos/bulk_render.py @@ -0,0 +1,542 @@ +#!/usr/bin/env python3 + +import csv +import hashlib +import json +import os +import re +import sys +import time +from datetime import datetime, timezone +from email.utils import parsedate_to_datetime +from pathlib import Path +from urllib.error import HTTPError, URLError +from urllib.parse import urlparse +from urllib.request import Request, urlopen + +COMMAND = sys.argv[1] if len(sys.argv) > 1 else "submit" +VALID_COMMANDS = {"submit", "status", "summary"} + +if COMMAND not in VALID_COMMANDS: + raise SystemExit("Usage: python3 bulk_render.py [submit|status|summary]") + +API_KEY = os.environ.get("SHOTSTACK_API_KEY") +ENVIRONMENT = os.environ.get("SHOTSTACK_ENV", "stage") +TEMPLATE_ID = os.environ.get("SHOTSTACK_TEMPLATE_ID") +CSV_PATH = Path(os.environ.get("CSV_PATH", "products.csv")) +MANIFEST_PATH = Path(os.environ.get("MANIFEST_PATH", "batch-results.json")) +REQUEST_INTERVAL_MS = int(os.environ.get("SHOTSTACK_REQUEST_INTERVAL_MS", "1000")) +ROW_LIMIT = int(os.environ.get("SHOTSTACK_ROW_LIMIT", "0")) +RETRY_FAILED = os.environ.get("SHOTSTACK_RETRY_FAILED") == "true" +MAX_RATE_LIMIT_RETRIES = 3 +EDIT_BASE = os.environ.get( + "SHOTSTACK_EDIT_BASE", f"https://api.shotstack.io/edit/{ENVIRONMENT}" +).rstrip("/") +SERVE_BASE = os.environ.get( + "SHOTSTACK_SERVE_BASE", f"https://api.shotstack.io/serve/{ENVIRONMENT}" +).rstrip("/") + +if ENVIRONMENT not in {"stage", "v1"}: + raise SystemExit("SHOTSTACK_ENV must be stage or v1.") + +if REQUEST_INTERVAL_MS < 0: + raise SystemExit("SHOTSTACK_REQUEST_INTERVAL_MS must be zero or greater.") + +if ROW_LIMIT < 0: + raise SystemExit("SHOTSTACK_ROW_LIMIT must be a non-negative integer.") + +if COMMAND != "summary" and not API_KEY: + raise SystemExit("Set SHOTSTACK_API_KEY before running this command.") + +if COMMAND == "submit" and not TEMPLATE_ID: + raise SystemExit("Set SHOTSTACK_TEMPLATE_ID before submitting renders.") + + +def sleep_between_requests(): + if REQUEST_INTERVAL_MS > 0: + time.sleep(REQUEST_INTERVAL_MS / 1000) + + +def row_hash(row): + payload = json.dumps( + row, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + ).encode("utf-8") + return hashlib.sha256(payload).hexdigest() + + +def response_message(body): + if not isinstance(body, dict): + return str(body) + response = body.get("response") + if isinstance(response, dict): + return response.get("error") or response.get("message") or str(response) + return body.get("message") or str(body) + + +def save_manifest(manifest): + manifest["updatedAt"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + temporary_path = MANIFEST_PATH.with_name(f"{MANIFEST_PATH.name}.tmp") + temporary_path.write_text( + json.dumps(manifest, indent=2) + "\n", encoding="utf-8" + ) + os.replace(temporary_path, MANIFEST_PATH) + + +def load_manifest(create=False): + if MANIFEST_PATH.exists(): + manifest = json.loads(MANIFEST_PATH.read_text(encoding="utf-8")) + + if manifest.get("environment") != ENVIRONMENT: + raise RuntimeError( + f"{MANIFEST_PATH} belongs to {manifest.get('environment')}, " + f"not {ENVIRONMENT}." + ) + + if ( + TEMPLATE_ID + and manifest.get("templateId") + and manifest["templateId"] != TEMPLATE_ID + ): + raise RuntimeError(f"{MANIFEST_PATH} belongs to a different template.") + + return manifest + + if not create: + raise FileNotFoundError(MANIFEST_PATH) + + now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + return { + "version": 1, + "environment": ENVIRONMENT, + "templateId": TEMPLATE_ID, + "createdAt": now, + "updatedAt": now, + "rows": [], + } + + +def load_rows(): + if not CSV_PATH.exists(): + raise FileNotFoundError( + f"Cannot find {CSV_PATH}. Run node generate-data.mjs first, " + "or set CSV_PATH." + ) + + with CSV_PATH.open(newline="", encoding="utf-8-sig") as csv_file: + rows = list(csv.DictReader(csv_file)) + + required_columns = [ + "row_id", + "product_name", + "headline", + "price", + "image_url", + "brand_color", + ] + seen_ids = set() + errors = [] + + for index, row in enumerate(rows, start=2): + for column in required_columns: + if not row.get(column): + errors.append(f"Line {index}: {column} is required.") + + row_id = row.get("row_id", "") + if not re.fullmatch(r"[A-Za-z0-9_-]+", row_id): + errors.append( + f"Line {index}: row_id may contain only letters, numbers, _ and -." + ) + + if row_id in seen_ids: + errors.append(f"Line {index}: duplicate row_id {row_id}.") + seen_ids.add(row_id) + + if len(row.get("product_name", "")) > 40: + errors.append( + f"Line {index}: product_name must be 40 characters or fewer." + ) + + if len(row.get("headline", "")) > 55: + errors.append(f"Line {index}: headline must be 55 characters or fewer.") + + if len(row.get("price", "")) > 20: + errors.append(f"Line {index}: price must be 20 characters or fewer.") + + image_url = urlparse(row.get("image_url", "")) + if image_url.scheme != "https" or not image_url.netloc: + errors.append(f"Line {index}: image_url must be a valid HTTPS URL.") + + if not re.fullmatch(r"#[0-9A-Fa-f]{6}", row.get("brand_color", "")): + errors.append( + f"Line {index}: brand_color must be a six-digit hex color." + ) + + if errors: + raise ValueError("CSV validation failed:\n" + "\n".join(errors)) + + return rows[:ROW_LIMIT] if ROW_LIMIT > 0 else rows + + +def merge_fields(row): + fields = [ + {"find": "PRODUCT_NAME", "replace": row["product_name"]}, + {"find": "HEADLINE", "replace": row["headline"]}, + {"find": "PRICE", "replace": row["price"]}, + {"find": "IMAGE_URL", "replace": row["image_url"]}, + {"find": "BRAND_COLOR", "replace": row["brand_color"]}, + ] + + # The AI data step adds an image_prompt column for templates that use a + # text-to-image asset with an {{IMAGE_PROMPT}} placeholder. Templates + # without the placeholder ignore the extra merge field. + if row.get("image_prompt"): + fields.append({"find": "IMAGE_PROMPT", "replace": row["image_prompt"]}) + + return fields + + +def parse_json_body(text): + if not text: + return {} + try: + return json.loads(text) + except json.JSONDecodeError: + return {"message": text} + + +def request_json(method, url, payload=None): + data = json.dumps(payload).encode() if payload is not None else None + headers = { + "Accept": "application/json", + "x-api-key": API_KEY, + } + if data is not None: + headers["Content-Type"] = "application/json" + + request = Request(url, data=data, headers=headers, method=method) + + try: + with urlopen(request, timeout=30) as response: + body = parse_json_body(response.read().decode()) + return response.status, body, response.headers + except HTTPError as error: + body = parse_json_body(error.read().decode()) + return error.code, body, error.headers + + +def retry_delay(headers, retry_number): + retry_after = headers.get("Retry-After") if headers else None + + if retry_after and retry_after.isdigit(): + return int(retry_after) + + if retry_after: + try: + date_delay = ( + parsedate_to_datetime(retry_after) - datetime.now(timezone.utc) + ).total_seconds() + if date_delay > 0: + return date_delay + except (TypeError, ValueError): + pass + + return 60 * (2**retry_number) + + +def submit_template(row): + payload = {"id": TEMPLATE_ID, "merge": merge_fields(row)} + + for retry in range(MAX_RATE_LIMIT_RETRIES + 1): + try: + status_code, body, headers = request_json( + "POST", f"{EDIT_BASE}/templates/render", payload + ) + except (URLError, TimeoutError, OSError) as error: + return { + "kind": "unknown", + "error": f"No definitive API response: {error}", + } + + if status_code == 429: + if retry == MAX_RATE_LIMIT_RETRIES: + return { + "kind": "rejected", + "statusCode": 429, + "error": response_message(body), + } + delay = retry_delay(headers, retry) + print(f"Rate limited. Waiting {delay} seconds.", file=sys.stderr) + time.sleep(delay) + continue + + render_id = ( + body.get("response", {}).get("id") if isinstance(body, dict) else None + ) + if status_code == 201 and render_id: + return {"kind": "accepted", "renderId": render_id} + + if 400 <= status_code < 500: + return { + "kind": "rejected", + "statusCode": status_code, + "error": response_message(body), + } + + return { + "kind": "unknown", + "statusCode": status_code, + "error": f"Unexpected response: {response_message(body)}", + } + + raise RuntimeError("Unreachable") + + +def submit_rows(): + rows = load_rows() + manifest = load_manifest(create=True) + failures = 0 + print(f"Validated {len(rows)} rows from {CSV_PATH}.") + + for index, row in enumerate(rows, start=1): + entry = next( + (item for item in manifest["rows"] if item["rowId"] == row["row_id"]), + None, + ) + + if entry and entry.get("status") == "submitting": + entry["status"] = "unknown" + entry["error"] = ( + "The previous process stopped during submission. " + "Check the dashboard before retrying." + ) + save_manifest(manifest) + + if entry and entry.get("status") == "unknown": + print( + f"[{row['row_id']}] skipped: previous outcome is unknown.", + file=sys.stderr, + ) + continue + + can_retry = RETRY_FAILED and entry and entry.get("status") in { + "failed", + "submission_failed", + } + current_hash = row_hash(row) + + if ( + entry + and entry.get("inputHash") + and entry["inputHash"] != current_hash + and not can_retry + ): + raise RuntimeError( + f"Row {row['row_id']} changed after its first submission. " + "Use a new row_id, or retry it only after confirming the " + "previous request failed." + ) + + if entry and entry.get("renderId") and not can_retry: + print(f"[{row['row_id']}] skipped: already has a render ID.") + continue + + if ( + entry + and entry.get("status") == "submission_failed" + and not can_retry + ): + print( + f"[{row['row_id']}] skipped: set SHOTSTACK_RETRY_FAILED=true." + ) + continue + + if entry is None: + entry = {"rowId": row["row_id"], "attempts": 0} + manifest["rows"].append(entry) + + if can_retry: + if entry.get("renderId"): + entry["previousRenderIds"] = [ + *entry.get("previousRenderIds", []), + entry["renderId"], + ] + entry.pop("renderId", None) + entry.pop("temporaryUrl", None) + entry.pop("hostedUrl", None) + entry.pop("hostingStatus", None) + entry.pop("statusUpdatedAt", None) + entry.pop("completedAt", None) + entry.pop("submittedAt", None) + entry.pop("statusCode", None) + + entry["inputHash"] = current_hash + entry["status"] = "submitting" + entry["error"] = None + entry["attempts"] = entry.get("attempts", 0) + 1 + entry["lastAttemptAt"] = time.strftime( + "%Y-%m-%dT%H:%M:%SZ", time.gmtime() + ) + save_manifest(manifest) + + result = submit_template(row) + + if result["kind"] == "accepted": + entry["renderId"] = result["renderId"] + entry["status"] = "queued" + entry["submittedAt"] = time.strftime( + "%Y-%m-%dT%H:%M:%SZ", time.gmtime() + ) + print( + f"[{index}/{len(rows)}] {row['row_id']} -> {result['renderId']}" + ) + elif result["kind"] == "rejected": + entry["status"] = "submission_failed" + entry["statusCode"] = result.get("statusCode") + entry["error"] = result["error"] + failures += 1 + print( + f"[{row['row_id']}] rejected: {result['error']}", + file=sys.stderr, + ) + else: + entry["status"] = "unknown" + entry["statusCode"] = result.get("statusCode") + entry["error"] = result["error"] + failures += 1 + print( + f"[{row['row_id']}] unknown outcome: {result['error']}", + file=sys.stderr, + ) + + save_manifest(manifest) + + if result["kind"] == "rejected" and result.get("statusCode") in {401, 403}: + print( + "The API key was rejected. Check SHOTSTACK_API_KEY and " + "SHOTSTACK_ENV, then run submit again.", + file=sys.stderr, + ) + break + + sleep_between_requests() + + print_summary(manifest) + + if failures > 0: + raise SystemExit(1) + + +def update_statuses(): + manifest = load_manifest() + + for entry in manifest["rows"]: + if ( + not entry.get("renderId") + or entry.get("status") == "failed" + or (entry.get("status") == "done" and entry.get("hostedUrl")) + ): + continue + + if entry.get("status") != "done": + try: + status_code, body, _ = request_json( + "GET", + f"{EDIT_BASE}/render/{entry['renderId']}?data=false", + ) + response = body.get("response", {}) if isinstance(body, dict) else {} + + if 200 <= status_code < 300 and response.get("status"): + entry["status"] = response["status"] + entry["error"] = response.get("error") or None + entry["temporaryUrl"] = response.get("url") + entry["statusUpdatedAt"] = response.get("updated") + + if entry["status"] in {"done", "failed"}: + entry["completedAt"] = response.get("updated") + else: + print( + f"[{entry['rowId']}] status lookup failed: " + f"{status_code} {response_message(body)}", + file=sys.stderr, + ) + except (URLError, TimeoutError, OSError) as error: + print( + f"[{entry['rowId']}] status lookup failed: {error}", + file=sys.stderr, + ) + + if entry.get("status") == "done" and not entry.get("hostedUrl"): + try: + status_code, body, _ = request_json( + "GET", + f"{SERVE_BASE}/assets/render/{entry['renderId']}", + ) + assets = body.get("data", []) if isinstance(body, dict) else [] + + if 200 <= status_code < 300 and isinstance(assets, list): + video = next( + ( + asset.get("attributes", {}) + for asset in assets + if asset.get("attributes", {}).get("status") == "ready" + and asset.get("attributes", {}) + .get("filename", "") + .endswith(".mp4") + ), + None, + ) + first_asset = ( + assets[0].get("attributes", {}) if assets else {} + ) + entry["hostingStatus"] = ( + "ready" if video else first_asset.get("status", "pending") + ) + entry["hostedUrl"] = video.get("url") if video else None + else: + entry["hostingStatus"] = "pending" + except (URLError, TimeoutError, OSError): + entry["hostingStatus"] = "pending" + + save_manifest(manifest) + sleep_between_requests() + + print_summary(manifest) + + +def print_summary(manifest): + counts = {} + for entry in manifest["rows"]: + status = entry.get("status", "unknown") + counts[status] = counts.get(status, 0) + 1 + + print("status\tcount") + for status in sorted(counts): + print(f"{status}\t{counts[status]}") + + hosted = sum(bool(row.get("hostedUrl")) for row in manifest["rows"]) + print(f"Hosted videos ready: {hosted}/{len(manifest['rows'])}") + + +try: + if COMMAND == "submit": + submit_rows() + elif COMMAND == "status": + update_statuses() + else: + print_summary(load_manifest()) +except FileNotFoundError as error: + if str(error) == str(MANIFEST_PATH): + print( + f"Cannot find {MANIFEST_PATH}. Run python3 bulk_render.py submit " + "first.", + file=sys.stderr, + ) + else: + print(error, file=sys.stderr) + raise SystemExit(1) from error +except (OSError, ValueError, RuntimeError) as error: + print(error, file=sys.stderr) + raise SystemExit(1) from error diff --git a/examples/bulk-csv-videos/generate-data-ai.mjs b/examples/bulk-csv-videos/generate-data-ai.mjs new file mode 100644 index 0000000..c514ea9 --- /dev/null +++ b/examples/bulk-csv-videos/generate-data-ai.mjs @@ -0,0 +1,173 @@ +import { readFile, writeFile } from 'node:fs/promises'; +import { parse } from 'csv-parse/sync'; + +const API_KEY = process.env.ANTHROPIC_API_KEY; +const MODEL = process.env.ANTHROPIC_MODEL ?? 'claude-opus-5'; +const CSV_IN = process.env.CSV_PATH ?? 'products.csv'; +const CSV_OUT = process.env.CSV_AI_PATH ?? 'products-ai.csv'; +const MAX_HEADLINE_LENGTH = 55; +const MAX_PROMPT_LENGTH = 300; + +const fail = message => { + console.error(message); + process.exit(1); +}; + +if (!API_KEY) { + fail('Set ANTHROPIC_API_KEY before running this script.'); +} + +let csvText; + +try { + csvText = await readFile(CSV_IN, 'utf8'); +} catch (error) { + if (error.code === 'ENOENT') { + fail( + `Cannot find ${CSV_IN}. Run node generate-data.mjs first, or set CSV_PATH.` + ); + } + throw error; +} + +const rows = parse(csvText, { + bom: true, + columns: true, + skip_empty_lines: true, + trim: true +}); + +const schema = { + type: 'object', + properties: { + headlines: { + type: 'array', + items: { + type: 'object', + properties: { + row_id: { type: 'string' }, + headline: { + type: 'string', + description: `Marketing headline, ${MAX_HEADLINE_LENGTH} characters or fewer` + }, + image_prompt: { + type: 'string', + description: `Text-to-image prompt for a product background, ${MAX_PROMPT_LENGTH} characters or fewer` + } + }, + required: ['row_id', 'headline', 'image_prompt'], + additionalProperties: false + } + } + }, + required: ['headlines'], + additionalProperties: false +}; + +const products = rows.map(({ row_id, product_name, price }) => ({ + row_id, + product_name, + price +})); + +let response; + +try { + response = await fetch('https://api.anthropic.com/v1/messages', { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'x-api-key': API_KEY, + 'anthropic-version': '2023-06-01' + }, + body: JSON.stringify({ + model: MODEL, + max_tokens: 16000, + output_config: { + effort: 'low', + format: { type: 'json_schema', schema } + }, + messages: [ + { + role: 'user', + content: [ + 'For each product below, write one short marketing headline and one text-to-image prompt.', + `Every headline must be ${MAX_HEADLINE_LENGTH} characters or fewer, plain text, no quotes or emoji.`, + `Every image prompt must be ${MAX_PROMPT_LENGTH} characters or fewer and describe a clean product background photo.`, + 'Return exactly one entry per row_id.', + '', + JSON.stringify(products) + ].join('\n') + } + ] + }), + signal: AbortSignal.timeout(120_000) + }); +} catch (error) { + fail(`Could not reach the Anthropic API: ${error.message}`); +} + +if (!response.ok) { + fail(`Anthropic API error ${response.status}: ${await response.text()}`); +} + +const body = await response.json(); + +if (body.stop_reason === 'refusal') { + fail('The model declined the request; keep the original headlines.'); +} +if (body.stop_reason === 'max_tokens') { + fail('The response was truncated. Raise max_tokens or send fewer rows.'); +} + +const text = body.content.find(block => block.type === 'text')?.text ?? '{}'; +const generated = new Map( + (JSON.parse(text).headlines ?? []).map(item => [item.row_id, item]) +); + +// Validate the model's output with the same rules as any other input. +let headlines = 0; +let prompts = 0; +for (const row of rows) { + const item = generated.get(row.row_id); + const headline = item?.headline?.trim(); + const imagePrompt = item?.image_prompt?.trim(); + + if (headline && headline.length <= MAX_HEADLINE_LENGTH) { + row.headline = headline; + headlines += 1; + } + + if (imagePrompt && imagePrompt.length <= MAX_PROMPT_LENGTH) { + row.image_prompt = imagePrompt; + prompts += 1; + } else { + // Deterministic fallback so the text-to-image asset always has a prompt. + row.image_prompt = `Studio product photo of ${row.product_name} on a plain background`; + } +} + +const columns = [ + 'row_id', + 'product_name', + 'headline', + 'price', + 'image_url', + 'brand_color', + 'image_prompt' +]; +const csvEscape = value => { + const textValue = String(value); + return /[",\n]/.test(textValue) + ? `"${textValue.replaceAll('"', '""')}"` + : textValue; +}; +const csv = [ + columns.join(','), + ...rows.map(row => columns.map(column => csvEscape(row[column])).join(',')) +].join('\n'); + +await writeFile(CSV_OUT, `${csv}\n`, 'utf8'); +console.log( + `Wrote ${CSV_OUT}: ${headlines}/${rows.length} headlines and ${prompts}/${rows.length} image prompts AI-generated.` +); diff --git a/examples/bulk-csv-videos/generate-data.mjs b/examples/bulk-csv-videos/generate-data.mjs new file mode 100644 index 0000000..4ced0e0 --- /dev/null +++ b/examples/bulk-csv-videos/generate-data.mjs @@ -0,0 +1,87 @@ +import { writeFile } from 'node:fs/promises'; + +const imageUrls = [ + 'https://shotstack-assets.s3.amazonaws.com/images/slideshow1.jpeg', + 'https://shotstack-assets.s3.amazonaws.com/images/slideshow2.jpeg', + 'https://shotstack-assets.s3.amazonaws.com/images/slideshow3.jpeg', + 'https://shotstack-assets.s3.amazonaws.com/images/slideshow4.jpeg', + 'https://shotstack-assets.s3.amazonaws.com/images/slideshow5.jpeg', + 'https://shotstack-assets.s3.amazonaws.com/images/slideshow6.jpeg', + 'https://shotstack-assets.s3.amazonaws.com/images/slideshow7.jpeg' +]; + +const headlines = [ + 'New this week', + 'Made for everyday use', + 'A customer favorite', + 'Limited release', + 'Built to last' +]; + +const brandColors = ['#0f766e', '#1d4ed8', '#7c3aed', '#be123c', '#b45309']; + +const preflightRows = [ + { + row_id: 'product-001', + product_name: 'Limited Edition Travel Backpack - XL Pro', + headline: 'Longest approved headline checks wrapping before launch', + price: 'From $199', + image_url: imageUrls[0], + brand_color: brandColors[0] + }, + { + row_id: 'product-002', + product_name: 'Mug', + headline: 'New', + price: '$9.00', + image_url: imageUrls[1], + brand_color: brandColors[1] + }, + { + row_id: 'product-003', + product_name: 'Café "Voyager", Édition', + headline: 'Built for Nairobi, Montréal, and everywhere between', + price: 'From $49', + image_url: imageUrls[2], + brand_color: brandColors[2] + } +]; + +const csvEscape = value => { + const text = String(value); + return /[",\n]/.test(text) ? `"${text.replaceAll('"', '""')}"` : text; +}; + +const rows = Array.from({ length: 100 }, (_, index) => { + const number = index + 1; + + if (index < preflightRows.length) { + return preflightRows[index]; + } + + return { + row_id: `product-${String(number).padStart(3, '0')}`, + product_name: `Demo Product ${String(number).padStart(3, '0')}`, + headline: headlines[index % headlines.length], + price: `$${29 + ((number * 7) % 170)}.00`, + image_url: imageUrls[index % imageUrls.length], + brand_color: brandColors[index % brandColors.length] + }; +}); + +const columns = [ + 'row_id', + 'product_name', + 'headline', + 'price', + 'image_url', + 'brand_color' +]; + +const csv = [ + columns.join(','), + ...rows.map(row => columns.map(column => csvEscape(row[column])).join(',')) +].join('\n'); + +await writeFile('products.csv', `${csv}\n`, 'utf8'); +console.log(`Created products.csv with ${rows.length} rows.`); diff --git a/examples/bulk-csv-videos/package-lock.json b/examples/bulk-csv-videos/package-lock.json new file mode 100644 index 0000000..bf651e8 --- /dev/null +++ b/examples/bulk-csv-videos/package-lock.json @@ -0,0 +1,24 @@ +{ + "name": "bulk-csv-videos", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "bulk-csv-videos", + "version": "1.0.0", + "dependencies": { + "csv-parse": "^7.0.2" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/csv-parse": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-7.0.2.tgz", + "integrity": "sha512-uKZghv9UmPkMVLYy//KZ9HFAIJsl7wkhoEdIL0+rhuSY9pZQlhaeGEDPIe+/w7eh81MOql8Q/9+inAGWG6ZHYA==", + "license": "MIT" + } + } +} diff --git a/examples/bulk-csv-videos/package.json b/examples/bulk-csv-videos/package.json new file mode 100644 index 0000000..8a64548 --- /dev/null +++ b/examples/bulk-csv-videos/package.json @@ -0,0 +1,16 @@ +{ + "name": "bulk-csv-videos", + "version": "1.0.0", + "private": true, + "description": "Render one video per CSV row from a single Shotstack template, with an optional AI data step", + "type": "module", + "scripts": { + "format": "npx prettier@3 --write ." + }, + "engines": { + "node": ">=20" + }, + "dependencies": { + "csv-parse": "^7.0.2" + } +} diff --git a/examples/bulk-csv-videos/template.json b/examples/bulk-csv-videos/template.json new file mode 100644 index 0000000..da06ac1 --- /dev/null +++ b/examples/bulk-csv-videos/template.json @@ -0,0 +1,138 @@ +{ + "name": "Bulk catalog promo - 3 seconds", + "template": { + "timeline": { + "background": "{{BRAND_COLOR}}", + "tracks": [ + { + "clips": [ + { + "asset": { + "type": "rich-text", + "text": "{{PRICE}}", + "font": { + "family": "Montserrat", + "size": 52, + "weight": 800, + "color": "#111827" + }, + "background": { + "color": "#ffffff", + "opacity": 1, + "borderRadius": 24 + }, + "padding": 20, + "align": { + "horizontal": "center", + "vertical": "middle" + } + }, + "start": 0, + "length": 3, + "width": 320, + "height": 120, + "position": "bottom", + "offset": { + "y": 0.05 + }, + "transition": { + "in": "fade" + } + } + ] + }, + { + "clips": [ + { + "asset": { + "type": "rich-text", + "text": "{{HEADLINE}}", + "font": { + "family": "Montserrat", + "size": 38, + "weight": 600, + "color": "#ffffff" + }, + "align": { + "horizontal": "center", + "vertical": "middle" + } + }, + "start": 0, + "length": 3, + "width": 620, + "height": 120, + "position": "bottom", + "offset": { + "y": 0.2 + }, + "transition": { + "in": "fade" + } + } + ] + }, + { + "clips": [ + { + "asset": { + "type": "rich-text", + "text": "{{PRODUCT_NAME}}", + "font": { + "family": "Montserrat", + "size": 58, + "weight": 800, + "color": "#ffffff" + }, + "align": { + "horizontal": "center", + "vertical": "middle" + } + }, + "start": 0, + "length": 3, + "width": 620, + "height": 180, + "position": "top", + "offset": { + "y": -0.08 + }, + "transition": { + "in": "fade" + } + } + ] + }, + { + "clips": [ + { + "asset": { + "type": "image", + "src": "{{IMAGE_URL}}" + }, + "start": 0, + "length": 3, + "width": 560, + "height": 560, + "fit": "crop", + "position": "center", + "offset": { + "y": 0.04 + }, + "effect": "zoomInSlow", + "transition": { + "in": "fade" + } + } + ] + } + ] + }, + "output": { + "format": "mp4", + "resolution": "hd", + "aspectRatio": "9:16", + "fps": 25 + } + } +}