Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
## [Unreleased]

### 📚 Documentation

- Rename the user-facing term "managed database" to "instant database" across help text, README, and skill docs. Command names, flags, and server wire-format values (such as the `managed_load` job type) are intentionally unchanged.

## [0.27.1] - 2026-08-22

### 🐛 Bug Fixes
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ hotdata databases load --catalog demo --table trips \
hotdata query "SELECT count(*) FROM demo.public.trips"
```

The core loop: create a **managed database**, put data in it, query it with
The core loop: create an **instant database**, put data in it, query it with
PostgreSQL-dialect SQL. Everything else builds on that.

## Getting your data in
Expand All @@ -62,7 +62,7 @@ hotdata ingest run <run_…> --wait # show one attempt; exits 0 don
```

`ingest sources update-config` rotates credentials; `ingest pause|resume|schedule`
control a scheduled ingest. Data lands in a managed database — query it like any
control a scheduled ingest. Data lands in an instant database — query it like any
other.

## Query and explore
Expand All @@ -88,7 +88,7 @@ with `hotdata databases queries list`.

## Join across sources

Attach another catalog to a managed database and join its live tables directly,
Attach another catalog to an instant database and join its live tables directly,
no copying:

```sh
Expand Down Expand Up @@ -134,10 +134,10 @@ The full command surface. The top level has eight groups — `auth`, `workspaces
| `auth status` | Show authentication status |
| `workspaces list` | List all workspaces |
| `workspaces use` | Set the default workspace |
| `databases list` | List managed databases in the workspace |
| `databases count` | Count managed databases in the workspace |
| `databases show` | Show details for a managed database |
| `databases create` | Create a new managed database |
| `databases list` | List instant databases in the workspace |
| `databases count` | Count instant databases in the workspace |
| `databases show` | Show details for an instant database |
| `databases create` | Create a new instant database |
| `databases fork` | Fork a database into a new, independent database |
| `databases attach` | Attach a catalog so its tables are queryable |
| `databases detach` | Detach a previously attached catalog |
Expand Down
38 changes: 19 additions & 19 deletions skills/hotdata/SKILL.md

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions skills/hotdata/references/DATA_MODEL.template.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,17 +52,17 @@ For each business entity:

Document safe join paths and caveats (fan-out, timing, different refresh cadence, type mismatches).

> A cross-catalog join runs inside one managed database; each catalog it touches must be **attached** to that database (`hotdata databases attach <catalog>`) so its live tables are in query scope. Note here which catalogs a join requires attached, and the alias each is attached under. See **`hotdata`** skill → Querying across catalogs.
> A cross-catalog join runs inside one instant database; each catalog it touches must be **attached** to that database (`hotdata databases attach <catalog>`) so its live tables are in query scope. Note here which catalogs a join requires attached, and the alias each is attached under. See **`hotdata`** skill → Querying across catalogs.

## Search & index summary (optional)

| Table | Column | Kind (vector / text / …) | Index status | Notes |
|-------|--------|--------------------------|--------------|-------|
| | | | | |

_Use `hotdata search list` (no flags — covers catalog tables **and** managed databases; see **hotdata-search** skill). Record text/vector indexes here; sorted indexes for OLAP filters in **hotdata-analytics**._
_Use `hotdata search list` (no flags — covers catalog tables **and** instant databases; see **hotdata-search** skill). Record text/vector indexes here; sorted indexes for OLAP filters in **hotdata-analytics**._

## Managed databases (uploaded)
## Instant databases (uploaded)

Catalog from `hotdata databases list` / `hotdata databases tables list`:

Expand All @@ -72,7 +72,7 @@ Catalog from `hotdata databases list` / `hotdata databases tables list`:

## Derived tables (Chain)

Stable `<catalog>.public.*` tables built for **Chain** workflows (materialized into managed databases):
Stable `<catalog>.public.*` tables built for **Chain** workflows (materialized into instant databases):

| Table name | Built from | Purpose | Owner / TTL |
|------------|------------|---------|-------------|
Expand Down
14 changes: 7 additions & 7 deletions skills/hotdata/references/MODEL_BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ Optional **deep pass** for a single authoritative markdown document stored as **

## 1. Discover catalogs and tables

List the catalogs you can query — managed databases you own and any attached catalogs — and the tables they expose:
List the catalogs you can query — instant databases you own and any attached catalogs — and the tables they expose:

```bash
hotdata databases list # managed databases (catalogs you own)
hotdata databases list # instant databases (catalogs you own)
hotdata databases tables list # every workspace table, as <catalog>.<schema>.<table>
```

For each catalog, record its name and the tables it exposes. (Pulling *new* external data into a managed database is a separate step — see the `ingest sources` and `ingest` commands in the core skill.)
For each catalog, record its name and the tables it exposes. (Pulling *new* external data into an instant database is a separate step — see the `ingest sources` and `ingest` commands in the core skill.)

---

Expand All @@ -31,14 +31,14 @@ A datasource's schema is discovered when it is added. If the source schema may h
hotdata databases tables list --schema <schema> --table <table>
```

**Managed databases:**
**Instant databases:**

```bash
hotdata databases list
hotdata databases tables list
```

Capture schema for each managed-database table (columns, types) from the table listing.
Capture schema for each instant-database table (columns, types) from the table listing.

You can also re-check a datasource's discovered schema after enumeration if you suspect drift:

Expand Down Expand Up @@ -95,7 +95,7 @@ Per table when you only need one:
hotdata search list --schema <schema> --table <table> [-w <workspace_id>]
```

Managed-database indexes are included in the no-flag whole-workspace `search list` (shown under the internal `__db_<id>.<schema>.<table>` label); narrow to one with `--schema` / `--table` as above.
Instant-database indexes are included in the no-flag whole-workspace `search list` (shown under the internal `__db_<id>.<schema>.<table>` label); narrow to one with `--schema` / `--table` as above.

Note:

Expand All @@ -113,7 +113,7 @@ This Markdown body is what you store as **context:DATAMODEL** (`hotdata database

- **Overview** — Domains and what the workspace is for.
- **Per catalog** — Optional subsection per source; for **deep** models, **repeat** one block per `catalog.schema.table` (grain, column table with name/type/nullable/PK-FK/notes, relationships, queryability, caveats)—the template’s single `####` heading is a pattern to copy for each table.
- **Managed databases** — Same treatment as catalog tables where relevant.
- **Instant databases** — Same treatment as catalog tables where relevant.
- **Cross-catalog joins** — Keys, semantics, type caveats.
- **Search / index summary** — Table, column, index status, intended use.

Expand Down
26 changes: 13 additions & 13 deletions skills/hotdata/references/WORKFLOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ The `hotdata` skill is always loaded first (auth and workspace setup). The three
| User goal | Skill | Key commands |
|-----------|--------|----------------|
| Login, workspaces, datasources, tables, context | **`hotdata`** | `auth`, `workspaces`, `ingest sources`, `ingest`, `databases tables`, `databases context` |
| Load parquet files into a managed database | **`hotdata`** | `databases create` + `databases load` |
| Load parquet files into an instant database | **`hotdata`** | `databases create` + `databases load` |
| SQL analytics, aggregations, history, Chain | **`hotdata-analytics`** (`subskills/analytics/SKILL.md`) | `query`, `databases queries`, `databases results` |
| BM25 / vector search, retrieval indexes | **`hotdata-search`** (`subskills/search/SKILL.md`) | `search`, `search create`, `search embeddings` |
| Geospatial / PostGIS-style SQL | **`hotdata-geospatial`** (`subskills/geospatial/SKILL.md`) | `query` with `ST_*`, WKB columns |

| Concept | Where documented |
|--------|------------------|
| **Model** | This file — [Model](#model) |
| **Upload path (managed databases)** | This file — [Managed databases](#managed-databases) |
| **Upload path (instant databases)** | This file — [Instant databases](#instant-databases) |
| **History / Chain** | **`hotdata-analytics`** — [WORKFLOWS.md](../subskills/analytics/references/WORKFLOWS.md) |
| **Search indexes** | **`hotdata-search`** — [INDEXES.md](../subskills/search/references/INDEXES.md) |
| **Epic flows** | This file — [Epic flows](#epic-flows) |
Expand All @@ -42,14 +42,14 @@ End-to-end checklists. Use the linked sections for command detail and guardrails
6. [ ] (Optional) `hotdata databases context list` — if `DATAMODEL` is listed, `hotdata databases context show DATAMODEL`; else skip `show`
7. [ ] (Optional) Bootstrap **context:DATAMODEL** — [Model](#model), [DATA_MODEL.template.md](DATA_MODEL.template.md)

**Next:** upload data ([Managed databases](#managed-databases)) or run analytics (**Chain** below).
**Next:** upload data ([Instant databases](#instant-databases)) or run analytics (**Chain** below).

### Chain (materialize then query)

**Skill:** **`hotdata-analytics`** (catalog via **`hotdata`**)

1. [ ] Run base SQL: `hotdata query "SELECT …"` — poll `hotdata query status <id>` if async
2. [ ] Materialize into a managed database: `hotdata databases create --catalog <alias> --table <name>` then `hotdata databases load --catalog <alias> --table <name> --file ./….parquet`
2. [ ] Materialize into an instant database: `hotdata databases create --catalog <alias> --table <name>` then `hotdata databases load --catalog <alias> --table <name> --file ./….parquet`
3. [ ] Query with the catalog-qualified name `<alias>.public.<name>`
4. [ ] Chain: `hotdata query "SELECT … FROM <alias>.public.<name> WHERE …"`
5. [ ] Record stable chains in **context:DATAMODEL** when they should outlive the session
Expand All @@ -63,7 +63,7 @@ End-to-end checklists. Use the linked sections for command detail and guardrails
1. [ ] `hotdata databases tables list` (filter with `--schema`/`--table`) — pick text column (BM25) or embedding/text column (vector)
2. [ ] `hotdata search list` — avoid duplicate text/vector indexes on the same column
3. [ ] Create index (address by name):
- [ ] **Managed DB only:** `hotdata search create <tbl>_<col> --type text --from <alias>.public.<tbl> --column <text_col>` (vector: `--type vector [--provider <p>]`). An external catalog must be attached to a managed database first (`hotdata databases attach`).
- [ ] **Instant DB only:** `hotdata search create <tbl>_<col> --type text --from <alias>.public.<tbl> --column <text_col>` (vector: `--type vector [--provider <p>]`). An external catalog must be attached to an instant database first (`hotdata databases attach`).
- [ ] Large build: add `--async`, then `hotdata jobs <job_id>`
4. [ ] Search (address the index by name):
- [ ] `hotdata search "…" --index <tbl>_<col>`
Expand All @@ -75,9 +75,9 @@ End-to-end checklists. Use the linked sections for command detail and guardrails

**Skill:** **`hotdata`**

A `hotdata query` runs inside **one** managed database; its scope sees that database's own catalog plus **attached** catalog catalogs only. To query a catalog's tables — or join a managed table against a live catalog table in one query — attach the catalog. (No managed database set → *"a database is required."*; an unattached catalog → *"table not found."*)
A `hotdata query` runs inside **one** instant database; its scope sees that database's own catalog plus **attached** catalog catalogs only. To query a catalog's tables — or join a managed table against a live catalog table in one query — attach the catalog. (No instant database set → *"a database is required."*; an unattached catalog → *"table not found."*)

1. [ ] Pick/create the managed database that will be the query context (`hotdata databases use <id>` or `databases create --catalog <alias>`)
1. [ ] Pick/create the instant database that will be the query context (`hotdata databases use <id>` or `databases create --catalog <alias>`)
2. [ ] Attach the catalog(s) you need (live, sync intact): `hotdata databases attach <catalog> [--alias <a>]`
- Or attach at creation: `hotdata databases create --catalog <alias> --attach <catalog>[=<alias>]`
3. [ ] Confirm scope: `hotdata databases <id>` lists attached catalogs
Expand All @@ -88,11 +88,11 @@ A `hotdata query` runs inside **one** managed database; its scope sees that data

---

## Managed databases
## Instant databases

**Managed databases** land queryable tables you own in the workspace, addressed in SQL as `<catalog>.<schema>.<table>` where the catalog is the `--catalog` alias.
**Instant databases** land queryable tables you own in the workspace, addressed in SQL as `<catalog>.<schema>.<table>` where the catalog is the `--catalog` alias.

| | **Managed databases** |
| | **Instant databases** |
|---|------------------------|
| **Best for** | Parquet files you own; catalog-style `alias.schema.table` |
| **SQL prefix** | `<catalog>.<schema>.<table>` where catalog = `--catalog` alias |
Expand All @@ -101,9 +101,9 @@ A `hotdata query` runs inside **one** managed database; its scope sees that data
| **Parquet file uploads** | `databases load --file` / `--url` / `--upload-id` |
| **Refresh** | Replace via `databases load` again |

**Rule of thumb:** Parquet files you control as **`mydb.public.orders`** → **managed databases**.
**Rule of thumb:** Parquet files you control as **`mydb.public.orders`** → **instant databases**.

### Workflow: managed database (parquet)
### Workflow: instant database (parquet)

1. Create the database with a catalog alias:

Expand All @@ -127,7 +127,7 @@ A `hotdata query` runs inside **one** managed database; its scope sees that data
hotdata query "SELECT count(*) FROM sales.public.orders"
```

For **Chain** materializations into managed databases, see **`hotdata-analytics`**.
For **Chain** materializations into instant databases, see **`hotdata-analytics`**.

### Workflow: fork before risky changes

Expand Down
8 changes: 4 additions & 4 deletions skills/hotdata/subskills/analytics/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: hotdata-analytics
description: Use this skill when the user wants OLAP-style SQL analytics in Hotdata — aggregations, GROUP BY, JOINs, reporting, exploratory queries, query run history, stored results, or materialized follow-up tables (Chain into managed databases). Activate for "analyze", "aggregate", "rollup", "pivot", "report", "metrics", "GROUP BY", "query history", "past queries", "query runs", "stored results", "materialize", "chain", "intermediate table", or sorted indexes for filters/range scans. Do not load for BM25/vector search or geospatial SQL — use hotdata-search or hotdata-geospatial. Requires the core hotdata skill for tables and auth.
description: Use this skill when the user wants OLAP-style SQL analytics in Hotdata — aggregations, GROUP BY, JOINs, reporting, exploratory queries, query run history, stored results, or materialized follow-up tables (Chain into instant databases). Activate for "analyze", "aggregate", "rollup", "pivot", "report", "metrics", "GROUP BY", "query history", "past queries", "query runs", "stored results", "materialize", "chain", "intermediate table", or sorted indexes for filters/range scans. Do not load for BM25/vector search or geospatial SQL — use hotdata-search or hotdata-geospatial. Requires the core hotdata skill for tables and auth.
version: 0.27.1
---

Expand All @@ -25,7 +25,7 @@ hotdata query status <query_run_id>
- **`--dialect`** (default `hotsql`): write SQL in `duckdb`/`postgres`/`snowflake` and the server transpiles it to HotSQL before running (e.g. Snowflake `IFF(...)`, DuckDB `len(...)`). Read-only queries only for a non-`hotsql` dialect.
- Use **`hotdata databases tables list`** for schema discovery — not `information_schema` via `query`.
- Fully qualified names: `<catalog>.<schema>.<table>`, `<database>.<schema>.<table>`.
- **Query scope:** every query runs inside one managed database (active or `--database`); it sees that database's own catalog plus **attached** catalogs only. To query an attached catalog's table, or **join a managed table against an attached catalog's table**, attach the catalog first: `hotdata databases attach <catalog>` — see **`hotdata`** skill → [Querying across catalogs](../../SKILL.md#querying-across-catalogs-attach). No managed database set → *"a database is required."*
- **Query scope:** every query runs inside one instant database (active or `--database`); it sees that database's own catalog plus **attached** catalogs only. To query an attached catalog's table, or **join a managed table against an attached catalog's table**, attach the catalog first: `hotdata databases attach <catalog>` — see **`hotdata`** skill → [Querying across catalogs](../../SKILL.md#querying-across-catalogs-attach). No instant database set → *"a database is required."*
- Long-running queries may return `query_run_id` → poll with **`query status`** (exit `2` = still running). Do not re-run identical heavy SQL while polling.
- For **workspace-wide** joins and naming, load **context:DATAMODEL** when listed (`hotdata databases context list` → `show DATAMODEL`) — see **`hotdata`** skill.

Expand All @@ -34,7 +34,7 @@ hotdata query status <query_run_id>
Typical analytics SQL (all via `hotdata query`):

- **Aggregations:** `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` with `GROUP BY`
- **Joins:** `INNER` / `LEFT JOIN` across `<catalog>.<schema>.<table>` names — every referenced catalog (the managed database's own or an attached one) must be in the active database's scope; attach catalogs first (`hotdata databases attach`)
- **Joins:** `INNER` / `LEFT JOIN` across `<catalog>.<schema>.<table>` names — every referenced catalog (the instant database's own or an attached one) must be in the active database's scope; attach catalogs first (`hotdata databases attach`)
- **Filtering:** `WHERE` on partition-friendly columns (consider **sorted** indexes below)
- **Ordering:** `ORDER BY` on metrics or dimensions
- **Bounded exploration:** always `LIMIT` while iterating; widen once validated
Expand Down Expand Up @@ -82,7 +82,7 @@ hotdata databases results get <result_id> [--workspace-id <workspace_id>] [--out
hotdata query status <query_run_id> # if async
```

2. **Materialize** into a managed database (parquet)
2. **Materialize** into an instant database (parquet)

```bash
hotdata databases create --catalog analytics
Expand Down
6 changes: 3 additions & 3 deletions skills/hotdata/subskills/analytics/references/WORKFLOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

OLAP-style SQL, **History** (query runs and stored results), and **Chain** (materialized follow-ups). Requires **`hotdata`** for auth, workspaces, and catalog commands.

**Related:** **`hotdata-search`** ([`../../search/SKILL.md`](../../search/SKILL.md)) for BM25/vector indexes and `hotdata search`; **`hotdata`** [WORKFLOWS.md](../../../references/WORKFLOWS.md) for managed databases.
**Related:** **`hotdata-search`** ([`../../search/SKILL.md`](../../search/SKILL.md)) for BM25/vector indexes and `hotdata search`; **`hotdata`** [WORKFLOWS.md](../../../references/WORKFLOWS.md) for instant databases.

---

Expand Down Expand Up @@ -64,7 +64,7 @@ hotdata query "SELECT ..."

### 2. Materialize

Land a smaller table in a **managed database** (parquet → `<database>.<schema>.<table>`):
Land a smaller table in an **instant database** (parquet → `<database>.<schema>.<table>`):

```bash
hotdata databases create --catalog chain_db
Expand Down Expand Up @@ -92,4 +92,4 @@ hotdata query "SELECT * FROM chain_db.public.revenue_slice WHERE ..."

- Materialize when the base scan is large and the follow-up runs many times.
- Keep Chain tables focused; avoid wide `SELECT *` materializations when a narrow projection suffices.
- For managed-database uploads, see **`hotdata`** WORKFLOWS — [Managed databases](../../../references/WORKFLOWS.md#managed-databases).
- For instant-database uploads, see **`hotdata`** WORKFLOWS — [Instant databases](../../../references/WORKFLOWS.md#instant-databases).
Loading
Loading