Skip to content

feat: migrate ClickHouse adapter to utopia-php/query 0.3.x builder#4

Open
lohanidamodar wants to merge 1 commit into
mainfrom
feat/utopia-query-0.3.x
Open

feat: migrate ClickHouse adapter to utopia-php/query 0.3.x builder#4
lohanidamodar wants to merge 1 commit into
mainfrom
feat/utopia-query-0.3.x

Conversation

@lohanidamodar

@lohanidamodar lohanidamodar commented May 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrates the ClickHouse adapter to utopia-php/query 0.3.x (stable pin 0.3.*, currently resolving to the tagged 0.3.2 release — no dev-branch pins). Every table CREATE, materialized view, INSERT envelope, SELECT and DELETE now compiles through Schema\ClickHouse / Builder\ClickHouse; the adapter keeps only its HTTP transport, routing logic, and the few DDL shapes the library cannot express yet.

Rebased on current main, so the migration now covers everything that landed since the PR was opened: per-call tenant scoping, the daily-MV routing planner (raw / daily / hybrid), per-dim projections, dim-column backfills, the resourceType rename, the ip dimension, and the PSR-18 client transport.

What changed

Schema (DDL via Schema\ClickHouse)

  • createTable() emits the events/gauges tables through Table\ClickHouse: typed columns with codec(), Engine::MergeTree, partitionBy, orderBy, settings.
  • createDailyTable() uses Engine::SummingMergeTree; createDailyMaterializedView() goes through Schema\ClickHouse::createMaterializedView (the MV body remains a hand SELECT — subquery-aggregation MV bodies do not round-trip through the builder yet).
  • Column shapes the typed API cannot express are emitted via rawColumn() so the deployed DDL stays byte-for-byte identical: DateTime64(3, 'UTC') (timezone argument), LowCardinality(Nullable(String)) (the typed API emits the invalid Nullable(LowCardinality(...)) nesting), and the hyphenated skip-index names (index-path), which the typed index API rejects.
  • Projections (ADD PROJECTION), MODIFY SETTING, and the ADD COLUMN IF NOT EXISTS dim backfills stay raw SQL — no schema-layer support yet.

Reads (SELECT via Builder\ClickHouse)

  • Every builder is initialised with useNamedBindings() + withParamTypes(); all schema columns are pinned to their ClickHouse parameter types (DateTime64(3, 'UTC'), Int64, String) so placeholders never fall back to PHP value inference.
  • Migrated: find/findFromTable, findAggregatedFromTable (interval buckets + dim group-bys), count (including the bounded LIMIT max subquery), sum and the routed daily/hybrid paths, findDaily, sumDaily, sumDailyBatch, getTotal*, getTimeSeries.
  • The hybrid daily+raw sum compiles its two sides as independent builder statements and merges them under SUM(...) FROM (... UNION ALL ...), prefixing one side's named bindings to avoid placeholder collisions.
  • Cursor pagination keeps the tuple-keyset comparison as a whereRaw fragment with its own named bindings (upstream builder helper deferred).
  • contains()/containsAny()/notContains() filters are rewritten to IN / NOT IN before reaching the builder to keep set-membership semantics (the builder compiles Contains as substring LIKE).

Writes

  • addBatch() compiles the INSERT INTO t (...) FORMAT JSONEachRow envelope via insertFormat(); JSONEachRow body assembly and the non-retrying POST transport are unchanged.

Deletes

  • purge() / purgeDaily() emit lightweight DELETE FROM through the builder.

Query 0.3 API surface

  • Query::getMethod() returns the Method enum, so all string TYPE_* comparisons are gone (ClickHouse and Database adapters).
  • BREAKING: the UsageQuery::TYPE_GROUP_BY_INTERVAL / TYPE_GROUP_BY string constants are removed. UsageQuery::groupByInterval() and UsageQuery::groupBy() remain and now map to Method::GroupByTimeBucket / Method::GroupBy; groupBy() also accepts an array of columns to stay signature-compatible with the 0.3 base class.

Test plan

  • composer lint clean
  • composer check (PHPStan level max) clean
  • Full integration suite against Docker ClickHouse 25.11 + MariaDB: 251 tests, 1139 assertions green — including the SHOW CREATE TABLE schema assertions (codecs, LowCardinality, DateTime64(3, 'UTC'), index-* skip indexes), routing/dim-routing/projection tests, cursor pagination, and purge propagation

Still deferred upstream (follow-up PRs)

  • Tuple-cursor builder helper (cursor pagination still uses whereRaw)
  • LowCardinality(Nullable(...)) column type and timezone argument for DateTime64
  • Skip-index names containing hyphens
  • Projections / MODIFY SETTING / ADD COLUMN IF NOT EXISTS in the schema layer
  • MV bodies with aggregation subqueries

@greptile-apps

greptile-apps Bot commented May 17, 2026

Copy link
Copy Markdown

Greptile Summary

This PR migrates ClickHouse.php from hand-rolled SQL string assembly to utopia-php/query 0.3.x builders and schema helpers, and updates Database.php and UsageQuery.php to use the Method enum. All DDL (CREATE TABLE, materialized view), DML (INSERT via insertFormat, lightweight DELETE), and every SELECT path now go through typed named bindings (useNamedBindings() + withParamTypes()), removing the previous manual {paramN:Type} counter logic.

  • Schema layer (createTable, createDailyTable, createDailyMaterializedView): columns declared through the typed table API; raw fallbacks kept for DateTime64(3,'UTC') and LowCardinality(Nullable(String)) which the schema layer cannot yet express.
  • Read paths: builder-compiled SELECT with qualifyDdl rewriting bare table identifiers to db.table form. Tenant scoping is moved out of parseQueries into a dedicated applyFilters/applyTenantFilter pair called by every read/delete path.
  • Write / delete: INSERT envelope built via insertFormat; DELETE via builder->delete() with an unconditional whereRaw('1 = 1') safety guard.

Confidence Score: 4/5

Safe to merge once the two open interval-related threads from the prior review round are resolved or explicitly deferred with a tracking issue.

The builder migration is mechanically thorough and all call sites correctly call applyFilters after parseQueries. The two issues from earlier rounds — getTimeSeries rejecting intervals that find() accepts, and getTimeSeriesFromTable using a narrower function map than findAggregatedFromTable — remain open with no fix yet landed and affect callers using sub-hourly intervals.

src/Usage/Adapter/ClickHouse.php — the INTERVAL_FUNCTIONS constant and getTimeSeriesFromTable carry the open interval-consistency issues from the prior review round.

Important Files Changed

Filename Overview
src/Usage/Adapter/ClickHouse.php Major refactor replacing hand-written SQL with ClickHouseBuilder/ClickHouseSchema; all query paths now use typed named bindings. Two previously-flagged open issues remain: INTERVAL_FUNCTIONS only covers 2 of 8 valid intervals, and getTimeSeriesFromTable diverges from the bucket function available via findAggregatedFromTable.
src/Usage/Adapter/Database.php Migrated convertQueriesToDatabase switch from Query::TYPE_* string constants to Method enum cases; all supported filter types are correctly mapped.
src/Usage/UsageQuery.php UsageQuery now extends Query using Method enum; groupByInterval factory retained with its validation guard against VALID_INTERVALS; groupBy updated to accept string arrays.
composer.json utopia-php/query bumped to 0.3.*; minimum-stability still dev per the acknowledged pre-merge gate in the previous thread.
tests/Usage/Adapter/ClickHouseTest.php Integration test setup updated to match new adapter API; shared-table per-metric tenant test added.
CHANGELOG.md Breaking changes for query 0.3.x migration documented accurately.

Reviews (6): Last reviewed commit: "feat!: migrate ClickHouse adapter to uto..." | Re-trigger Greptile

Comment thread src/Usage/Adapter/ClickHouse.php Outdated
Comment thread composer.json Outdated
Comment thread phpstan-baseline.neon Outdated
Comment thread src/Usage/Adapter/ClickHouse.php Outdated
Comment on lines 156 to 170
@@ -172,49 +166,24 @@ public function setTimeout(int $milliseconds): self
return $this;
}

/**
* Enable or disable query logging for debugging.
*
* @param bool $enable Whether to enable query logging
* @return self
*/
public function enableQueryLogging(bool $enable = true): self
{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep the docs on public methods

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored docblocks on the public setter / lifecycle methods that lost them during the migration rewrite (__construct, setTimeout, enableQueryLogging, setCompression, setKeepAlive, setMaxRetries, setRetryDelay, setAsyncInserts, clearQueryLog, getName, healthCheck, setNamespace, setDatabase, setSecure, plus the namespace/tenant/sharedTables getters). Done in 0e0bbd6.

Comment thread src/Usage/Adapter/ClickHouse.php Outdated
Comment on lines +977 to +980
'integer' => $table->addColumn($id, ColumnType::BigInteger),
'float' => $table->addColumn($id, ColumnType::Float),
'boolean' => $table->addColumn($id, ColumnType::Boolean),
'datetime' => $table->datetime($id, 3),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be shorthand methods we can use for integer, float and boolean too

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swapped the addColumn(..., ColumnType::BigInteger|Float|Boolean) calls in the match block to the dedicated bigInteger() / float() / boolean() shorthand, plus the value BigInteger column in createDailyTable(). ColumnType import is no longer needed. SQL snapshot tests still green. Done in c2a261e.

Comment thread src/Usage/Adapter/ClickHouse.php Outdated
Comment thread src/Usage/Adapter/ClickHouse.php Outdated
Bump utopia-php/query from 0.1.* to 0.3.* and compile the ClickHouse
adapter's SQL through the shared builder and schema layer instead of
hand-assembled strings:

- Reads (find, count, sum, totals, time series, daily, hybrid
  daily+raw union) build through the ClickHouse query builder with
  typed named bindings; every schema column is pinned to its
  ClickHouse parameter type so placeholder types never fall back to
  PHP value inference.
- CREATE TABLE for the events/gauges/daily tables and the daily
  materialized view go through the schema layer. Column shapes the
  typed API cannot express (DateTime64 timezone argument,
  LowCardinality(Nullable(...)), hyphenated skip-index names) are
  emitted as raw definitions so deployed DDL is byte-for-byte
  unchanged. Projections, ALTER-based dim backfills and the MV body
  stay raw SQL, which the schema layer cannot express yet.
- INSERT envelopes compile via insertFormat(); the JSONEachRow body
  assembly and transport are unchanged.
- Query 0.3 API: getMethod() returns the Method enum, so all string
  TYPE_* comparisons are gone. UsageQuery keeps its groupByInterval()
  and groupBy() factories, now mapped to Method::GroupByTimeBucket and
  Method::GroupBy; groupBy() also accepts an array of columns to stay
  signature-compatible with the base class.
- contains()/containsAny()/notContains() filters are rewritten to
  IN / NOT IN before reaching the builder to keep set-membership
  semantics (the builder compiles Contains as substring LIKE).
@lohanidamodar lohanidamodar force-pushed the feat/utopia-query-0.3.x branch from 4530574 to 8fc13b3 Compare July 6, 2026 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants