Feat/variant benchmark - #11
Merged
Merged
Conversation
Compare DuckDB v1.5.5 VARIANT against shredded typed columns on the same OTLP/JSON traces, with ingest/query/storage split and host metadata in the JSON so a powerful remote can reproduce and send results back.
VARIANT in v1.5.5 compresses flat rows well but loses ingest and every query to typed columns; pin DuckDB workers on many-core ARM so nested UNNEST does not oversubscribe.
VARIANT-flat was not a better compressor: same DICT_FSST/BitPacking after shredding. The gap was per-worker optimistic flush plus DELETE residue. Default drain now keeps collections in memory until CHECKPOINT (global partial blocks), so used size matches VARIANT-flat. Envelope VARIANT is off by default.
The v1.5 shell can stall on piped stdin via terminal color probing; skip that with -dark-mode and DUCKDB_NO_HIGHLIGHT, pre-create v1.5.0 DBs via one-shot -c, and read stdout from a dedicated thread instead of select on BufferedReader.
duckdb was already on v1.5.5; bump extension-ci-tools from an older v1.5-variegata SHA to the v1.5.5 branch tip and point reusable workflows at @v1.5.5 so Actions and the submodule stay aligned.
Confirms the packed-size tie and query ranking after the Linux CLI stdin fix on NVIDIA GB10 (Cortex-X925/A725).
Same ranking on Linux aarch64: packed size ties/beats VARIANT-flat, typed columns stay in the low-ms club, VARIANT extract still slower than JSON ->>.
Drop editorial narrative, commit SHAs, and packing/dev commentary from the VARIANT section; trim similar prose elsewhere.
…ert races. Row-routing: replace the per-JSON-key hash+allocation lookup in RawExtractor::Traverse/RawNode::GetOrCreateChild with an allocation-free linear scan for the common case of <=24 children per schema node (falls back to the hash map for pathologically wide objects). Confirmed via profiling to remove real CPU work on this path. Batch reader: avoid a redundant large-buffer copy at NDJSON batch boundaries in the file-ingest reader thread (move the batch prefix out, copy only the small unconsumed tail). OTLP normalize: replace a per-object heap vector allocation in OtlpNormalizeObject with an inline stack buffer for the common case. JSON parsing: back RawPayload::Parse()'s yyjson_read calls with a per-payload pool allocator (sized via yyjson_read_max_memory_usage, falls back to the default allocator if ever insufficient) instead of one malloc/free pair per NDJSON line. Also pool Explode()'s mutable working doc via yyjson's dynamic allocator. Fixed a real use-after-free this surfaced: MergeParsedPayloads (small-batch coalescing) moved parsed docs between RawPayload objects without moving pool-buffer ownership with them, crashing the sqllogictest suite deterministically. Concurrency fix: concurrent HTTP/programmatic requests racing to INSERT into a table that doesn't exist yet each open their own transaction, and DuckDB's catalog allows only one to CREATE TABLE -- the rest saw a TransactionException surfaced as an HTTP 400, which OTLP exporters correctly do not retry, silently dropping that batch. Fixed with RawIngestSerialized: a table's first-ever insert queues behind an in-process lock instead of racing (or retrying blind against an uncommitted winner); every request afterward never touches the lock, just a cached membership check, so steady-state throughput is unchanged. Verified: full sqllogictest suite repeatedly, 30/30 clean in a 16-way concurrent stress test that previously failed intermittently, and the regression test added to test/http/raw_api_compat.sh.
…rd arm64 numbers.
run_otel_streaming.{sh,py} + otel_gen_load.py drive real OTLP/HTTP traffic
(actual OpenTelemetry Python SDK, protobuf wire format, concurrent exporter
processes) into raw_serve() instead of bulk-loading an NDJSON file --
this is what caught the concurrent first-insert race fixed alongside it.
run_otel_streaming.sh manages a dedicated venv for the SDK dependency
(a plain pip install is refused on externally-managed Python installs).
BENCHMARK.md: replace the 100k-record NVIDIA Spark GB10 cross-check with a
full 1M-record run (matching the M3 Ultra scale) taken after the perf
fixes, confirming the VARIANT/JSON ranking holds on a different core
architecture entirely (same order of magnitude on every query speedup
ratio, ingest edge over JSON-exploded matching M3 almost exactly).
Integrate arm64 parse/ingest perf fixes, concurrent first-insert serialization, OTLP streaming benchmark harness; refresh Spark GB10 numbers in the lean BENCHMARK.md tables.
…steps. Update OTEL bulk, streaming, and VARIANT tables from the latest 1M runs; drop inline generators, dev notes, and GH appendix extras.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmark DuckDB VARIANT (v1.5.5) against RawDuck on the same OTLP trace workload, and land the fixes those benchmarks exposed.
run_variant.sh): 1M-span ingest, storage, and query comparison across RawDuck typed columns, VARIANT-flat, VARIANT OTLP, JSON-flat, and JSON OTLP — reproducible on M3 Ultra and Spark GB10 aarch64.RawIngestSerializedserializes a table’s first HTTP insert so racing OTLP exporters don’t hit catalog write-write conflicts (HTTP 400, silent batch loss). Steady-state appends are unaffected.run_otel_streaming.sh): real OpenTelemetry SDK →raw_serve()protobuf traffic; caught and validates the concurrency fix.Headline (1M spans, M3 Ultra): RawDuck ~1.0s ingest, ~1–3 ms queries, ~39 MB disk. VARIANT OTLP ~12s ingest, hundreds–thousands of ms queries. Storage ties VARIANT-flat; VARIANT extract is slower than JSON
->>.