[CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers#12433
Open
taiyang-li wants to merge 2 commits into
Open
[CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers#12433taiyang-li wants to merge 2 commits into
taiyang-li wants to merge 2 commits into
Conversation
|
Run Gluten Clickhouse CI on x86 |
…ut-iterator transformers Introduce an opt-in framework to pass stage input statistics (scan / shuffle / broadcast) to the native engine as estimated row-size hints. - Add InputStats / InputStatsKind / ShuffleStageWrapper and ApplyStageInputStatsRule. - Add RelNode.childNode() with implementations across all rel nodes; carry rowSize / inputStats on ReadRelNode and InputIteratorRelNode; expose PlanNode.getRelNodes() and AdvancedExtensionNode optimization/enhancement getters. - Add BasicScanExecTransformer.getInputStats hook and optional inputStats field on FileSourceScanExecTransformer; thread wsContext through genFirstStageIterator and the whole-stage RDDs; inject shuffle/broadcast stats in InputIteratorTransformer. - Gate everything behind spark.gluten.sql.enablePassStageInputStats (default false), so velox / clickhouse behavior is unchanged. Generated-by: TraeCli openrouter-3o Co-Authored-By: Aime <aime@bytedance.com> Change-Id: I7cd0a5abc98b6c41857230131513bfcb5ec6bf95
taiyang-li
force-pushed
the
pr/optional-stage-input-stats
branch
from
7c24700 to
1be15ce
Compare
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an opt-in plumbing path (gated by spark.gluten.sql.enablePassStageInputStats, default false) to propagate scan/shuffle/broadcast stage input statistics through Gluten’s transformers into Substrait rel nodes, so native engines can consume an estimated input hint.
Changes:
- Introduces
InputStats/InputStatsKind,ShuffleStageWrapper, andApplyStageInputStatsRuleto capture and apply stage statistics (including AQE shuffle read adjustments). - Threads
WholeStageTransformContextthrough first-stage iterators / whole-stage RDD compute paths so per-partition hints can be applied before native execution. - Extends Substrait rel node traversal (
RelNode.childNode()) and injects arowSize=...optimization string intoReadReladvanced extensions.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| gluten-substrait/src/main/scala/org/apache/spark/sql/execution/ColumnarCollapseTransformStages.scala | Extracts stage runtime stats in InputIteratorTransformer and attaches optional InputStats to InputIteratorRelNode. |
| gluten-substrait/src/main/scala/org/apache/gluten/extension/ApplyStageInputStatsRule.scala | New rule to traverse rel nodes and apply per-partition estimated hints derived from stage stats. |
| gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageZippedPartitionsRDD.scala | Applies stage input stats (when enabled) during final-stage RDD compute. |
| gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala | Threads wsCtx and partitionLength into whole-stage RDD creation paths. |
| gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala | Applies stage input stats (when enabled) during first-stage RDD compute and passes wsContext into iterator API. |
| gluten-substrait/src/main/scala/org/apache/gluten/execution/FileSourceScanExecTransformer.scala | Adds optional inputStats field and exposes it via getInputStats. |
| gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicScanExecTransformer.scala | Adds getInputStats hook and sets scan stats onto ReadRelNode when available. |
| gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala | Adds enablePassStageInputStats config entry and accessor. |
| gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/IteratorApi.scala | Extends iterator API signature to accept WholeStageTransformContext. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WriteRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/WindowGroupLimitRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/TopNNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SortRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/SetRelNode.java | Implements childNode() for rel traversal over multi-input rels. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/RelNode.java | Adds childNode() to the rel node interface. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java | Adds rowSize/inputStats fields and injects rowSize into advanced extensions. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ProjectRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/JoinRelNode.java | Implements childNode() returning both join children. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/InputIteratorRelNode.java | Adds rowSize/inputStats fields and injects rowSize into advanced extensions. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/GenerateRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FilterRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/FetchRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ExpandRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/CrossRelNode.java | Implements childNode() returning both cross children. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/AggregateRelNode.java | Implements childNode() for rel traversal. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/plan/PlanNode.java | Exposes rel list via getRelNodes() to support traversal from the whole-stage root. |
| gluten-substrait/src/main/java/org/apache/gluten/substrait/extensions/AdvancedExtensionNode.java | Adds getters for optimization/enhancement Any payloads. |
| gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShuffleStageWrapper.scala | New wrapper and stats types to extract AQE stage runtime stats. |
| backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala | Updates iterator API signature to accept wsContext (currently unused in this backend). |
| backends-clickhouse/src/main/scala/org/apache/gluten/backendsapi/clickhouse/CHIteratorApi.scala | Updates iterator API signature to accept wsContext (currently unused in this backend). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+80
to
+84
| if (GlutenConfig.get.enablePassStageInputStats) { | ||
| val inputStatsOpt: Option[InputStats] = { | ||
| val inputAdapter = child.asInstanceOf[ColumnarInputAdapter] | ||
| inputAdapter.child match { | ||
| case shuffle @ ShuffleStageWrapper(stats) => |
marin-ma
reviewed
Jul 3, 2026
marin-ma
left a comment
marin-ma
left a comment
Contributor
There was a problem hiding this comment.
@taiyang-li Thanks. Just add some minor comments. Can you please also address the comments from Copilot?
|
Run Gluten Clickhouse CI on x86 |
Contributor
|
@taiyang-li Hello, do we have any issues or design documents outlining the works required to introduce the BOLT engine? Also, it would be best to discuss and confirm the plan before starting the work. A more suitable approach would be to outline an issue and the corresponding design documents based on extending the Gluten backends interface to support additional backends. We can then discuss this and proceed with the discussion on Bolt only after the initial integration is complete. |
This was referenced Jul 6, 2026
Contributor
Author
|
@jackylee-ch Thanks for pointing this out. We have created a tracking issue here: #12456, and I also updated the PR description in #12454 accordingly. Our current plan is not to land the full Bolt backend directly in one step. Instead, we want to first split out the common-layer changes into a few smaller PRs (such as #12432, #12437, #12435, #12433 and #12431), so they can be reviewed firstly and make sure they do not affect the existing Velox / CH backends. PR #12454 is the Bolt backend part built on top of those common patches. We’d like the common patches to be discussed first, and then continue the discussion on whether Bolt itself should be accepted as one of the supported backends in Gluten through the normal dev discussion / vote process. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
One of a small series of common-code changes to introduce a new backend — Bolt (ByteDance’s unified lakehouse analytics acceleration engine) — into the Gluten community. The series minimizes the delta to Gluten common code so Bolt can plug in cleanly, while leaving Velox and ClickHouse backends unaffected. It is part of plan in #12456
This PR
Introduce an opt-in framework to pass stage input statistics (scan / shuffle / broadcast) to the native engine as estimated row-size hints.
InputStats/InputStatsKind/ShuffleStageWrapperandApplyStageInputStatsRule.RelNode.childNode()with implementations across all rel nodes; carryrowSize/inputStatsonReadRelNodeandInputIteratorRelNode; exposePlanNode.getRelNodes()andAdvancedExtensionNodeoptimization/enhancement getters.BasicScanExecTransformer.getInputStatshook and optionalinputStatsfield onFileSourceScanExecTransformer; threadwsContextthroughgenFirstStageIteratorand the whole-stage RDDs; inject shuffle/broadcast stats inInputIteratorTransformer.spark.gluten.sql.enablePassStageInputStats(defaultfalse), so Velox / ClickHouse behavior is unchanged.