Optimize decimal scale power calculation - #10567
Open
jaideeppyne wants to merge 2 commits into
Open
Conversation
Replace 10_f64.powi(scale) with a bit-identical lookup table for valid decimal scales (0..=Decimal256Type::MAX_SCALE) in arrow-cast and parquet-variant-compute. - Add public decimal_f64_power helper with exhaustive bit-equality test - Use helper in decimal/float casts and Variant float-to-decimal conversion - Fall back to powi for negative/out-of-range scales Closes apache#10523
Author
Follow-up verification (evidence)Pushed a tightened revision:
Local: cast tests 326 passed, clippy -D warnings clean, fmt clean, |
jaideeppyne
marked this pull request as ready for review
August 6, 2026 11:20
Contributor
|
Does this make a measurable difference in any benchmarks? The |
Contributor
|
run benchmark cast_kernels |
This comment was marked as duplicate.
This comment was marked as duplicate.
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing agent/10523-decimal-power-lookup (0aa845f) to 6b7d6b3 (merge-base) diff Run configurationrun benchmark cast_kernelsCPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Contributor
this is a good point, i think we could achieve the same speedup via fixing the cast kernel to not have the power inside in the hot loop |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Decimal↔float casts recompute
10^scalewithf64::powi. Valid Arrow decimal scales are bounded byDecimal256Type::MAX_SCALE(76), so these powers can be looked up instead of recomputed.What changes are included in this PR?
10_f64.powi(0..=76)in one shared table.decimal_f64_power(scale)and use it in:arrow-castdecimal→float (single_decimal_to_float_lossy)arrow-castfloat→decimal (cast_floating_point_to_decimal)parquet-variant-computeVariant float→decimal (variant_to_unscaled_decimal)powifor negative or out-of-range scales (preserves prior behavior).10_f64.powifor every signed scale in±MAX_SCALE.Scope note (issue file list)
arrow-cast/src/cast/mod.rs10_f64.powiwith lookuparrow-cast/src/cast/decimal.rs10_f64.powiwith lookuparrow-arith/src/numeric.rs10_f64.powi; uses integerpow_checked/pow_wrappingon decimal natives (different path)arrow-cast/src/parse.rsf64::powi; uses integer powers for interval/decimal string parseparquet-variant-compute/.../type_conversion.rs10_f64.powi(scale)pattern — also convertedAre these changes tested?
Evidence from local validation on branch
agent/10523-decimal-power-lookup:Table bit-identity (standalone
rustccheck)Compared all 77 table entries to
10_f64.powi(i).to_bits()fori in 0..=76→bad=0 total=77.Unit test
cargo test -p arrow-cast test_decimal_f64_power_matches_powi --lib→ 1 passedCompares
.to_bits()for every scale in-(MAX_SCALE)..=MAX_SCALE.Cast suite
cargo test -p arrow-cast cast:: --lib→ 326 passed.Clippy
cargo clippy -p arrow-cast --all-targets --all-features -- -D warnings→ passed.Fmt
cargo fmt --all -- --check→ passed.Downstream compile
cargo check -p parquet-variant-compute→ passed (uses publicdecimal_f64_powerviaarrow::compute).Are there any user-facing changes?
powiresults).arrow_cast::cast::decimal_f64_power(also reachable viaarrow::computere-export). Pure function; no breaking change.AI assistance disclosure
Assisted generation/investigation was used; Jaideep Pyne is the human operator responsible for the submission. Changes were validated with the commands above.