-
Notifications
You must be signed in to change notification settings - Fork 3.4k
API: Harden variant binary parsing against malformed input #16568
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
6cd1f92
94b3ff7
28b8d7e
63e0283
cd133ee
e465ed9
b99f75d
cea6d79
76c6409
1699d65
9ef504a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,8 @@ | |
| import java.nio.ByteOrder; | ||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.function.Function; | ||
| import org.apache.iceberg.relocated.com.google.common.base.Preconditions; | ||
| import org.apache.iceberg.util.ByteBuffers; | ||
|
|
||
| class VariantUtil { | ||
| private static final int BASIC_TYPE_MASK = 0b11; | ||
|
|
@@ -30,8 +32,45 @@ class VariantUtil { | |
| private static final int BASIC_TYPE_OBJECT = 2; | ||
| private static final int BASIC_TYPE_ARRAY = 3; | ||
|
|
||
| /** | ||
| * Maximum nesting depth in a Variant (permitted depths 0..MAX_VARIANT_DEPTH). Safety limit, not a | ||
| * spec bound. Matches parquet-java (apache/parquet-java#3562). | ||
| */ | ||
| static final int MAX_VARIANT_DEPTH = 1000; | ||
|
nssalian marked this conversation as resolved.
|
||
|
|
||
| /** | ||
| * Maximum element count for Variant containers and metadata dictionaries. Safety limit against | ||
| * buffer-to-heap allocation amplification. | ||
| */ | ||
| static final int MAX_ELEMENTS = 16_777_216; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the parquet pr doesn't enforce any limit here, and I'm not going to worry about one; the cost of array size is less than recursing down nested structures and I'm not aware of other bits of parquet imposing limits other than "you are free to run out of memory if you want to" |
||
|
|
||
|
nssalian marked this conversation as resolved.
|
||
| private VariantUtil() {} | ||
|
|
||
| /** Parses a variant value; validates input and enforces {@link #MAX_VARIANT_DEPTH}. */ | ||
| static VariantValue fromBuffer(VariantMetadata metadata, ByteBuffer value, int depth) { | ||
|
nssalian marked this conversation as resolved.
|
||
| Preconditions.checkArgument(depth >= 0, "Invalid variant: negative depth %s", depth); | ||
| Preconditions.checkArgument( | ||
| depth <= MAX_VARIANT_DEPTH, | ||
| "Invalid variant: nesting depth %s exceeds maximum %s", | ||
|
nssalian marked this conversation as resolved.
|
||
| depth, | ||
| MAX_VARIANT_DEPTH); | ||
| Preconditions.checkArgument(value.remaining() >= 1, "Invalid variant: empty value buffer"); | ||
| int header = ByteBuffers.readByte(value, 0); | ||
| BasicType basicType = basicType(header); | ||
| switch (basicType) { | ||
| case PRIMITIVE: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what about java17 switch now there's been the move and this is new, or at least moved, code? |
||
| return SerializedPrimitive.from(value, header); | ||
| case SHORT_STRING: | ||
| return SerializedShortString.from(value, header); | ||
| case OBJECT: | ||
| return SerializedObject.from(metadata, value, header, depth); | ||
| case ARRAY: | ||
| return SerializedArray.from(metadata, value, header, depth); | ||
| } | ||
|
|
||
| throw new UnsupportedOperationException("Unsupported basic type: " + basicType); | ||
| } | ||
|
|
||
| static float readFloat(ByteBuffer buffer, int offset) { | ||
| return buffer.getFloat(buffer.position() + offset); | ||
| } | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.