steveloughran commented on code in PR #16568:
URL: https://github.com/apache/iceberg/pull/16568#discussion_r3310358921
##########
api/src/main/java/org/apache/iceberg/variants/SerializedArray.java:
##########
@@ -35,28 +35,44 @@ static SerializedArray from(VariantMetadata metadata,
byte[] bytes) {
}
static SerializedArray from(VariantMetadata metadata, ByteBuffer value, int
header) {
+ return from(metadata, value, header, 0);
+ }
+
+ static SerializedArray from(VariantMetadata metadata, ByteBuffer value, int
header, int depth) {
Review Comment:
could this be private?
##########
api/src/main/java/org/apache/iceberg/variants/VariantUtil.java:
##########
@@ -31,8 +31,37 @@ class VariantUtil {
private static final int BASIC_TYPE_OBJECT = 2;
private static final int BASIC_TYPE_ARRAY = 3;
+ /**
+ * Maximum permitted nesting depth of a Variant value. The top-level value
is depth 0, so a
+ * Variant may contain up to {@code MAX_VARIANT_DEPTH} nested levels.
+ */
+ static final int MAX_VARIANT_DEPTH = 500;
+
private VariantUtil() {}
+ static VariantValue fromBuffer(VariantMetadata metadata, ByteBuffer value,
int depth) {
Review Comment:
add a javadoc, mention that the function includes validation, including
depth limits
##########
api/src/main/java/org/apache/iceberg/variants/SerializedArray.java:
##########
@@ -35,28 +35,44 @@ static SerializedArray from(VariantMetadata metadata,
byte[] bytes) {
}
static SerializedArray from(VariantMetadata metadata, ByteBuffer value, int
header) {
+ return from(metadata, value, header, 0);
+ }
+
+ static SerializedArray from(VariantMetadata metadata, ByteBuffer value, int
header, int depth) {
Preconditions.checkArgument(
value.order() == ByteOrder.LITTLE_ENDIAN, "Unsupported byte order: big
endian");
BasicType basicType = VariantUtil.basicType(header);
Preconditions.checkArgument(
basicType == BasicType.ARRAY, "Invalid array, basic type: " +
basicType);
- return new SerializedArray(metadata, value, header);
+ return new SerializedArray(metadata, value, header, depth);
}
private final VariantMetadata metadata;
private final ByteBuffer value;
private final int offsetSize;
private final int offsetListOffset;
private final int dataOffset;
+ private final int depth;
private final VariantValue[] array;
- private SerializedArray(VariantMetadata metadata, ByteBuffer value, int
header) {
+ private SerializedArray(VariantMetadata metadata, ByteBuffer value, int
header, int depth) {
this.metadata = metadata;
this.value = value;
+ this.depth = depth;
this.offsetSize = 1 + ((header & OFFSET_SIZE_MASK) >> OFFSET_SIZE_SHIFT);
int numElementsSize = ((header & IS_LARGE) == IS_LARGE) ? 4 : 1;
+ Preconditions.checkArgument(
+ value.remaining() >= HEADER_SIZE + numElementsSize,
+ "Invalid variant array: buffer too small for element count field");
int numElements = VariantUtil.readLittleEndianUnsigned(value, HEADER_SIZE,
numElementsSize);
+ Preconditions.checkArgument(
+ numElements >= 0, "Invalid variant array: negative element count %s",
numElements);
this.offsetListOffset = HEADER_SIZE + numElementsSize;
+ long offsetTableEnd = (long) offsetListOffset + ((long) numElements + 1L)
* offsetSize;
+ Preconditions.checkArgument(
+ offsetTableEnd <= value.remaining(),
+ "Invalid variant array: element count %s exceeds buffer",
+ numElements);
Review Comment:
I've just realised that this and the parquet-java hardening don't worry
about leftover data. "don't do that" is implicit the policy there, being as it
is useless.
I wonder what the rust parquet reader does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
