raghav-reglobe commented on code in PR #16363:
URL: https://github.com/apache/iceberg/pull/16363#discussion_r3558756790
##########
parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java:
##########
@@ -516,6 +341,63 @@ public void testAvroWriterRejectsVariantType() {
.hasMessage("Avro writer does not support variant types");
}
+ @Test
+ public void adaptiveBloomFilterSizingShrinksFile() throws IOException {
+ // without PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED, the writer allocates the
full
+ // PARQUET_BLOOM_FILTER_MAX_BYTES buffer (4 MiB here) regardless of the
number
+ // of distinct values written
+ long sizeWithoutAdaptive = writeWithBloomFilter(false);
+ assertThat(sizeWithoutAdaptive)
+ .as("non-adaptive file should pad the bloom filter to
PARQUET_BLOOM_FILTER_MAX_BYTES")
+ .isGreaterThan(3_500_000L);
+
+ // with PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED, the writer picks the
smallest candidate
+ // bloom filter that satisfies the actual number of distinct values (5) at
the
+ // configured FPP
+ long sizeWithAdaptive = writeWithBloomFilter(true);
+ assertThat(sizeWithAdaptive)
+ .as("adaptive file should be at least 2x smaller than the non-adaptive
file")
+ .isLessThan(sizeWithoutAdaptive / 2);
+ }
+
+ @Test
+ public void adaptiveBloomFilterDisabledByDefault() throws IOException {
+ // when PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED is not set, the existing
behavior is
+ // preserved: the full PARQUET_BLOOM_FILTER_MAX_BYTES buffer is allocated
+ long size = writeWithBloomFilter(false);
+
+ assertThat(size)
+ .as("default write should pad the bloom filter to
PARQUET_BLOOM_FILTER_MAX_BYTES")
+ .isGreaterThan(3_500_000L);
Review Comment:
Semi-intentional — it was meant to pin the default (property-not-set)
behavior as its own test, but you're right that it duplicated the baseline of
the test above verbatim. Merged them: the non-adaptive write in
`adaptiveBloomFilterSizingShrinksFile` is the default path, and its comment now
says so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
