pvary commented on code in PR #16363:
URL: https://github.com/apache/iceberg/pull/16363#discussion_r3559500028
##########
parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java:
##########
@@ -516,6 +341,52 @@ public void testAvroWriterRejectsVariantType() {
.hasMessage("Avro writer does not support variant types");
}
+ @Test
+ public void adaptiveBloomFilterSizingShrinksFile() throws IOException {
+ // when PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED is not set (the default),
the writer
+ // allocates the full PARQUET_BLOOM_FILTER_MAX_BYTES buffer (4 MiB here)
regardless
+ // of the number of distinct values written
+ long sizeWithoutAdaptive = writeWithBloomFilter(null);
+ assertThat(sizeWithoutAdaptive)
+ .as("non-adaptive file should pad the bloom filter to
PARQUET_BLOOM_FILTER_MAX_BYTES")
+ .isGreaterThan(3_500_000L);
+
+ // with PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED, the writer picks the
smallest candidate
+ // bloom filter that satisfies the actual number of distinct values (5) at
the
+ // configured FPP
+ long sizeWithAdaptive = writeWithBloomFilter(true);
+ assertThat(sizeWithAdaptive)
+ .as("adaptive file should be at least 2x smaller than the non-adaptive
file")
+ .isLessThan(sizeWithoutAdaptive / 2);
+ }
+
+ private long writeWithBloomFilter(Boolean adaptiveEnabled) throws
IOException {
Review Comment:
The Boolean is good if we want to keep the default value test.
We need to update the test cases, and also the `if` statement in the row
370-372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
