alessandro-nori commented on code in PR #5724:
URL: https://github.com/apache/datafusion-comet/pull/5724#discussion_r3949849148
##########
native/core/src/execution/operators/iceberg_write.rs:
##########
@@ -665,8 +656,140 @@ fn build_writer_properties(settings:
&IcebergParquetWriteSettings) -> DFResult<W
.set_dictionary_page_size_limit(settings.dict_size_bytes as usize)
.set_data_page_row_count_limit(settings.page_row_limit as usize)
.set_statistics_enabled(EnabledStatistics::Page)
- .set_statistics_truncate_length(None)
- .build())
+ .set_statistics_truncate_length(None);
+ for column in &settings.bloom_filter_enabled_columns {
+ let path = ColumnPath::from(column.as_str());
Review Comment:
this differs from the Java implementation for map and list columns (e.g. for
a list `tags.element` vs `tags.list.element`) and the filter would be silently
omitted by Java readers.
In iceberg-rust there is a schema visitor called `IndexByParquetPathName`
but it is private.
Could we consider resolving the Parquet paths on the driver (Scala)? Or
maybe making the iceberg-rust visitor public
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
