Elbehery opened a new pull request, #1550: URL: https://github.com/apache/iceberg-rust/pull/1550
1. PositionDeleteFileWriter (crates/iceberg/src/writer/base_writer/position_delete_writer.rs) - Schema Implementation: Fixed schema with file_path (String) and pos (Long) columns - Field IDs: Correct Iceberg spec field IDs (2147483546, 2147483545) - Writer Integration: Full IcebergWriter trait implementation - Test Coverage: Unit tests for schema validation, batch creation, and writer functionality - Module Integration: Properly exposed in mod.rs 2. Equality Delete Parsing (crates/iceberg/src/arrow/caching_delete_file_loader.rs) - RecordBatch Processing: Converts Arrow batches to Iceberg predicates - Multi-type Support: Handles all Arrow data types (String, Int, Float, Date, etc.) - Logical Operations: Constructs proper AND/OR predicate trees - Error Handling: Robust handling of missing field IDs and null values 3. LZ4 Compression Support (crates/iceberg/src/puffin/) - Dependency Added: lz4_flex = "0.11" in Cargo.toml - Compression Logic: lz4_flex::compress_prepend_size() implementation - Decompression Logic: lz4_flex::decompress_size_prepended() implementation - Test Updates: All LZ4 tests now expect success instead of FeatureUnsupported 4. Delete Manifest Processing (crates/iceberg/src/transaction/snapshot.rs) - SnapshotProducer Extensions: Added added_delete_files field - Validation Logic: add_delete_files() with content type validation - Manifest Generation: write_delete_manifest() for delete entries - Summary Integration: Delete files included in snapshot summaries 5. Partition Support (crates/iceberg/src/writer/file_writer/parquet_writer.rs) - Hive-style Parsing: Extracts col=value patterns from file paths - Type Conversion: Converts string values to appropriate Literal types - Base64 Support: Handles binary partition values - Integration: Fully integrated into parquet_files_to_data_files() 6. NaN Value Counts (crates/iceberg/src/writer/file_writer/parquet_writer.rs) - Statistics Extraction: Reads Parquet row group statistics - NaN Detection: Uses NanValueCountVisitor for float/double columns - Async Implementation: Proper async file I/O handling - Integration: Passed to DataFileBuilder construction 7. Deletion Vector Support (crates/iceberg/src/arrow/caching_delete_file_loader.rs) - Puffin Detection: is_puffin_file() helper function - Blob Parsing: parse_delete_vector_blob() with RoaringTreemap - Index Integration: Added to PopulatedDeleteFileIndex - Context Enum: New DelVecs variant in DeleteFileContext 8. Case-Insensitive Predicate Binding (crates/iceberg/src/scan/) - Context Propagation: Added case_sensitive to FileScanTask, ManifestEntryContext, ManifestFileContext - Filter Integration: Used in build_equality_delete_predicate() - End-to-End Flow: Properly passed through scan planning pipeline Fixes https://github.com/apache/iceberg-rust/issues/1518 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
