hoseinismm commented on issue #1884: URL: https://github.com/apache/iceberg-rust/issues/1884#issuecomment-4885123571
Based on all of the requests and proposals I have reviewed so far, the common idea has been to store spatial bounding regions (bounding boxes) within the table metadata so that query engines can prune irrelevant data during query planning. In contrast, I have proposed and implemented a different approach that I believe is more efficient than relying on metadata-based bounding boxes. Instead of storing spatial extents in metadata, the approach partitions the data itself using a structure inspired by the QuadTree spatial indexing method. As a result, spatial locality is preserved directly through the physical organization of the data. This design offers several advantages. First, it eliminates the need to sort data using space-filling curves such as Hilbert or Z-order, while achieving query performance that is very close to approaches based on those techniques. Second, and more importantly, it is well suited for datasets that grow incrementally over time. New records can be appended without requiring expensive global reorganization of the existing data. By contrast, approaches that rely on space-filling curve ordering face a significant challenge in incremental data ingestion. Maintaining the ordering typically requires repeated large-scale data reorganization and expensive shuffle operations as the dataset grows, making them considerably less practical for continuously accumulating datasets. To demonstrate the generality of this approach, I implemented it using all three spatial storage formats: **WKB**, **Spatial Parquet**, and **Flatten Spatial Parquet**. Importantly, the implementation does not require any modifications to either the Iceberg or Parquet frameworks. Instead, it is provided as an independent framework that operates on top of the existing systems. Consequently, it is not tied to a specific version of Iceberg or Parquet and can evolve independently. Naturally, if native support for this approach were to be incorporated directly into Iceberg or Parquet in the future, the implementation strategy and integration details would be different. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
