raunaqmorarka opened a new pull request, #18359: URL: https://github.com/apache/iceberg/pull/18359
`DeleteFileIndex` copies every v2 position delete with its `file_path` lower and upper bounds, plus the column sizes, value counts and null counts for that column. For a file-scoped position delete the two bounds are the same path, which the index only uses to compute the referenced data file location. This PR copies such deletes without column stats and records the location in `referencedDataFile`, the same shape a DV already has. Deletes that reference the same data file share one location `String`. Position deletes that span multiple data files keep the `file_path` bounds as before. ### Motivation A Trino coordinator heap dump showed `DeleteFileIndex` holding 4.3M position delete entries for 1.57M data files at about 2.25 KB per entry, roughly 9.7 GB, and the coordinator ran out of memory while planning. Most of the retained size per entry was the two `ByteBuffer` bounds and the stats maps around them, duplicated for every delete on the same data file. ### Results JOL retained size of a `DeleteFileIndex` built from a v2 delete manifest with 100k position deletes and 150-char paths: | data files | before | after | |---|---|---| | 100k (one delete per file) | 2122.6 B/delete | 810.6 B/delete | | 37k (2.7 deletes per file) | 1908.1 B/delete | 596.1 B/delete | ### Testing Extended `DeleteFileIndexTestBase` with file-scoped, preset `referencedDataFile`, and multi-file position delete cases that check the copied fields and stats. AI assistance (Claude Code) was used to draft the code, tests and the JOL measurement harness. I reviewed the logic and ran the tests and measurements myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
