chiragkava opened a new pull request, #17742:
URL: https://github.com/apache/iceberg/pull/17742
Fixes #17722.
`ColumnVector.getArrowVector()` allocates a decoded vector per call for
dictionary encoded columns and nothing released it (details in the issue).
Following the ownership model of #17296, the decoded vector is now materialized
once per batch and `ArrowBatchReader` releases it when the next `read()`
invalidates the batch and on `close()`. Only dict-decoded vectors are released,
so `reuseContainers` and the reader-owned holder vectors are untouched.
Spark/Flink don't use this code path.
Tested with a differential pair in `TestArrowReader`: the materializing test
fails on main (33,280 bytes retained for 100 rows / 1 batch / 1 string column)
and passes with the fix. the non-materializing control passes on both.
This contribution was developed with AI assistance Claude Code (Opus 5) and
I have reviewed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
