waterWang opened a new pull request, #17738:
URL: https://github.com/apache/iceberg/pull/17738
Every call to `ColumnVector.getArrowVector()` on a dictionary-encoded column
allocates a new decoded `FieldVector` that is never released. Since
`ColumnarBatch.createVectorSchemaRootFromVectors()` calls it for every column
of every batch, a scan over dict-encoded data leaks one vector per batch per
dict-encoded column, and the memory survives a fully drained and fully closed
scan.
**Root cause**
`ColumnVector.getArrowVector()` routes dict-encoded columns through
`DictEncodedArrowConverter.toArrowVector()` which allocates and populates a new
vector from the reader's allocator on each call. `ColumnVector.close()` only
closes the accessor, never the decoded vector.
**Fix**
Cache the decoded vector in a private field, materializing it once per batch
instead of on every `getArrowVector()` call, and release it in `close()`.
Repeated calls within the same batch return the same cached vector. This
matches the documented contract ("the arrow vectors are owned by the reader").
Closes #17722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
