chiragkava opened a new issue, #17722: URL: https://github.com/apache/iceberg/issues/17722
### Apache Iceberg version 1.11.0 (latest release) ### Query engine Other ### Please describe the bug 🐞 ### Summary Every call to `ColumnVector.getArrowVector()` on a dictionary encoded column allocates a new decoded `FieldVector` that is never released. Since `ColumnarBatch.createVectorSchemaRootFromVectors()` calls it for every column of every batch, a scan over dict-encoded data leaks one vector per batch per dict-encoded column. The memory survives a fully drained and fully closed scan.. `ColumnVector.getArrowVector()` routes dict-encoded columns through [`DictEncodedArrowConverter.toArrowVector()`](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/DictEncodedArrowConverter.java#L51-L85) which allocates and populates a new vector from the reader's allocator on each call. And [`ColumnVector.close()`](https://github.com/apache/iceberg/blob/main/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/ColumnVector.java#L85-L88) closes only the accessor , `BaseBatchReader.closeVectors()` closes only the holders' vectors The leak goes undetected by Arrow's allocator leak checking because the `VectorizedReadBuilder` child allocator the memory is charged to is itself never closed. so the check never runs. ### Reproduction Write a Parquet file whose string column has few distinct values (so pages are dict-encoded), scan it with `VectorizedTableScanIterable`, call `createVectorSchemaRootFromVectors()` on each batch, close everything, and compare `ArrowAllocation.rootAllocator().getAllocatedMemory()` against the pre scan baseline taken after creating the dictionaryEncodedTable. The allocated memory never comes back down it stays higher by exactly the size of the decoded vectors (33,280 bytes for 100 rows / 1 batch / 1 string column on current main). Running the same scan without calling createVectorSchemaRootFromVectors() returns to baseline, so the conversion is what's leaking. ### Related issues - #13937 — `VectorizedReadBuilder` child allocator never closed (masks this leak) ### Proposed fix Make the reader own the decoded vectors. - `ColumnVector` materializes the decoded vector once per batch, instead of allocating on every `getArrowVector()`. - `ArrowBatchReader` keeps a reference to the `ColumnVector`s it hands out and releases their decoded vectors at the start of the next `read()` and in `close()`. - Only vectors allocated by the dictionary decoding are released. This also matches the documented contract ("the arrow vectors are owned by the reader") without requiring callers to close batches. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
