andygrove commented on PR #6371: URL: https://github.com/apache/datafusion-comet/pull/6371#issuecomment-6149002206
Thanks @parthchandra, all four are addressed in 1c8f68c048. The leak: `getBatchFieldVectorsWithProviders` now closes the 32-bit copies and the materialized constants it already made when a later column fails, as in the thread with sunchao, and a new test checks the allocator after such a failure. Nested children: I filed #6933, and the `narrowOffsets` scaladoc points at it. A throwaway probe on `main` shows the same drift one level down. A broadcast that mixes `struct<c: large_string>` and `struct<c: string>` batches is left uncoalesced, and the native input stream then loads the second batch at the first one's width, which either fails or reads wrong values. The validity copy: nothing can hand it a bitmap that is shifted against the values. An Arrow Java vector has no array offset, so validity bit i always belongs to value i, whatever offset value i's data starts at, and every reader of the vector relies on that, `CometPlainVector` included. The only way to get a shifted bitmap is a C Data import with a non-zero `offset`, which Arrow Java ignores (apache/arrow-java#88). arrow-rs exports a sliced string or binary array with offset 0 and its bitmap realigned to the first value, leaving only the offsets starting past zero, which is the case the rebasing handles. There's a comment above the copy now. The transient peak: yes, that's understood and acceptable. The copy holds only the bytes the column's values cover, at most 2 GiB in one batch, and is released when `serializeBatches` clears its root, the same as a materialized `ConstantColumnVector` on this path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
