dwsmith1983 commented on PR #5565: URL: https://github.com/apache/datafusion-comet/pull/5565#issuecomment-5571159250
> Haven't looked through the code yet, but this might relate to #5446 and #5008 #5038 moved Arrow's `IpcWriteContext` from per batch to per partition writer; this PR moves it once more, to one per task (`ShuffleCodecContext.arrow_ipc`, borrowed by each `BufBatchWriter` and the spill writer), and puts the outer zstd `CCtx` beside it, which #5446 lists as out of scope. Under Arrow 58.4 the task-level hoist was cosmetic, the context held an empty Vec and a None. Under 59.2 the context owns the `FlatBufferBuilder` and keeps its buffer across writes, so per partition it is one retained builder per partition, ten thousand at the shape in the benchmark, and per task it is one. So the allocation measurement in #5446 would differ between main and this branch; running it on both would show whether the task scope is worth anything on its own. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
