mohitgurav20 commented on PR #25716:
URL: https://github.com/apache/datafusion/pull/25716#issuecomment-6144292607
Thanks for running that @kosiew — that result makes sense to me. TPC-DS at
SF1 is 1GB of data, so everything sits comfortably in memory and the buffer
explosion never really gets a chance to kick in. The fix targets a pretty
specific pattern: multiple chained broadcast joins on the same
StringView/BinaryView column, where each concat pass accumulates duplicate
references to the same underlying buffers. At small scale that overhead is
invisible.
The issue description actually measured this with a single column growing to
1.5M buffer references where only 2 were distinct — that's the real scenario
where you'd see the 7x memory and 10x latency impact. TPC-DS SF1 just doesn't
stress that path.
Happy to put together a focused micro-benchmark that directly reproduces the
exponential growth, or run at SF10/SF100 if you'd prefer a more standard signal.
What makes more sense to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
