tanmay442 commented on PR #26536: URL: https://github.com/apache/datafusion/pull/26536#issuecomment-6310356158
> Yes, I am on it, and the head has been rewritten since your runs, so the rows > in your table belong to an earlier version. > > The change replaces the per-batch scoreboard sweep with a list of the groups > a batch touched. That is the same list the pre-ordered fast path has used > since #25050, so the general path now carries the bookkeeping that path has > had all along. > > The price shows up in one distribution only, the one where every group is > touched by every batch. There the per-batch traffic goes up about 2×, 16 > bytes per group instead of the 8 bytes per group that version read. A batch > is 8,192 rows, so a batch touches at most 8,192 groups, and the touched list > stays at 64 KB or smaller, in cache, next to thousands of row comparisons. > The bound follows the batch rather than the group count, so the cost does not > grow with the size of the group-by. > > The distribution the change targets moves the other way. A million groups > with a batch touching 64 of them went from about 8 MB of traffic per batch to > about 1 KB, roughly 8,000× less. That is the whole reason for the change. > > The same eight cases, with the same two binaries, came out between 1.07% and > 2.45% in the full run and between 0.84% and 3.37% when the runner was asked > for just those eight, so I am not writing those rows off as noise. > > Nobody has a clean measurement of the dense case yet, us included, which is > why the benchmark goes into its own pull request first, #26554. It adds > `update_bench_many_groups_dense`, a dense case at 100,000 groups, so the > number comes from the project runner rather than from my laptop. If the > runner shows a regression on the dense case, I will report it and change the > approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
