LantaoJin commented on PR #16794: URL: https://github.com/apache/lucene/pull/16794#issuecomment-6162006359
> My guess is why `perGraph` option did not perform well is because you had 4 segments with 8 workers which is not typical. I would expect at least 8 segments to be merged. > > But I am happy to have the current design implemented in this PR, may be we can just add TODO// experiment with different concurrency model: a worker per graph I checked with cohere-v3 1024d vectors, 1M docs total, segments of exactly 25000, built in parallel and combined with `addIndexes` so nothing merges before the measured one. 8 workers, merger level, one run each. `perGraph` has one task per joined graph, largest first: | layout | joined graphs | old | phased (PR) | perGraph | overlap | |----------|---------------|-----------------|----------------|----------------|----------------| | 9 x 25k | 8 | 53.90 s, 7.63x | 37.70 s, 7.75x | 39.76 s, 7.70x | 36.93 s, 7.86x | | 10 x 25k | 9 | 61.06 s, 7.77x | 42.12 s, 7.77x | 73.86 s, 4.63x | 41.49 s, 7.87x | | 16 x 25k | 15 | 103.90 s, 7.90x | 74.11 s, 7.76x | 79.66 s, 7.41x | 72.67 s, 7.90x | So `perGraph` is 5% slower than `phased` even in its best case (8 equal graphs, matching the 8 workers), and once there are slightly more graphs than workers it falls off a cliff: with 9 joined graphs most workers finish quickly and then one worker joins the last graph alone, taking 1.75x longer than `phased` and longer than the old code. That is also what TieredMergePolicy's default of 10 segments per merge gives. Given this I dropped the TODO. Happy to keep it if you'd rather, or we can revisit if a different graph-level model comes up. Recall at ef=100 is within about 0.003 for `phased`, `perGraph` and `overlap` in every layout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
