ethbak commented on PR #16555: URL: https://github.com/apache/lucene/pull/16555#issuecomment-5411321135
I don't think this will entirely fix #16552, and it would slow merges doing unnecessary work every time: #### First, it uses `M` at all levels including level 0, but the L0 cap is `2M`. `M` is only the upper-layer cap. This means `newNeighbors.size() < M * DISCONNECTED_NODE_FACTOR` undercounts on L0, which is the level the recorded drop in out-degree is on. If we do keep a cap-based floor, we should use `newNeighbors.maxSize() - 1`, which is M / 2M by level by construction. #### Even with that fixed, a cap-based floor inherently has some issues: By design the diversity heuristic (`diversityCheck`) stops adding neighbors once nothing left is closer to the node than to a neighbor it already picked, so plenty of nodes end up well under the cap. **Example**: full-rebuild, GloVe-100, M=32, 90k docs, zero deletes: ``` Graph level=0 size=90000, Fanout min=1, mean=36.32, max=64 ``` Taking the implementation from the PR, this would flag all nodes with under `64 * 0.85 = 54.4` neighbors, which is the vast majority of the nodes in the healthy graph even before any deletes (mean=36.32) Repair can't do anything for those nodes, since the heuristic just rejects the same candidates again, so they get re-searched and re-flagged every merge and accumulate a lot of wasted work. #### In addition, we should benchmark any solution before thinking of merging I still have the test harness I built to show the initial bug in #16552, so I'm happy to help re-run some tests to show how recall & performance are affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
