msokolov commented on issue #14758: URL: https://github.com/apache/lucene/issues/14758#issuecomment-3606602369
ACORN doesn't prevent from having to evaluate nodes not matching the filter - it merely prioritizes those. So it helps, but it cannot beat the performance of a graph including *only* the filtered nodes, especially if the filter is highly selective. There is no guarantee that filtered nodes will be neighbors of each other - you have to be willing to traverse "dark" areas of the graph to get to the places you want to be. The sorting idea should help cluster filtered nodes together, but that;s not intrinsically part of ACORN, and you know it's unproven and needs further development to see how it plays out. We can miss around with alternative sorting criteria with the codec changes I'm proposing for use with BPV (we don't have to cluster using binary partitioning, but could apply different sorts). I agree the changes being proposed here are somewhat radical. That's why I had earlier proposed using a different approach - namely some kind of "view" or computed field, but I've been convinced that the changes required for that would be even more radical since they would require percolating this notion of field dependencies all the way up in to the public APIs, wheras this change is totally transparent to public APIs and is *merely* a performance optimization implemented in the codec, so free to be changed in the future. Personally I'm convinced that the benefit of precomputing filtered graphs is worth some pain, mostly because I just haven't seen any other approach that comes near to the performance (recall/latency). Not saying we shouldn't try to improve the current situation incrementally, but I also don't think we should necessarily avoid bold innovations if they can really help with something significant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
