benwtrent commented on issue #14208: URL: https://github.com/apache/lucene/issues/14208#issuecomment-2786605051
I was thinking more on this. What if we just wrote the "unoptimized" graph (e.g. the full graph, with static connections) to a temporary file on disk, then used OffHeap graph to build (doing direct writes to offsets, this is tricky I suppose given multi-threadedness, etc). Then once the graph is done being built, we read in the connections, and write the compressed versions of them to the true graph location. This would dramatically reduce the heap usage (to almost nothing), but might hurt performance and I don't know how it will work with parallelism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
