abernardi597 commented on issue #16729: URL: https://github.com/apache/lucene/issues/16729#issuecomment-5953784824
I appreciate you taking a look at this! I looked at your PR but I'm not convinced its the right approach. I had been thinking about this too and came up with these ideas: ### Static work assignment Break the total work into chunks/batches, then assign to each worker thread. This keeps a worker executed on the calling thread from doing all the work, but it still needs to complete its share before the other workers get a chance to start. ### Expose `trySubmit` API Notifying the `TaskExecutor` when there is no space for a worker (instead of executing on calling thread) would allow it to gracefully retry recruiting new workers (e.g. after every batch, exponential back-off, etc.). ### Internally Detect re-entrant workers and handle recruitment The `TaskExecutor` already reserves the calling thread as one of the workers. If a spawned worker detects it is still on the calling thread, it can return without completing any work. By tracking the number of successfully forked workers, the main thread can retry recruitment in between batches (similar to above). This seems like the most promising approach to me: - self-contained and needs no API changes - work assignments remain dynamic (i.e. all workers take from the same queue) - forked workers need no special logic/branching once they get going on a separate thread I could also see a hybrid version where `trySubmit` is implemented by wrapping the `Executor` with the caller-thread detection logic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
