I just want to follow-up here to let everyone know I have a PR and it includes a fundamental change in the nature of the Executor to be blocking. Ideally someone would benchmark it. But even if that doesn't happen, I'm inclined to merge to fix the RejectedExecutionException problem. This is an experimental feature, after all.
On Wed, Aug 21, 2024 at 10:12 AM David Smiley <dsmi...@apache.org> wrote: > Thanks; I filed https://issues.apache.org/jira/browse/SOLR-17414 > > On Tue, Aug 20, 2024 at 2:56 PM Ishan Chattopadhyaya > <ichattopadhy...@gmail.com> wrote: > > > > Thanks, I'll take a look. > > > > On Tue, 20 Aug, 2024, 11:22 pm David Smiley, <dsmi...@apache.org> wrote: > > > > > Since the new multiThreaded search feature landed, I see a new test > > > failure involving "RejectedExecutionException" being thrown: > > > > > > > https://ge.apache.org/s/5ack462ji4mlu/tests/task/:solr:core:test/details/org.apache.solr.search.TestRealTimeGet/testStressGetRealtime?top-execution=1 > > > It is thrown at a low level in Lucene building TermStates > > > concurrently. I doubt the problem is specific to that test > > > (TestRealTimeGet) but that test might induce more activity than most > > > tests, thus crossing some thresholds like the queue size -- apparently > > > 1000. > > > > > > ** I don't think we should be throwing a RejectedExecutionException > > > when running a Search query **. > > > Have others explored this topic further and have ideas to share, even > > > including ruled out ideas? Ishan, Noble, ? > > > > > > I'd prefer to avoid catching this exception to take some other action > > > -- would feel too much like a hack. I'm optimistic a more elegant > > > solution without doing this can be found. > > > > > > Perhaps SynchronousQueue? Maybe I can rule this out based on a little > > > test I did, as it throws a RejectedExecutionException (instead of > > > blocking), which I'd rather not catch. > > > > > > Perhaps a LinkedBlockingQueue (thus infinite queue size) but have an > > > ExecutorService subclass that observes the queue size to see if a > > > threshold is reached and if so then runs the task directly? This way > > > we never reject and the caller threads receive back pressure by doing > > > the work that they would have done anyway with multiThreaded=false ! > > > There is plenty of precedent for an ExecutorService that runs the task > > > in the caller thread -- I'm thinking of Lucene's > > > SameThreadExecutorService and Solr's SimpleFacets.directExecutor. > > > > > > ~ David Smiley > > > Apache Lucene/Solr Search Developer > > > http://www.linkedin.com/in/davidwsmiley > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > > > For additional commands, e-mail: dev-h...@solr.apache.org > > > > > > >
