I've done a bit of research here, but I am not an expert by any stretch. I added the "add-opens" flag to just the llm module test task, and it seems happy with that. After trying to add this to the Jar Manifest, so that users don't have to do anything (and bin/solr doesn't either), I cannot actually replicate the failure locally. Loading the cohere model, which fails in the tests without the fix, succeeds for me. I've tried with 2 Java 11 JVMs.
After some more searching to see if we do this automatically in bin/solr already, I noticed that the hdfs module already has this code too: // Hadoop mini cluster doesn't handle reflection properly in JDK 16+ > // > https://docs.gradle.org/current/userguide/upgrading_version_7.html#removes_implicit_add_opens_for_test_workers > tasks.withType(Test).configureEach { > jvmArgs(["--add-opens=java.base/java.lang=ALL-UNNAMED", > "--add-opens=java.base/java.util=ALL-UNNAMED"]) > } > > But we don't do anything special for the normal runtime of this module. Just for testing. This leads me to believe that my local tests are correct and for some reason we don't need to do anything outside of the unit tests? - Houston On Fri, Jan 3, 2025 at 7:17 AM Jason Gerlowski <gerlowsk...@gmail.com> wrote: > > And I don't know if there are negative implications with that (and > > where this should happen in the code > > To address the second part of this, unless SolrCLI does something > fancy at start time that I'm not aware of, I think this would have to > be set before the JVM starts up. i.e. in "bin/solr" and/or > "bin/solr.cmd" > > On Fri, Jan 3, 2025 at 7:59 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > > > > I don't follow Java/JVM development very closely, but my understanding > > is that "add-opens" is primarily about encapsulation. Previous > > versions of Java allowed users to access JDK internals via reflection, > > but that capability is disabled by default starting in Java 17 and > > requires the "add-opens" flag to re-enable partially or entirely. > > > > The main negative implication I'm aware of is portability. Internal > > JDK methods may change in the future, or differ across > > implementations, and having Solr rely on those internals (transitively > > through the langchain4j dep/usage) risks future breakage. But that's > > a risk anytime langchain4j is in the picture - the "add-opens" flag > > doesn't make things any better or worse, it just accommodates Java > > 17's firmer stance on encapsulation. IMO "add-opens" is safe to add > > and probably our best option, unless we're willing to patch or replace > > langchain4j altogether. > > > > On Fri, Dec 27, 2024 at 7:18 AM Alessandro Benedetti > > <a.benede...@sease.io> wrote: > > > > > > So, thanks to @sanjayd...@apache.org <sanjayd...@apache.org> , adding: > > > > > > --add-opens=java.base/java.lang.invoke=ALL-UNNAMED > > > > > > in: > > > > > > /solr/gradle/testing/defaults-tests.gradle > > > > > > [propName: 'tests.jvmargs', > > > value: { -> propertyOrEnvOrDefault("tests.jvmargs", "TEST_JVM_ARGS", > > > "-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 > > > -XX:ReservedCodeCacheSize=120m > > > --add-opens=java.base/java.lang.invoke=ALL-UNNAMED") }, > > > description: "Arguments passed to each forked JVM."], > > > > > > Solves the issue. > > > > > > > > > Reading about what it is, it has to do with deep reflection and it > > > makes sense given the mechanism used to instantiate the LLM clients in > > > the code. > > > > > > So to make the module work this should be added as general JVM arg. > > > > > > And I don't know if there are negative implications with that (and > > > where this should happen in the code).-------------------------- > > > > > > *Alessandro Benedetti* > > > Director @ Sease Ltd. > > > *Apache Lucene/Solr Committer* > > > *Apache Solr PMC Member* > > > > > > e-mail: a.benede...@sease.io > > > > > > > > > *Sease* - Information Retrieval Applied > > > Consulting | Training | Open Source > > > > > > Website: Sease.io <http://sease.io/> > > > LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter > > > <https://twitter.com/seaseltd> | Youtube > > > <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github > > > <https://github.com/seaseltd> > > > > > > > > > On Fri, 27 Dec 2024 at 10:57, Alessandro Benedetti < > a.benede...@sease.io> > > > wrote: > > > > > > > Yes Anshum, it's the java version, I just tried switching locally > between > > > > 11 and 21, it succeeds with 21 and fails with 11. > > > > Not sure what to do though, I'll check now! > > > > > > > > Cheers > > > > -------------------------- > > > > *Alessandro Benedetti* > > > > Director @ Sease Ltd. > > > > *Apache Lucene/Solr Committer* > > > > *Apache Solr PMC Member* > > > > > > > > e-mail: a.benede...@sease.io > > > > > > > > > > > > *Sease* - Information Retrieval Applied > > > > Consulting | Training | Open Source > > > > > > > > Website: Sease.io <http://sease.io/> > > > > LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter > > > > <https://twitter.com/seaseltd> | Youtube > > > > <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github > > > > <https://github.com/seaseltd> > > > > > > > > > > > > On Thu, 26 Dec 2024 at 17:22, Anshum Gupta <ans...@anshumgupta.net> > wrote: > > > > > > > >> I haven't spent time looking at this but could it be the Java > version? > > > >> > > > >> On Thu, Dec 26, 2024 at 1:28 PM Alessandro Benedetti < > > > >> a.benede...@sease.io> > > > >> wrote: > > > >> > > > >> > Some additional info: > > > >> > Most tests fail at least because of: > > > >> > > > > >> > Caused by: java.lang.reflect.InaccessibleObjectException: Unable > to make > > > >> > private java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) > > > >> > accessible: module java.base does not "opens java.lang.invoke" to > > > >> unnamed > > > >> > module @774e92eb > > > >> > > > > >> > 2> at > > > >> > > > > >> > > > > >> > java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340) > > > >> > ~[?:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > > > > >> > java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280) > > > >> > ~[?:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > > > > >> > java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:189) > > > >> > ~[?:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:182) > > > >> > ~[?:?] > > > >> > > > > >> > 2> at retrofit2.Platform.<init>(Platform.java:59) > > > >> > ~[retrofit-2.9.0.jar:?] > > > >> > > > > >> > > > > >> > > > > >> > Caused by: java.lang.NoClassDefFoundError: Could not initialize > class > > > >> > retrofit2.Platform > > > >> > > > > >> > 2> at retrofit2.Retrofit$Builder.<init>(Retrofit.java:441) > > > >> > ~[retrofit-2.9.0.jar:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > dev.ai4j.openai4j.DefaultOpenAiClient.<init>(DefaultOpenAiClient.java:91) > > > >> > ~[openai4j-0.22.0.jar:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > dev.ai4j.openai4j.DefaultOpenAiClient.<init>(DefaultOpenAiClient.java:26) > > > >> > ~[openai4j-0.22.0.jar:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > > > > >> > dev.ai4j.openai4j.DefaultOpenAiClient$Builder.build(DefaultOpenAiClient.java:124) > > > >> > ~[openai4j-0.22.0.jar:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > > > > >> > dev.ai4j.openai4j.DefaultOpenAiClient$Builder.build(DefaultOpenAiClient.java:121) > > > >> > ~[openai4j-0.22.0.jar:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > > > > >> > dev.langchain4j.model.openai.OpenAiEmbeddingModel.<init>(OpenAiEmbeddingModel.java:75) > > > >> > ~[langchain4j-open-ai-0.35.0.jar:?] > > > >> > > > > >> > 2> at > > > >> > > > > >> > > > > >> > dev.langchain4j.model.openai.OpenAiEmbeddingModel$OpenAiEmbeddingModelBuilder.build(OpenAiEmbeddingModel.java:40) > > > >> > ~[langchain4j-open-ai-0.35.0.jar:?] > > > >> > > > > >> > 2> ... 55 more > > > >> > > > > >> > > > > >> > Retrofit is used in langChain4j, no idea why it's complaining in > 9.x, > > > >> my 2 > > > >> > cents is that has to do with the gradle backporting, but I have > no clue > > > >> > yet. > > > >> > > > > >> > I'll continue investigating tomorrow, but if anyone has any idea, > feel > > > >> free > > > >> > to tell me! > > > >> > -------------------------- > > > >> > *Alessandro Benedetti* > > > >> > Director @ Sease Ltd. > > > >> > *Apache Lucene/Solr Committer* > > > >> > *Apache Solr PMC Member* > > > >> > > > > >> > e-mail: a.benede...@sease.io > > > >> > > > > >> > > > > >> > *Sease* - Information Retrieval Applied > > > >> > Consulting | Training | Open Source > > > >> > > > > >> > Website: Sease.io <http://sease.io/> > > > >> > LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter > > > >> > <https://twitter.com/seaseltd> | Youtube > > > >> > <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | > Github > > > >> > <https://github.com/seaseltd> > > > >> > > > > >> > > > > >> > On Wed, 25 Dec 2024 at 20:12, Alessandro Benedetti < > > > >> a.benede...@sease.io> > > > >> > wrote: > > > >> > > > > >> > > I've done a first investigation and it seems related to class > loading > > > >> and > > > >> > > related: > > > >> > > > > > >> > > module java.base does not "opens java.lang.invoke" to unnamed > module > > > >> and > > > >> > > classNotFound is one of the inner stacktrace I found, related > with the > > > >> > > retrofit library and reflection. > > > >> > > > > > >> > > I'll keep investigating this, but I'm not able to reproduce all > the 28 > > > >> > > failures locally, the best I was able to do was 7 failures. > > > >> > > Not sure if it could be related to gradle or security policy, it > > > >> smells > > > >> > > like that. > > > >> > > Has anybody ever encountered similar issues? > > > >> > > > > > >> > > Cheers > > > >> > > > > > >> > > Cheers > > > >> > > > > > >> > > -------------------------- > > > >> > > *Alessandro Benedetti* > > > >> > > Director @ Sease Ltd. > > > >> > > *Apache Lucene/Solr Committer* > > > >> > > *Apache Solr PMC Member* > > > >> > > > > > >> > > e-mail: a.benede...@sease.io > > > >> > > > > > >> > > > > > >> > > *Sease* - Information Retrieval Applied > > > >> > > Consulting | Training | Open Source > > > >> > > > > > >> > > Website: Sease.io <http://sease.io/> > > > >> > > LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter > > > >> > > <https://twitter.com/seaseltd> | Youtube > > > >> > > <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | > Github > > > >> > > <https://github.com/seaseltd> > > > >> > > > > > >> > > > > > >> > > On Tue, 24 Dec 2024 at 18:53, Alessandro Benedetti < > > > >> a.benede...@sease.io > > > >> > > > > > >> > > wrote: > > > >> > > > > > >> > >> It's definitely the LLM stuff I added! > > > >> > >> Something didn't work, just run them locally and they are > green so I > > > >> > >> suspect I'll need more time investigating what's going on. > > > >> > >> I can take a look on the 27th! > > > >> > >> > > > >> > >> I take the occasion to wish happy festivities to everybody! > > > >> > >> > > > >> > >> > > > >> > >> -------------------------- > > > >> > >> *Alessandro Benedetti* > > > >> > >> Director @ Sease Ltd. > > > >> > >> *Apache Lucene/Solr Committer* > > > >> > >> *Apache Solr PMC Member* > > > >> > >> > > > >> > >> e-mail: a.benede...@sease.io > > > >> > >> > > > >> > >> > > > >> > >> *Sease* - Information Retrieval Applied > > > >> > >> Consulting | Training | Open Source > > > >> > >> > > > >> > >> Website: Sease.io <http://sease.io/> > > > >> > >> LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter > > > >> > >> <https://twitter.com/seaseltd> | Youtube > > > >> > >> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | > Github > > > >> > >> <https://github.com/seaseltd> > > > >> > >> > > > >> > >> > > > >> > >> On Mon, 23 Dec 2024 at 14:17, Anshum Gupta < > ans...@anshumgupta.net> > > > >> > >> wrote: > > > >> > >> > > > >> > >>> The following tests are consistently failing for me. > > > >> > >>> > > > >> > >>> - TestModelManagerPersistence.testModelAreStoredCompact > > > >> > >>> - TestModelManagerPersistence.testModelStorePersistence > > > >> > >>> - TestModelManager.loadModel_openAi_shouldLoadModelConfig > > > >> > >>> - TestModelManager.loadModel_huggingface_shouldLoadModelConfig > > > >> > >>> - TestModelManager.loadModel_cohere_shouldLoadModelConfig > > > >> > >>> - TestModelManager.testRestManagerEndpoints > > > >> > >>> - TestModelManager.loadModel_mistralAi_shouldLoadModelConfig > > > >> > >>> > > > >> > >>> On looking at the recent runs, it seems like it's been > failing on > > > >> 9x > > > >> > and > > > >> > >>> 9.8. Can someone help out with this? > > > >> > >>> > > > >> > >>> On Mon, Dec 23, 2024 at 3:37 AM Anshum Gupta < > > > >> ans...@anshumgupta.net> > > > >> > >>> wrote: > > > >> > >>> > > > >> > >>> > Thank you everyone for keeping the momentum while I was > away the > > > >> last > > > >> > >>> few > > > >> > >>> > days. I plan to build the RC tomorrow. > > > >> > >>> > > > > >> > >>> > On Sat, Dec 21, 2024 at 1:33 AM Alessandro Benedetti < > > > >> > >>> a.benede...@sease.io> > > > >> > >>> > wrote: > > > >> > >>> > > > > >> > >>> >> Cherry-picked to 9.8! > > > >> > >>> >> We should be good to go from my side, thanks everyone! > > > >> > >>> >> -------------------------- > > > >> > >>> >> *Alessandro Benedetti* > > > >> > >>> >> Director @ Sease Ltd. > > > >> > >>> >> *Apache Lucene/Solr Committer* > > > >> > >>> >> *Apache Solr PMC Member* > > > >> > >>> >> > > > >> > >>> >> e-mail: a.benede...@sease.io > > > >> > >>> >> > > > >> > >>> >> > > > >> > >>> >> *Sease* - Information Retrieval Applied > > > >> > >>> >> Consulting | Training | Open Source > > > >> > >>> >> > > > >> > >>> >> Website: Sease.io <http://sease.io/> > > > >> > >>> >> LinkedIn <https://linkedin.com/company/sease-ltd> | > Twitter > > > >> > >>> >> <https://twitter.com/seaseltd> | Youtube > > > >> > >>> >> <https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> > | > > > >> Github > > > >> > >>> >> <https://github.com/seaseltd> > > > >> > >>> >> > > > >> > >>> >> > > > >> > >>> >> On Fri, 20 Dec 2024 at 17:05, Alessandro Benedetti < > > > >> > >>> a.benede...@sease.io> > > > >> > >>> >> wrote: > > > >> > >>> >> > > > >> > >>> >> > So, I merged to 10.x and backported to 9.x (thanks > Christos for > > > >> > the > > > >> > >>> >> review > > > >> > >>> >> > of gradle stuff revert). > > > >> > >>> >> > > > > >> > >>> >> > I'm proceeding with 9.8 branch now > > > >> > >>> >> > -------------------------- > > > >> > >>> >> > *Alessandro Benedetti* > > > >> > >>> >> > Director @ Sease Ltd. > > > >> > >>> >> > *Apache Lucene/Solr Committer* > > > >> > >>> >> > *Apache Solr PMC Member* > > > >> > >>> >> > > > > >> > >>> >> > e-mail: a.benede...@sease.io > > > >> > >>> >> > > > > >> > >>> >> > > > > >> > >>> >> > *Sease* - Information Retrieval Applied > > > >> > >>> >> > Consulting | Training | Open Source > > > >> > >>> >> > > > > >> > >>> >> > Website: Sease.io <http://sease.io/> > > > >> > >>> >> > LinkedIn <https://linkedin.com/company/sease-ltd> | > Twitter > > > >> > >>> >> > <https://twitter.com/seaseltd> | Youtube > > > >> > >>> >> > < > https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | > > > >> > Github > > > >> > >>> >> > <https://github.com/seaseltd> > > > >> > >>> >> > > > > >> > >>> >> > > > > >> > >>> >> > On Fri, 20 Dec 2024 at 11:11, Alessandro Benedetti < > > > >> > >>> >> a.benede...@sease.io> > > > >> > >>> >> > wrote: > > > >> > >>> >> > > > > >> > >>> >> >> P.S. we can always change the CHANGES.txt later if > necessary > > > >> > >>> >> >> -------------------------- > > > >> > >>> >> >> *Alessandro Benedetti* > > > >> > >>> >> >> Director @ Sease Ltd. > > > >> > >>> >> >> *Apache Lucene/Solr Committer* > > > >> > >>> >> >> *Apache Solr PMC Member* > > > >> > >>> >> >> > > > >> > >>> >> >> e-mail: a.benede...@sease.io > > > >> > >>> >> >> > > > >> > >>> >> >> > > > >> > >>> >> >> *Sease* - Information Retrieval Applied > > > >> > >>> >> >> Consulting | Training | Open Source > > > >> > >>> >> >> > > > >> > >>> >> >> Website: Sease.io <http://sease.io/> > > > >> > >>> >> >> LinkedIn <https://linkedin.com/company/sease-ltd> | > Twitter > > > >> > >>> >> >> <https://twitter.com/seaseltd> | Youtube > > > >> > >>> >> >> < > https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | > > > >> > >>> Github > > > >> > >>> >> >> <https://github.com/seaseltd> > > > >> > >>> >> >> > > > >> > >>> >> >> > > > >> > >>> >> >> On Fri, 20 Dec 2024 at 11:10, Alessandro Benedetti < > > > >> > >>> >> a.benede...@sease.io> > > > >> > >>> >> >> wrote: > > > >> > >>> >> >> > > > >> > >>> >> >>> Ok, I'll proceed merging now to 10.x, I'll align with > > > >> Christos > > > >> > >>> for the > > > >> > >>> >> >>> backporting as there's gradle stuff to do, I am not > super > > > >> > >>> confident > > > >> > >>> >> doing > > > >> > >>> >> >>> it alone, > > > >> > >>> >> >>> Cheers > > > >> > >>> >> >>> -------------------------- > > > >> > >>> >> >>> *Alessandro Benedetti* > > > >> > >>> >> >>> Director @ Sease Ltd. > > > >> > >>> >> >>> *Apache Lucene/Solr Committer* > > > >> > >>> >> >>> *Apache Solr PMC Member* > > > >> > >>> >> >>> > > > >> > >>> >> >>> e-mail: a.benede...@sease.io > > > >> > >>> >> >>> > > > >> > >>> >> >>> > > > >> > >>> >> >>> *Sease* - Information Retrieval Applied > > > >> > >>> >> >>> Consulting | Training | Open Source > > > >> > >>> >> >>> > > > >> > >>> >> >>> Website: Sease.io <http://sease.io/> > > > >> > >>> >> >>> LinkedIn <https://linkedin.com/company/sease-ltd> | > Twitter > > > >> > >>> >> >>> <https://twitter.com/seaseltd> | Youtube > > > >> > >>> >> >>> < > https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | > > > >> > >>> Github > > > >> > >>> >> >>> <https://github.com/seaseltd> > > > >> > >>> >> >>> > > > >> > >>> >> >>> > > > >> > >>> >> >>> On Thu, 19 Dec 2024 at 15:06, Houston Putman < > > > >> > >>> houstonput...@gmail.com > > > >> > >>> >> > > > > >> > >>> >> >>> wrote: > > > >> > >>> >> >>> > > > >> > >>> >> >>>> I say we just merge it. It's safe enough and we have > enough > > > >> > >>> people in > > > >> > >>> >> >>>> agreement. > > > >> > >>> >> >>>> > > > >> > >>> >> >>>> - Houston > > > >> > >>> >> >>>> > > > >> > >>> >> >>>> On Thu, Dec 19, 2024 at 2:51 PM Jason Gerlowski < > > > >> > >>> >> gerlowsk...@gmail.com> > > > >> > >>> >> >>>> wrote: > > > >> > >>> >> >>>> > > > >> > >>> >> >>>> > Yeah, Anshum is on vacation, but he also mentioned > > > >> wanting to > > > >> > >>> keep > > > >> > >>> >> >>>> > momentum on the release - so it wouldn't surprise > me to > > > >> see > > > >> > >>> him pop > > > >> > >>> >> >>>> > back online. > > > >> > >>> >> >>>> > > > > >> > >>> >> >>>> > I think a "best judgement" approach is fine here: > > > >> optionally > > > >> > >>> wait > > > >> > >>> >> >>>> > another day, but then feel free to target 9.8 if > you don't > > > >> > hear > > > >> > >>> >> >>>> > anything. > > > >> > >>> >> >>>> > > > > >> > >>> >> >>>> > Best, > > > >> > >>> >> >>>> > > > > >> > >>> >> >>>> > Jason > > > >> > >>> >> >>>> > > > > >> > >>> >> >>>> > On Thu, Dec 19, 2024 at 8:57 AM David Smiley < > > > >> > >>> dsmi...@apache.org> > > > >> > >>> >> >>>> wrote: > > > >> > >>> >> >>>> > > > > > >> > >>> >> >>>> > > Jason; at the meetup, you mentioned Anshum was on > > > >> vacation. > > > >> > >>> >> Would > > > >> > >>> >> >>>> it > > > >> > >>> >> >>>> > make > > > >> > >>> >> >>>> > > sense for us to make decisions (using our best > > > >> judgement) > > > >> > >>> >> instead of > > > >> > >>> >> >>>> > > waiting on a response? > > > >> > >>> >> >>>> > > > > >> > >>> >> >>>> > > > > >> > >>> >> > > > >> > > --------------------------------------------------------------------- > > > >> > >>> >> >>>> > To unsubscribe, e-mail: > dev-unsubscr...@solr.apache.org > > > >> > >>> >> >>>> > For additional commands, e-mail: > dev-h...@solr.apache.org > > > >> > >>> >> >>>> > > > > >> > >>> >> >>>> > > > > >> > >>> >> >>>> > > > >> > >>> >> >>> > > > >> > >>> >> > > > >> > >>> > > > > >> > >>> > > > > >> > >>> > -- > > > >> > >>> > Anshum Gupta > > > >> > >>> > > > > >> > >>> > > > >> > >>> > > > >> > >>> -- > > > >> > >>> Anshum Gupta > > > >> > >>> > > > >> > >> > > > >> > > > > >> > > > >> > > > >> -- > > > >> Anshum Gupta > > > >> > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org > >
