> if we go for Java 17 this means we bump to 5.0 instead of 4.1 at least > because we will remove the already deprecated scripted UDFs and this is > breaking change. Ah. I assumed optionality on build (exclude on JDK17, include on 11). If that's not possible then yeah, 5.0 would make more sense.
On Thu, Mar 24, 2022, at 12:39 PM, Ekaterina Dimitrova wrote: > Thank you Josh, I definitely share the excitement but I also want to bring a > few more things. > > " Having a new section in the build.xml for JDK17 runtime env w/more > --add-exports and --add-opens is consistent with how we got jdk11 support > working (and run it to this day looks like). It's worth considering if we > want to move away from this paradigm eventually but if it comes down to > "super experimental JDK17 support in 4.1 vs. tackling this now", I'd > *personally* advocate for the former." > My biggest worry is that the temporary decisions are the most permanent ones. > We shouldn't let it slip in time. It might be a bigger problem in the long > term. > > About the dependencies - my understanding from our latest mail threads is > there is a plan directly to go for beta version in about a month; there is no > concrete plan I've heard of for performance testing for example. Even if we > carefully revise the changelogs I bet there will be things not written there. > Now on the other hand if we go for Java 17 this means we bump to 5.0 instead > of 4.1 at least because we will remove the already deprecated scripted UDFs > and this is breaking change. > > Also, our review cycles of bigger sensitive works tend to be long for a good > reason. I am not even sure who will want to do the reviews. > > Instead of rushing it into this release, do we want instead a feature branch > for now if we want it out there super experimental so people can have easy > access for testing and visibility/awareness? > > > On Wed, Mar 23, 2022 at 3:02 PM Josh McKenzie <jmcken...@apache.org> wrote: >> __ >> My .02 (Ekaterina and I have been chatting on slack about this a bit): >> >> Having a new section in the build.xml for JDK17 runtime env w/more >> --add-exports and --add-opens is consistent with how we got jdk11 support >> working (and run it to this day looks like). It's worth considering if we >> want to move away from this paradigm eventually but if it comes down to >> "super experimental JDK17 support in 4.1 vs. tackling this now", I'd >> *personally* advocate for the former. >> >> For updating our dependencies, it's going to be a balance between needing to >> support newer JDK's (and older JDK's going out of support) and introducing >> risk to our runtime env by updating dependencies for JDK's that aren't yet >> formally supported. I think we're always going to have tension where the >> newest supported JDK is going to introduce dependency upgrade requirements >> that will "force" updates to things that the oldest JDK doesn't strictly >> need, and in this case it happens to be for a JDK we won't officially >> support yet. >> >> I'd be in favor of the exercise of updating our dependencies as required to >> have bleeding edge experimental support for JDK 17 in 4.1 (i.e. "tests run >> with it and don't necessarily pass, we don't recommend you use it in >> production") with the motion Scott mentioned about changelog validation. >> >> Having a little more rigor around how and when we do this as a project (i.e. >> start validating new JDK at point N in lifecycle of oldest supported falling >> out of support, upgrade dependencies to make new JDK M work, validate those >> new dependency additions by doing behavior O) would probably be pretty wise. >> This is something we've done multiple times in the past and will need to >> keep doing into the future. >> >> Thanks for all the hard work on this Ekaterina! >> >> ~Josh >> >> >> On Wed, Mar 23, 2022, at 11:31 AM, Ekaterina Dimitrova wrote: >>> Thank you for your quick response Scott. >>> >>> I agree on all points you made. Also, with our current setup the >>> dependencies updates would affect the stable Java 11. We cannot afford to >>> not consider potential changes in behavior, performance, etc >>> But also we should work on potential blockers and not leave the dependency >>> to atrophy. >>> >>> On Wed, 23 Mar 2022 at 11:07, C. Scott Andreas <sc...@paradoxica.net> wrote: >>>> Ekaterina, thank you very much for sharing this! >>>> >>>> I admit, it’s much more involved than I expected to be. >>>> >>>> The —add-opens and —add-exports flags seem suitable for development and >>>> perhaps experimental support, but we’ll probably want to make changes to >>>> remove as many as we can before considering JDK17 suitable for general use. >>>> >>>> Updating dependencies will be good for general hygiene, but it would also >>>> be important for us to scan changelogs for the intervening versions of >>>> each we’re upgrading to flag any unexpected behavior changes that may >>>> impact Cassandra or which the community might want to know about. >>>> >>>> Thank you for working to get Cassandra launching on JDK17. Excited for >>>> what’s ahead. >>>> >>>> — Scott >>>> >>>>> On Mar 23, 2022, at 7:58 AM, Ekaterina Dimitrova <e.dimitr...@gmail.com> >>>>> wrote: >>>>> >>>>> Hi everyone, >>>>> >>>>> Looking into our way to Java 17, I wanted to share with the community >>>>> findings/thoughts and align on course of action. >>>>> >>>>> We already deprecated scripted UDFs so we can remove them when the time >>>>> to switch from Java8&11 to Java 11&17 comes. I removed the ant script >>>>> tasks and created custom ant tasks to workaround the need of Nashorn. Ant >>>>> is also upgraded now on trunk. >>>>> >>>>> With this Cassandra trunk compiles with warnings about the Security >>>>> manager being deprecated and other security deprecation warnings(I >>>>> mention it for awareness here). >>>>> I pushed to my personal docker hub account a version of our testing image >>>>> that has Java 17 installed and I worked on the build file, shell scripts >>>>> and config to push testing with Java 17 to CircleCI. >>>>> >>>>> To just start Cassandra out of the box on Java 17 we also need as a least >>>>> minimum to add the following, further to what we already open in Java 11: >>>>> *--add-opens java.base/sun.nio.ch=ALL-UNNAMED* >>>>> >>>>> *--add-opens java.base/java.io=ALL-UNNAMED* >>>>> >>>>> *--add-opens java.base/java.util.concurrent=ALL-UNNAMED* >>>>> >>>>> *--add-opens java.base/java.util=ALL-UNNAMED* >>>>> >>>>> *--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED* >>>>> >>>>> *--add-opens java.base/java.nio=ALL-UNNAMED* >>>>> >>>>> *--add-opens java.base/java.lang=ALL-UNNAMED* >>>>> >>>>> ** >>>>> >>>>> *--add-exports java.base/sun.nio.ch=ALL-UNNAMED* >>>>> >>>>> *--add-exports java.base/java.io=ALL-UNNAMED* >>>>> >>>>> *--add-exports java.base/java.util.concurrent=ALL-UNNAMED* >>>>> >>>>> *--add-exports java.base/java.util=ALL-UNNAMED* >>>>> >>>>> *--add-exports java.base/java.util.concurrent.atomic=ALL-UNNAMED* >>>>> >>>>> *--add-exports java.base/java.nio=ALL-UNNAMED* >>>>> >>>>> *--add-exports java.base/java.lang=ALL-UNNAMED* >>>>> >>>>> >>>>> This is a quick run of *jdeps -jdkinternals --multi-release 17 >>>>> apache-cassandra-4.1-SNAPSHOT.jar.*: >>>>> * apache-cassandra-4.1-SNAPSHOT.jar -> java.rmi* >>>>> >>>>> * apache-cassandra-4.1-SNAPSHOT.jar -> jdk.unsupported* >>>>> >>>>> * org.apache.cassandra.io.util.Memory -> sun.misc.Unsafe >>>>> JDK internal API (jdk.unsupported)* >>>>> >>>>> * org.apache.cassandra.utils.FastByteOperations$UnsafeOperations -> >>>>> sun.misc.Unsafe JDK internal API >>>>> (jdk.unsupported)* >>>>> >>>>> * org.apache.cassandra.utils.FastByteOperations$UnsafeOperations$1 -> >>>>> sun.misc.Unsafe JDK internal API >>>>> (jdk.unsupported)* >>>>> >>>>> * org.apache.cassandra.utils.JMXServerUtils$JmxRegistry -> >>>>> sun.rmi.registry.RegistryImpl JDK internal API >>>>> (java.rmi)* >>>>> >>>>> * org.apache.cassandra.utils.memory.MemoryUtil -> sun.misc.Unsafe >>>>> JDK internal API (jdk.unsupported)* >>>>> >>>>> ** >>>>> >>>>> *Warning: JDK internal APIs are unsupported and private to JDK >>>>> implementation that are* >>>>> >>>>> *subject to be removed or changed incompatibly and could break your >>>>> application.* >>>>> >>>>> *Please modify your code to eliminate dependence on any JDK internal >>>>> APIs.* >>>>> >>>>> *For the most recent update on JDK internal API replacements, please >>>>> check: * >>>>> *https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool* >>>>> >>>>> >>>>> Also a quick workaround I applied for test purposes in order to be able >>>>> to run the in-jvm tests can be seen here[4], and I assume something like >>>>> that is also needed for the simulator and other places which I went ahead >>>>> and changed in order just to unblock the preliminary testing so we can >>>>> get the full picture. To be revised later. >>>>> This was the issue we were hitting: >>>>> java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers at >>>>> org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:691) >>>>> at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81) >>>>> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47) at >>>>> org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57) at >>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >>>>> at >>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>>>> at >>>>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >>>>> at java.base/java.lang.Thread.run(Thread.java:833) Caused by: >>>>> java.lang.NoSuchFieldException: modifiers at >>>>> java.base/java.lang.Class.getDeclaredField(Class.java:2610) at >>>>> org.apache.cassandra.net.Verb.unsafeSetSerializer(Verb.java:366) at >>>>> org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:612) >>>>> >>>>> To fire testing I used some config suggested by Jonathan Shook which he >>>>> used for testing with Java 16 some time ago[1]. We will need to tune it, >>>>> those were used to do preliminary work and initial investigations only. >>>>> >>>>> One important topic I want to bring to the community's attention is our >>>>> dependency management. [2] is a discussion we had a long time ago when I >>>>> joined the project. So is it correct to say for Java 17 we don't update >>>>> anything if there is no immediate need (and this considers only trunk of >>>>> course)? In my branch I updated bytebuddy, asm, chronicle queues, >>>>> byteman, mockito, jacoco, ecj, sjk so far. >>>>> >>>>> Based on feedback from CASSANDRA-17392 I tried to update netty and run >>>>> our tests with Java 17 and I ran into *java.lang.ClassCastException: >>>>> class org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast >>>>> to class sun.nio.ch.DirectBuffer* . I didn't have the chance to >>>>> investigate it but it is on our table so good to be mentioned. >>>>> >>>>> Even with the update of Chronicle queues[3] we still have to add for >>>>> FQLtool and AuditLogging: >>>>> --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \ >>>>> --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \ >>>>> --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \ >>>>> --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED \ >>>>> --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \ >>>>> --add-opens=jdk.compiler/com.sun.tools.javac=ALL-UNNAMED \ >>>>> --add-opens=java.base/java.lang=ALL-UNNAMED \ >>>>> --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \ >>>>> --add-opens=java.base/java.io=ALL-UNNAMED \ >>>>> --add-opens=java.base/java.util=ALL-UNNAMED\ >>>>> After that, from what I saw there is only one type of failure related to >>>>> those tools to be investigated that cannot be fixed directly with some >>>>> other internals opening. >>>>> >>>>> I also updated Jolokia in the DTest repo which seemed to solve any >>>>> Jolokia related problems, at least our tests are not failing due to >>>>> Jolokia related problems from what I saw so far but officially I see >>>>> Jolokia builds running on JDK 8, 9, and 11. I didn't find any info around >>>>> Java 17 and Jolokia so far. >>>>> >>>>> These are a few of the things we hit now, so I wanted to mention in order >>>>> to give you an idea of the direction the things are going. >>>>> >>>>> So I am interested to hear any comments, particularly on our approach to >>>>> dependencies updates as a project. Also, how much are we willing to >>>>> extend the usage of --add-opens and --add-exports at this moment? The >>>>> Java community recommendation is this to be our last option. We have some >>>>> dependencies which still do not officially support Java 17 or which are >>>>> not even fully tested with Java 17. Some of them will work out of the >>>>> box, some will need more internals to be opened maybe or some other >>>>> actions to be taken, I still don't know the answers to all failures I see >>>>> and sometimes I fix one but another one appears. There are moving parts >>>>> and I also didn't go ahead and update all dependencies based on the >>>>> previous discussions. (I also saw that when Java 11 was introduced we >>>>> didn't update all dependencies?) All of this is only to be able to bring >>>>> our CI to some stable condition on Java 17 as a starting point and get an >>>>> idea of what is needed. >>>>> >>>>> Also, I guess soon we can join the OpenJDK Quality Outreach program[5] >>>>> suggested before as we get more details that can be already discussed. I >>>>> can go ahead and do it if no one is against it, the contact can be just >>>>> our dev-mailing list I guess. >>>>> >>>>> [1]https://jaxenter.com/apache-cassandra-java-174575.html >>>>> [2]https://lists.apache.org/thread/l1ch2ffcnv2m5l9tp9mpvgzwlvx667sd >>>>> [3]https://chronicle.software/chronicle-support-java-17/ >>>>> [4]https://github.com/apache/cassandra/commit/4daf0b149bc1524dfb2212ab958f4d45feb79577 >>>>> [5]https://wiki.openjdk.java.net/display/quality/Quality+Outreach >>>>> >>>>> Ekaterina Dimitrova >>
