“ exclude on JDK17, include on 11” At this point we do not support that setup, yes
On Thu, 24 Mar 2022 at 12:43, Josh McKenzie <jmcken...@apache.org> wrote: > if we go for Java 17 this means we bump to 5.0 instead of 4.1 at least > because we will remove the already deprecated scripted UDFs and this is > breaking change. > > Ah. I assumed optionality on build (exclude on JDK17, include on 11). If > that's not possible then yeah, 5.0 would make more sense. > > On Thu, Mar 24, 2022, at 12:39 PM, Ekaterina Dimitrova wrote: > > Thank you Josh, I definitely share the excitement but I also want to bring > a few more things. > > " Having a new section in the build.xml for JDK17 runtime env w/more > --add-exports and --add-opens is consistent with how we got jdk11 support > working (and run it to this day looks like). It's worth considering if we > want to move away from this paradigm eventually but if it comes down to > "super experimental JDK17 support in 4.1 vs. tackling this now", I'd > *personally* advocate for the former." > My biggest worry is that the temporary decisions are the most permanent > ones. We shouldn't let it slip in time. It might be a bigger problem in the > long term. > > About the dependencies - my understanding from our latest mail threads is > there is a plan directly to go for beta version in about a month; there is > no concrete plan I've heard of for performance testing for example. Even if > we carefully revise the changelogs I bet there will be things not written > there. Now on the other hand if we go for Java 17 this means we bump to 5.0 > instead of 4.1 at least because we will remove the already deprecated > scripted UDFs and this is breaking change. > > Also, our review cycles of bigger sensitive works tend to be long for a > good reason. I am not even sure who will want to do the reviews. > > Instead of rushing it into this release, do we want instead a feature > branch for now if we want it out there super experimental so people can > have easy access for testing and visibility/awareness? > > > On Wed, Mar 23, 2022 at 3:02 PM Josh McKenzie <jmcken...@apache.org> > wrote: > > > My .02 (Ekaterina and I have been chatting on slack about this a bit): > > Having a new section in the build.xml for JDK17 runtime env w/more > --add-exports and --add-opens is consistent with how we got jdk11 support > working (and run it to this day looks like). It's worth considering if we > want to move away from this paradigm eventually but if it comes down to > "super experimental JDK17 support in 4.1 vs. tackling this now", I'd > *personally* advocate for the former. > > For updating our dependencies, it's going to be a balance between needing > to support newer JDK's (and older JDK's going out of support) and > introducing risk to our runtime env by updating dependencies for JDK's that > aren't yet formally supported. I think we're always going to have tension > where the newest supported JDK is going to introduce dependency upgrade > requirements that will "force" updates to things that the oldest JDK > doesn't strictly need, and in this case it happens to be for a JDK we won't > officially support yet. > > I'd be in favor of the exercise of updating our dependencies as required > to have bleeding edge experimental support for JDK 17 in 4.1 (i.e. "tests > run with it and don't necessarily pass, we don't recommend you use it in > production") with the motion Scott mentioned about changelog validation. > > Having a little more rigor around how and when we do this as a project > (i.e. start validating new JDK at point N in lifecycle of oldest supported > falling out of support, upgrade dependencies to make new JDK M work, > validate those new dependency additions by doing behavior O) would probably > be pretty wise. This is something we've done multiple times in the past and > will need to keep doing into the future. > > Thanks for all the hard work on this Ekaterina! > > ~Josh > > > On Wed, Mar 23, 2022, at 11:31 AM, Ekaterina Dimitrova wrote: > > Thank you for your quick response Scott. > > I agree on all points you made. Also, with our current setup the > dependencies updates would affect the stable Java 11. We cannot afford to > not consider potential changes in behavior, performance, etc > But also we should work on potential blockers and not leave the dependency > to atrophy. > > On Wed, 23 Mar 2022 at 11:07, C. Scott Andreas <sc...@paradoxica.net> > wrote: > > Ekaterina, thank you very much for sharing this! > > I admit, it’s much more involved than I expected to be. > > The —add-opens and —add-exports flags seem suitable for development and > perhaps experimental support, but we’ll probably want to make changes to > remove as many as we can before considering JDK17 suitable for general use. > > Updating dependencies will be good for general hygiene, but it would also > be important for us to scan changelogs for the intervening versions of each > we’re upgrading to flag any unexpected behavior changes that may impact > Cassandra or which the community might want to know about. > > Thank you for working to get Cassandra launching on JDK17. Excited for > what’s ahead. > > — Scott > > On Mar 23, 2022, at 7:58 AM, Ekaterina Dimitrova <e.dimitr...@gmail.com> > wrote: > > > > Hi everyone, > > Looking into our way to Java 17, I wanted to share with the community > findings/thoughts and align on course of action. > > We already deprecated scripted UDFs so we can remove them when the time to > switch from Java8&11 to Java 11&17 comes. I removed the ant script tasks > and created custom ant tasks to workaround the need of Nashorn. Ant is also > upgraded now on trunk. > > With this Cassandra trunk compiles with warnings about the Security > manager being deprecated and other security deprecation warnings(I mention > it for awareness here). > I pushed to my personal docker hub account a version of our testing image > that has Java 17 installed and I worked on the build file, shell scripts > and config to push testing with Java 17 to CircleCI. > > To just start Cassandra out of the box on Java 17 we also need as a least > minimum to add the following, further to what we already open in Java 11: > > *--add-opens java.base/sun.nio.ch <http://sun.nio.ch/>=ALL-UNNAMED* > > *--add-opens java.base/java.io <http://java.io/>=ALL-UNNAMED* > > *--add-opens java.base/java.util.concurrent=ALL-UNNAMED* > > *--add-opens java.base/java.util=ALL-UNNAMED* > > *--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED* > > *--add-opens java.base/java.nio=ALL-UNNAMED* > > *--add-opens java.base/java.lang=ALL-UNNAMED* > > > *--add-exports java.base/sun.nio.ch <http://sun.nio.ch/>=ALL-UNNAMED* > > *--add-exports java.base/java.io <http://java.io/>=ALL-UNNAMED* > > *--add-exports java.base/java.util.concurrent=ALL-UNNAMED* > > *--add-exports java.base/java.util=ALL-UNNAMED* > > *--add-exports java.base/java.util.concurrent.atomic=ALL-UNNAMED* > > *--add-exports java.base/java.nio=ALL-UNNAMED* > > *--add-exports java.base/java.lang=ALL-UNNAMED* > > This is a quick run of *jdeps -jdkinternals --multi-release 17 > apache-cassandra-4.1-SNAPSHOT.jar.*: > > * apache-cassandra-4.1-SNAPSHOT.jar -> java.rmi* > > * apache-cassandra-4.1-SNAPSHOT.jar -> jdk.unsupported* > > * org.apache.cassandra.io.util.Memory -> sun.misc.Unsafe > JDK internal API (jdk.unsupported)* > > * org.apache.cassandra.utils.FastByteOperations$UnsafeOperations -> > sun.misc.Unsafe JDK internal API > (jdk.unsupported)* > > * org.apache.cassandra.utils.FastByteOperations$UnsafeOperations$1 -> > sun.misc.Unsafe JDK internal API > (jdk.unsupported)* > > * org.apache.cassandra.utils.JMXServerUtils$JmxRegistry -> > sun.rmi.registry.RegistryImpl JDK internal API > (java.rmi)* > > * org.apache.cassandra.utils.memory.MemoryUtil -> sun.misc.Unsafe > JDK internal API (jdk.unsupported)* > > > *Warning: JDK internal APIs are unsupported and private to JDK > implementation that are* > > *subject to be removed or changed incompatibly and could break your > application.* > > *Please modify your code to eliminate dependence on any JDK internal APIs.* > *For the most recent update on JDK internal API replacements, please > check: * > > *https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool > <https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool>* > > Also a quick workaround I applied for test purposes in order to be able to > run the in-jvm tests can be seen here[4], and I assume something like that > is also needed for the simulator and other places which I went ahead and > changed in order just to unblock the preliminary testing so we can get the > full picture. To be revised later. > This was the issue we were hitting: > java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers at > org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:691) > at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81) at > org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47) at > org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57) at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) Caused by: > java.lang.NoSuchFieldException: modifiers at > java.base/java.lang.Class.getDeclaredField(Class.java:2610) at > org.apache.cassandra.net.Verb.unsafeSetSerializer(Verb.java:366) at > org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:612) > > To fire testing I used some config suggested by Jonathan Shook which he > used for testing with Java 16 some time ago[1]. We will need to tune it, > those were used to do preliminary work and initial investigations only. > > One important topic I want to bring to the community's attention is our > dependency management. [2] is a discussion we had a long time ago when I > joined the project. So is it correct to say for Java 17 we don't update > anything if there is no immediate need (and this considers only trunk of > course)? In my branch I updated bytebuddy, asm, chronicle queues, byteman, > mockito, jacoco, ecj, sjk so far. > > Based on feedback from CASSANDRA-17392 I tried to update netty and run our > tests with Java 17 and I ran into *java.lang.ClassCastException: class > org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class > sun.nio.ch.DirectBuffer* . I didn't have the chance to investigate it but > it is on our table so good to be mentioned. > > Even with the update of Chronicle queues[3] we still have to add for > FQLtool and AuditLogging: > --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \ > --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \ > --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \ > --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED \ > --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \ > --add-opens=jdk.compiler/com.sun.tools.javac=ALL-UNNAMED \ > --add-opens=java.base/java.lang=ALL-UNNAMED \ > --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \ > --add-opens=java.base/java.io=ALL-UNNAMED \ > --add-opens=java.base/java.util=ALL-UNNAMED\ > After that, from what I saw there is only one type of failure related to > those tools to be investigated that cannot be fixed directly with some > other internals opening. > > I also updated Jolokia in the DTest repo which seemed to solve any Jolokia > related problems, at least our tests are not failing due to Jolokia related > problems from what I saw so far but officially I see Jolokia builds running > on JDK 8, 9, and 11. I didn't find any info around Java 17 and Jolokia so > far. > > These are a few of the things we hit now, so I wanted to mention in order > to give you an idea of the direction the things are going. > > So I am interested to hear any comments, particularly on our approach to > dependencies updates as a project. Also, how much are we willing to extend > the usage of --add-opens and --add-exports at this moment? The > Java community recommendation is this to be our last option. We have some > dependencies which still do not officially support Java 17 or which are not > even fully tested with Java 17. Some of them will work out of the box, some > will need more internals to be opened maybe or some other actions to be > taken, I still don't know the answers to all failures I see and sometimes I > fix one but another one appears. There are moving parts and I also didn't > go ahead and update all dependencies based on the previous discussions. (I > also saw that when Java 11 was introduced we didn't update all > dependencies?) All of this is only to be able to bring our CI to some > stable condition on Java 17 as a starting point and get an idea of what is > needed. > > Also, I guess soon we can join the OpenJDK Quality Outreach program[5] > suggested before as we get more details that can be already discussed. I > can go ahead and do it if no one is against it, the contact can be just our > dev-mailing list I guess. > > [1]https://jaxenter.com/apache-cassandra-java-174575.html > [2]https://lists.apache.org/thread/l1ch2ffcnv2m5l9tp9mpvgzwlvx667sd > [3]https://chronicle.software/chronicle-support-java-17/ > [4] > https://github.com/apache/cassandra/commit/4daf0b149bc1524dfb2212ab958f4d45feb79577 > [5]https://wiki.openjdk.java.net/display/quality/Quality+Outreach > > Ekaterina Dimitrova > > > >
