[DISSCUSS] Cassandra and Java 17

[Ekaterina Dimitrova]

Hi everyone,

Looking into our way to Java 17, I wanted to share with the community
findings/thoughts and align on course of action.

We already deprecated scripted UDFs so we can remove them when the time to
switch from Java8&11 to Java 11&17 comes. I removed the ant script tasks
and created custom ant tasks to workaround the need of Nashorn. Ant is also
upgraded now on trunk.

With this Cassandra trunk compiles with warnings about the Security manager
being deprecated and other security deprecation warnings(I mention it for
awareness here).
I pushed to my personal docker hub account a version of our testing image
that has Java 17 installed and I worked on the build file, shell scripts
and config to push testing with Java 17 to CircleCI.

To just start Cassandra out of the box on Java 17 we also need as a least
minimum to add the following, further to what we already open in Java 11:

*--add-opens java.base/sun.nio.ch =ALL-UNNAMED*

*--add-opens java.base/java.io =ALL-UNNAMED*

*--add-opens java.base/java.util.concurrent=ALL-UNNAMED*

*--add-opens java.base/java.util=ALL-UNNAMED*

*--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED*

*--add-opens java.base/java.nio=ALL-UNNAMED*

*--add-opens java.base/java.lang=ALL-UNNAMED*


*--add-exports java.base/sun.nio.ch =ALL-UNNAMED*

*--add-exports java.base/java.io =ALL-UNNAMED*

*--add-exports java.base/java.util.concurrent=ALL-UNNAMED*

*--add-exports java.base/java.util=ALL-UNNAMED*

*--add-exports java.base/java.util.concurrent.atomic=ALL-UNNAMED*

*--add-exports java.base/java.nio=ALL-UNNAMED*

*--add-exports java.base/java.lang=ALL-UNNAMED*

This is a quick run of *jdeps -jdkinternals --multi-release 17
apache-cassandra-4.1-SNAPSHOT.jar.*:

*   apache-cassandra-4.1-SNAPSHOT.jar -> java.rmi*

*   apache-cassandra-4.1-SNAPSHOT.jar -> jdk.unsupported*

*   org.apache.cassandra.io.util.Memory-> sun.misc.Unsafe
  JDK internal API (jdk.unsupported)*

*   org.apache.cassandra.utils.FastByteOperations$UnsafeOperations ->
sun.misc.UnsafeJDK internal API
(jdk.unsupported)*

*   org.apache.cassandra.utils.FastByteOperations$UnsafeOperations$1 ->
sun.misc.UnsafeJDK internal API
(jdk.unsupported)*

*   org.apache.cassandra.utils.JMXServerUtils$JmxRegistry ->
sun.rmi.registry.RegistryImpl  JDK internal API
(java.rmi)*

*   org.apache.cassandra.utils.memory.MemoryUtil   -> sun.misc.Unsafe
  JDK internal API (jdk.unsupported)*


*Warning: JDK internal APIs are unsupported and private to JDK
implementation that are*

*subject to be removed or changed incompatibly and could break your
application.*

*Please modify your code to eliminate dependence on any JDK internal APIs.*
*For the most recent update on JDK internal API replacements, please
check: *

*https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool
*

Also a quick workaround I applied for test purposes in order to be able to
run the in-jvm tests can be seen here[4], and I assume something like that
is also needed for the simulator and other places which I went ahead and
changed in order just to unblock the preliminary testing so we can get the
full picture. To be revised later.
This was the issue we were hitting:
java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers at
org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:691)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81) at
org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47) at
org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833) Caused by:
java.lang.NoSuchFieldException: modifiers at
java.base/java.lang.Class.getDeclaredField(Class.java:2610) at
org.apache.cassandra.net.Verb.unsafeSetSerializer(Verb.java:366) at
org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:612)

To fire testing I used some config suggested by Jonathan Shook which he
used for testing with Java 16 some time ago[1]. We will need to tune it,
those were used to do preliminary work and initial investigations only.

One important topic I want to bring to the community's attention is our
dependency management. [2] is a discussion we had a long time ago when I
joined the project. So is it correct to say for Java 17 we don't update
anything if there is no immediate need (and this considers only trunk of
course)?

Re: [DISSCUSS] Cassandra and Java 17

[C. Scott Andreas]

Ekaterina, thank you very much for sharing this!

I admit, it’s much more involved than I expected to be.

The —add-opens and —add-exports flags seem suitable for development and perhaps 
experimental support, but we’ll probably want to make changes to remove as many 
as we can before considering JDK17 suitable for general use.

Updating dependencies will be good for general hygiene, but it would also be 
important for us to scan changelogs for the intervening versions of each we’re 
upgrading to flag any unexpected behavior changes that may impact Cassandra or 
which the community might want to know about.

Thank you for working to get Cassandra launching on JDK17. Excited for what’s 
ahead.

— Scott

> On Mar 23, 2022, at 7:58 AM, Ekaterina Dimitrova  
> wrote:
> 
> 
> Hi everyone,
> 
> Looking into our way to Java 17, I wanted to share with the community 
> findings/thoughts and align on course of action.
> 
> We already deprecated scripted UDFs so we can remove them when the time to 
> switch from Java8&11 to Java 11&17 comes. I removed the ant script tasks and 
> created custom ant tasks to workaround the need of Nashorn. Ant is also 
> upgraded now on trunk.
> 
> With this Cassandra trunk compiles with warnings about the Security manager 
> being deprecated and other security deprecation warnings(I mention it for 
> awareness here).
> I pushed to my personal docker hub account a version of our testing image 
> that has Java 17 installed and I worked on the build file, shell scripts and 
> config to push testing with Java 17 to CircleCI.
> 
> To just start Cassandra out of the box on Java 17 we also need as a least 
> minimum to add the following, further to what we already open in Java 11:
> --add-opens java.base/sun.nio.ch=ALL-UNNAMED
> --add-opens java.base/java.io=ALL-UNNAMED
> --add-opens java.base/java.util.concurrent=ALL-UNNAMED
> --add-opens java.base/java.util=ALL-UNNAMED
> --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED
> --add-opens java.base/java.nio=ALL-UNNAMED
> --add-opens java.base/java.lang=ALL-UNNAMED
> 
> --add-exports java.base/sun.nio.ch=ALL-UNNAMED
> --add-exports java.base/java.io=ALL-UNNAMED
> --add-exports java.base/java.util.concurrent=ALL-UNNAMED
> --add-exports java.base/java.util=ALL-UNNAMED
> --add-exports java.base/java.util.concurrent.atomic=ALL-UNNAMED
> --add-exports java.base/java.nio=ALL-UNNAMED
> --add-exports java.base/java.lang=ALL-UNNAMED
> 
> This is a quick run of jdeps -jdkinternals --multi-release 17 
> apache-cassandra-4.1-SNAPSHOT.jar.: 
>apache-cassandra-4.1-SNAPSHOT.jar -> java.rmi
>apache-cassandra-4.1-SNAPSHOT.jar -> jdk.unsupported
>org.apache.cassandra.io.util.Memory-> sun.misc.Unsafe  
>   JDK internal API (jdk.unsupported)
>org.apache.cassandra.utils.FastByteOperations$UnsafeOperations -> 
> sun.misc.UnsafeJDK internal API 
> (jdk.unsupported)
>org.apache.cassandra.utils.FastByteOperations$UnsafeOperations$1 -> 
> sun.misc.UnsafeJDK internal API 
> (jdk.unsupported)
>org.apache.cassandra.utils.JMXServerUtils$JmxRegistry -> 
> sun.rmi.registry.RegistryImpl  JDK internal API (java.rmi)
>org.apache.cassandra.utils.memory.MemoryUtil   -> sun.misc.Unsafe  
>   JDK internal API (jdk.unsupported)
> 
> Warning: JDK internal APIs are unsupported and private to JDK implementation 
> that are
> subject to be removed or changed incompatibly and could break your 
> application.
> Please modify your code to eliminate dependence on any JDK internal APIs.
> For the most recent update on JDK internal API replacements, please check: 
> https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool
> 
> Also a quick workaround I applied for test purposes in order to be able to 
> run the in-jvm tests can be seen here[4], and I assume something like that is 
> also needed for the simulator and other places which I went ahead and changed 
> in order just to unblock the preliminary testing so we can get the full 
> picture. To be revised later.
> This was the issue we were hitting:
> java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers
>   at 
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:691)
>   at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81)
>   at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
>   at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
>   at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>   at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>   at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>   at java.base/java.lang.Thread.run(Thread.java:833)
> C

Re: [DISSCUSS] Cassandra and Java 17

[Ekaterina Dimitrova]

Thank you for your quick response Scott.

I agree on all points you made. Also, with our current setup the
dependencies updates would affect the stable Java 11. We cannot afford to
not consider potential changes in behavior, performance, etc
But also we should work on potential blockers and not leave the dependency
to atrophy.

On Wed, 23 Mar 2022 at 11:07, C. Scott Andreas  wrote:

> Ekaterina, thank you very much for sharing this!
>
> I admit, it’s much more involved than I expected to be.
>
> The —add-opens and —add-exports flags seem suitable for development and
> perhaps experimental support, but we’ll probably want to make changes to
> remove as many as we can before considering JDK17 suitable for general use.
>
> Updating dependencies will be good for general hygiene, but it would also
> be important for us to scan changelogs for the intervening versions of each
> we’re upgrading to flag any unexpected behavior changes that may impact
> Cassandra or which the community might want to know about.
>
> Thank you for working to get Cassandra launching on JDK17. Excited for
> what’s ahead.
>
> — Scott
>
> On Mar 23, 2022, at 7:58 AM, Ekaterina Dimitrova 
> wrote:
>
> 
>
> Hi everyone,
>
> Looking into our way to Java 17, I wanted to share with the community
> findings/thoughts and align on course of action.
>
> We already deprecated scripted UDFs so we can remove them when the time to
> switch from Java8&11 to Java 11&17 comes. I removed the ant script tasks
> and created custom ant tasks to workaround the need of Nashorn. Ant is also
> upgraded now on trunk.
>
> With this Cassandra trunk compiles with warnings about the Security
> manager being deprecated and other security deprecation warnings(I mention
> it for awareness here).
> I pushed to my personal docker hub account a version of our testing image
> that has Java 17 installed and I worked on the build file, shell scripts
> and config to push testing with Java 17 to CircleCI.
>
> To just start Cassandra out of the box on Java 17 we also need as a least
> minimum to add the following, further to what we already open in Java 11:
>
> *--add-opens java.base/sun.nio.ch =ALL-UNNAMED*
>
> *--add-opens java.base/java.io =ALL-UNNAMED*
>
> *--add-opens java.base/java.util.concurrent=ALL-UNNAMED*
>
> *--add-opens java.base/java.util=ALL-UNNAMED*
>
> *--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED*
>
> *--add-opens java.base/java.nio=ALL-UNNAMED*
>
> *--add-opens java.base/java.lang=ALL-UNNAMED*
>
>
> *--add-exports java.base/sun.nio.ch =ALL-UNNAMED*
>
> *--add-exports java.base/java.io =ALL-UNNAMED*
>
> *--add-exports java.base/java.util.concurrent=ALL-UNNAMED*
>
> *--add-exports java.base/java.util=ALL-UNNAMED*
>
> *--add-exports java.base/java.util.concurrent.atomic=ALL-UNNAMED*
>
> *--add-exports java.base/java.nio=ALL-UNNAMED*
>
> *--add-exports java.base/java.lang=ALL-UNNAMED*
>
> This is a quick run of *jdeps -jdkinternals --multi-release 17
> apache-cassandra-4.1-SNAPSHOT.jar.*:
>
> *   apache-cassandra-4.1-SNAPSHOT.jar -> java.rmi*
>
> *   apache-cassandra-4.1-SNAPSHOT.jar -> jdk.unsupported*
>
> *   org.apache.cassandra.io.util.Memory-> sun.misc.Unsafe
>   JDK internal API (jdk.unsupported)*
>
> *   org.apache.cassandra.utils.FastByteOperations$UnsafeOperations ->
> sun.misc.UnsafeJDK internal API
> (jdk.unsupported)*
>
> *   org.apache.cassandra.utils.FastByteOperations$UnsafeOperations$1 ->
> sun.misc.UnsafeJDK internal API
> (jdk.unsupported)*
>
> *   org.apache.cassandra.utils.JMXServerUtils$JmxRegistry ->
> sun.rmi.registry.RegistryImpl  JDK internal API
> (java.rmi)*
>
> *   org.apache.cassandra.utils.memory.MemoryUtil   -> sun.misc.Unsafe
>   JDK internal API (jdk.unsupported)*
>
>
> *Warning: JDK internal APIs are unsupported and private to JDK
> implementation that are*
>
> *subject to be removed or changed incompatibly and could break your
> application.*
>
> *Please modify your code to eliminate dependence on any JDK internal APIs.*
> *For the most recent update on JDK internal API replacements, please
> check: *
>
> *https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool
> *
>
> Also a quick workaround I applied for test purposes in order to be able to
> run the in-jvm tests can be seen here[4], and I assume something like that
> is also needed for the simulator and other places which I went ahead and
> changed in order just to unblock the preliminary testing so we can get the
> full picture. To be revised later.
> This was the issue we were hitting:
> java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers at
> org.apache.cassandra.distributed.impl.Instance.lambda$startup$

Re: [DISSCUSS] Cassandra and Java 17

[Josh McKenzie]

My .02 (Ekaterina and I have been chatting on slack about this a bit):

Having a new section in the build.xml for JDK17 runtime env w/more 
--add-exports and --add-opens is consistent with how we got jdk11 support 
working (and run it to this day looks like). It's worth considering if we want 
to move away from this paradigm eventually but if it comes down to "super 
experimental JDK17 support in 4.1 vs. tackling this now", I'd *personally* 
advocate for the former.

For updating our dependencies, it's going to be a balance between needing to 
support newer JDK's (and older JDK's going out of support) and introducing risk 
to our runtime env by updating dependencies for JDK's that aren't yet formally 
supported. I think we're always going to have tension where the newest 
supported JDK is going to introduce dependency upgrade requirements that will 
"force" updates to things that the oldest JDK doesn't strictly need, and in 
this case it happens to be for a JDK we won't officially support yet.

I'd be in favor of the exercise of updating our dependencies as required to 
have bleeding edge experimental support for JDK 17 in 4.1 (i.e. "tests run with 
it and don't necessarily pass, we don't recommend you use it in production") 
with the motion Scott mentioned about changelog validation.

Having a little more rigor around how and when we do this as a project (i.e. 
start validating new JDK at point N in lifecycle of oldest supported falling 
out of support, upgrade dependencies to make new JDK M work, validate those new 
dependency additions by doing behavior O) would probably be pretty wise. This 
is something we've done multiple times in the past and will need to keep doing 
into the future.

Thanks for all the hard work on this Ekaterina!

~Josh
 

On Wed, Mar 23, 2022, at 11:31 AM, Ekaterina Dimitrova wrote:
> Thank you for your quick response Scott. 
> 
> I agree on all points you made. Also, with our current setup the dependencies 
> updates would affect the stable Java 11. We cannot afford to not consider 
> potential changes in behavior, performance, etc 
> But also we should work on potential blockers and not leave the dependency to 
> atrophy. 
> 
> On Wed, 23 Mar 2022 at 11:07, C. Scott Andreas  wrote:
>> Ekaterina, thank you very much for sharing this!
>> 
>> I admit, it’s much more involved than I expected to be.
>> 
>> The —add-opens and —add-exports flags seem suitable for development and 
>> perhaps experimental support, but we’ll probably want to make changes to 
>> remove as many as we can before considering JDK17 suitable for general use.
>> 
>> Updating dependencies will be good for general hygiene, but it would also be 
>> important for us to scan changelogs for the intervening versions of each 
>> we’re upgrading to flag any unexpected behavior changes that may impact 
>> Cassandra or which the community might want to know about.
>> 
>> Thank you for working to get Cassandra launching on JDK17. Excited for 
>> what’s ahead.
>> 
>> — Scott
>> 
>>> On Mar 23, 2022, at 7:58 AM, Ekaterina Dimitrova  
>>> wrote:
>>> 
>>> Hi everyone,
>>> 
>>> Looking into our way to Java 17, I wanted to share with the community 
>>> findings/thoughts and align on course of action.
>>> 
>>> We already deprecated scripted UDFs so we can remove them when the time to 
>>> switch from Java8&11 to Java 11&17 comes. I removed the ant script tasks 
>>> and created custom ant tasks to workaround the need of Nashorn. Ant is also 
>>> upgraded now on trunk.
>>> 
>>> With this Cassandra trunk compiles with warnings about the Security manager 
>>> being deprecated and other security deprecation warnings(I mention it for 
>>> awareness here).
>>> I pushed to my personal docker hub account a version of our testing image 
>>> that has Java 17 installed and I worked on the build file, shell scripts 
>>> and config to push testing with Java 17 to CircleCI.
>>> 
>>> To just start Cassandra out of the box on Java 17 we also need as a least 
>>> minimum to add the following, further to what we already open in Java 11:
>>> *--add-opens java.base/sun.nio.ch=ALL-UNNAMED*
>>> 
>>> *--add-opens java.base/java.io=ALL-UNNAMED*
>>> 
>>> *--add-opens java.base/java.util.concurrent=ALL-UNNAMED*
>>> 
>>> *--add-opens java.base/java.util=ALL-UNNAMED*
>>> 
>>> *--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED*
>>> 
>>> *--add-opens java.base/java.nio=ALL-UNNAMED*
>>> 
>>> *--add-opens java.base/java.lang=ALL-UNNAMED*
>>> 
>>> **
>>> 
>>> *--add-exports java.base/sun.nio.ch=ALL-UNNAMED*
>>> 
>>> *--add-exports java.base/java.io=ALL-UNNAMED*
>>> 
>>> *--add-exports java.base/java.util.concurrent=ALL-UNNAMED*
>>> 
>>> *--add-exports java.base/java.util=ALL-UNNAMED*
>>> 
>>> *--add-exports java.base/java.util.concurrent.atomic=ALL-UNNAMED*
>>> 
>>> *--add-exports java.base/java.nio=ALL-UNNAMED*
>>> 
>>> *--add-exports java.base/java.lang=ALL-UNNAMED*
>>> 
>>> 
>>> This is a quick run of *jdeps -jdkinternals --multi-release 17 
>>> a