jbewing commented on issue #10960: URL: https://github.com/apache/pinot/issues/10960#issuecomment-1604721257
One thing to look out for is that some of the 0.12.0/1 runtime images for pinot are based on an older, deprecated [openjdk:11-jdk-slim docker image](https://hub.docker.com/layers/library/openjdk/11-jdk-slim/images/sha256-d2b6af2093e823ba0cdee4bcd45a905afe3fa054d08bde55b1d850515da69a08?context=explore) which use Java 11.0.16 which has [a C2 compiler bug which results in memory leaks under load](https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/memory-leak-c2-jit-compiler-java-11.0.16). We found that this led to mysterious OOM's in k8s. Since it's a spike under load, the bug is hard to notice via metrics or direct memory profiles. Ironically, there's another java runtime (11.0.18) in that image that you can use at `/usr/lib/jvm/java-11-openjdk-amd64` (just remove the openjdk:11-jdk-slim 11.0.16 java from the PATH and update your JAVA_HOME/JAVA_VERSION to point at the other JDK11. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
