gortiz commented on issue #10783: URL: https://github.com/apache/pinot/issues/10783#issuecomment-1556883919
> Review the SPI and Unsafe based implementation in https://github.com/apache/pinot/pull/10528 I have some issues there. Some tests always get out of memory in GitHub Actions, while my amd pc and my M1 laptop run them without problems, even when running with `mvn -P github-actions`. Given that I cannot reproduce the problem locally, it is being very difficult to fix the problem. Any help there would be helpful. > Later on we should move to Foreign Memory when stable support is available ? I guess we should decide what we want to do with that. We can either: - Wait until Foreign Memory API is production ready and stay in Java 11 meanwhile. - Versions on which we could run Pinot: [11, 15] and [22, ...). - Assuming that Foreign Memory API is released in Java 22. - Advantage: we will only use proven and stable apis. - Disadvantages: But it implies that we would need to either run with Java 11 until Java 23 or we would need to use non LTS versions. - Support preview of Foreign Memory API: - Versions on which we could run Pinot: [11, 15] and [21, ...). - Advantage: We would support Java 21 if running with --enable-preview - Disadvantage: - If OpenJDK decides to add more breaking changes in Java 22, we would need to maintain different versions of the code. - We would need to run with --preview-enable, which is not intended to be executed in production. - Use our own library on top of Unsafe - Versions on which we could run Pinot: [11, ...). - Advantages: - In Java [11, 15] we can choose to use LArray or Unsafe - In Java [16, 20] we can only use Unsafe - In Java 21 we can use Unsafe or run with --enable-preview and use Foreign Memory API - Once Foreign Memory API is released, we could always use Foreign Memory API - Disadvantages: - We need to maintain the Unsafe library. - Errors there may kill the JVM with segmentation faults. Details may depend on CPU architectures and OS. For example, mmap needs to be page aligned in Linux (otherwise segmentation fault kills the JVM). - Unsafe methods do change between major JVM versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
