uschindler opened a new pull request, #12873: URL: https://github.com/apache/lucene/pull/12873
This PR rewrites the JavascriptCompiler for the main branch to use more modern JVM features to be easier configurable and does not need a separate classloader per expression (uses hidden classes). Hidden classes are a new feature of Java 15: It allows to define a hidden class with byte code without using a classloader using the access constraints of a given private lookup. This is the same technique used by lambdas and other invokeDynamic features and replaces the old Unsafe API that Lucene never used. The downsides with a separate on-class classloader are described in #8933. Mainly with many expressions compiled in parallel there's a lock in parent classloader while looking up methods/fields and loading classes. The new Java 15 hidden classes are unloaded by the JVM if the get out of scope, so the workaround with a separate classloader is no longer needed. A second reason why separate classloaders were needed is to allow code access functions form other classloaders. This stopped me doing the whole optimization at first until I rewrote the whole function lookup to be more modern: - The function array is now a `Map<String, MethodHandle>`. This allows to create arbitrary functions anyhere in code like Elasticsearch/Opensearch (the code may even be private or in another classloader). The access to those method is no longer the deal of the compiler, if the MethodHandle is there, it can be used. Access is the deal of the user of the expressions API. In addition you can also use "synthetic" methodhandles like those generated by `MethodHandles#filterArguments` or you can use non-static methods as long as you bind the receiver to a fixed instance. Some examples were added to the tests. - The problem with MethodHandles is that they should be `static final` constants in the class or must appear in the constant pool of the class. This is now the second trick used here: Since Java 11 there is the feature of dynamic constants: Basically the LDC instruction can load a MethodHandle constant (which is treated as the "perfect" MethodHandle for Hotspot) using a bootstrap method (invokedynamic v2). The Javascript compiler now implements a bootstrap method that looks up the function dynamically from the above `Map<String,MethodHandle>`. - Java 16 brought another feature that allows to add `classData` to hidden classes (Java 15). This classData is saved in the private bootstrapper `Lookup` and is accessible during bootstrapping constants. When loading the hidden class we pass our function map as classdata to the lookup and this allows the constant bootstrapping to return the MethodHandle. - I have to do more benchmarking, but generally the dynamic constants and using LDC and `MethodHandle.invokeExact()` should be enough to call ALL functions. For now I added a shortcut: If the `MethodHandle` in our map links directly to a static Method in a public class in same classloader, the bytecode generator for now uses the `MethodHandleInfo` to generate a conventional `invokestatic`. I plan to remove this "optimization" after more benchmarking. You can easily disable the optimization by setting `directFunctions` to an empty Map. There is one downside in the hidden class feature that I added a workaround for: Hidden classes never appear in stack traces so my initial version did not print the actual JS function in stack trace. I worked around this by adding "exception patching" in the base class. It catches all Throwables throws by the generated code in the subclass and changes the stack trace to include the source code and hidden class name. This allows same user experience before. The current code is not backward compatible and needs an entry in `MIGRATE.txt`. I will investiagte how to add some helper methods for those people using `java.lang.reflect.Method` maps. Those can be easily converted to `MethodHandle` with some wrapper method. Support for separate classloader was removed. This is not a security issue, because the methods available for calling are defined externally and the API user is still responsible to only supply methods which should be accessible by users. The access checking is done by the provider of function map. For safety you can still let run whole of Lucene or the expressions module in a separate module/classloader. @rjernst, @jdconrad: Maybe this is also an idea for Painless. This PR replaces the outdated https://github.com/apache/lucene-solr/pull/1837. This closes #8933. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
