I realize this is something of a broad question, but I hope that someone has already had to deal with it and can share some experience. Our application has multiple classloaders. This is to avoid dependency conflicts between the various APIs that our software accesses, including but not limited to Hadoop. Without using a multi-classloader strategy (or Java 9 modules, which we cannot yet support) we encounter a lot of dependency conflicts between the Hadoop jars and everything else. So, suffice it to say that we have an URLClassLoader for all of the Hadoop jars, and other classloaders for other set of libraries.
The problem is this: The system classloader (the one that the JVM starts with) does not have the Hadoop jars. This means that the thread-context classloader, which is the system classloader by default, does not have the Hadoop jars. Unfortunately, Hadoop APIs of all sorts seem to use the thread-context classloader implicitly, which leads to all manner of ClassNotFound errors, ServiceLoaders returning empty lists, factories not finding implementations, etc. This is driving us slowly mad, because new "not found" issues crop up occasionally in the field as our customers discover new security options and other configuration variants, which leads down code paths that fail to find classes or implementations of interfaces. You may suggest the obvious: Always set the thread-context classloader to be the Hadoop classloader. Unfortunately this doesn't seem to be a solution. First, threads get spawned which have the default system classloader as the thread-context-classloader, and I don't have control over all of those threads. Second, it is error-prone and probably too expensive to save and restore the thread-context classloader at every possible Hadoop entry point. The only answer I've found is to learn by trial-and-error which classes are loaded by the thread-context classloader (or by ServiceLoader), and force them to pre-load where I've set the thread-context classloader. Is there a better way? If not, is there a list of such classes? Thanks, John Lilley
