It seems that a possible workaround is to set the property "jdk.lang.processReaperUseDefaultStackSize" to "true". If that property is "true", then the worker thread uses the "default" stack size, whatever that is; if not, then the worker thread uses a stack size of 128 * 1024.
https://sources.debian.org/src/openjdk-11/11.0.7+10-3/src/java.base/share/classes/java/lang/ProcessHandleImpl.java/#L81 This property can be defined either by setting the environment variable JAVA_TOOL_OPTIONS, or by calling java.lang.System.setProperty before starting any processes: JAVA_TOOL_OPTIONS=-Djdk.lang.processReaperUseDefaultStackSize=true octave -q -W -f --eval 'b = javaObject("java.lang.ProcessBuilder", {"/bin/true"}); p = b.start(); p.waitFor()' octave -q -W -f --eval 'javaMethod("setProperty", "java.lang.System", "jdk.lang.processReaperUseDefaultStackSize", "true"); b = javaObject("java.lang.ProcessBuilder", {"/bin/true"}); p = b.start(); p.waitFor()' seems to work on buster or bullseye. This doesn't answer the questions of: - Why does the stack overflow when the JVM is loaded from Octave, and not when the JVM is launched "normally"? (What is that worker thread doing that would cause it to use any appreciable amount of stack space?) - Why, on bullseye, is it unable to create the worker thread in the first place? (The error message suggests that pthread_create doesn't like the specified attributes for some reason - why is that?) And again, why only in Octave? (I think the "OutOfMemoryError" is a red herring and that is simply the exception that Java always uses when pthread_create fails.)
