On 28/06/23(Wed) 08:58, Claudio Jeker wrote: > On Tue, Jun 27, 2023 at 08:18:15PM -0400, Kurt Miller wrote: > > On Jun 27, 2023, at 1:52 PM, Kurt Miller <[email protected]> wrote: > > > > > > On Jun 14, 2023, at 12:51 PM, Vitaliy Makkoveev <[email protected]> wrote: > > >> > > >> On Tue, May 30, 2023 at 01:31:08PM +0200, Martin Pieuchot wrote: > > >>> So it seems the java process is holding the `sysctl_lock' for too long > > >>> and block all other sysctl(2). This seems wrong to me. We should come > > >>> up with a clever way to prevent vslocking too much memory. A single > > >>> lock obviously doesn't fly with that many CPUs. > > >>> > > >> > > >> We vslock memory to prevent context switch while doing copyin() and > > >> copyout(), right? This is required for avoid context switch within > > >> foreach > > >> loops of kernel lock protected lists. But this seems not be required for > > >> simple sysctl_int() calls or rwlock protected data. So sysctl_lock > > >> acquisition and the uvm_vslock() calls could be avoided for significant > > >> count of mibs and pushed deep down for the rest. > > > > > > I’m back on -current testing and have some additional findings that > > > may help a bit. The memory leak fix had no effect on this issue. -current > > > behavior is as I previously described. When java trips the issue, it > > > goes into a state where many threads are all running at 100% cpu but > > > does not make forward progress. I’m going to call this state run-away java > > > process. Java is calling sched_yield(2) when in this state. > > > > > > When java is in run-away state, a different process can trip > > > the next stage were processes block waiting on sysctllk indefinitely. > > > Top with process arguments is one, pgrep and ps -axl also trip this. > > > My last test on -current java was stuck in run-away state for 7 hours > > > 45 minutes before cron daily ran and cause the lockups. > > > > > > I did a test with -current + locking sched_yield() back up with the > > > kernel lock. The behavior changed slightly. Java still enters run-away > > > state occasionally but eventually does make forward progress and > > > complete. When java is in run-away state the sysctllk issue can still > > > be tripped, but if it is not tripped java eventually completes. For > > > about 200 invocations of a java command that usually takes 50 seconds > > > to complete, 4 times java entered run-away state but eventually completed: > > > > > > Typically it runs like this: > > > 0m51.16s real 5m09.37s user 0m49.96s system > > > > > > The exceptions look like this: > > > 1m11.15s real 5m35.88s user 13m20.47s system > > > 27m18.93s real 31m13.19s user 754m48.41s system > > > 13m44.44s real 19m56.11s user 501m39.73s system > > > 19m23.72s real 24m40.97s user 629m08.16s system > > > > > > Testing -current with dumbsched.3 behaves the same as -current described > > > above. > > > > > > One other thing I observed so far is what happens when egdb is > > > Attached to the run-away java process. egdb stops the process > > > using ptrace(2) PT_ATTACH. Now if I issue a command that would > > > typically lock up the system like top displaying command line > > > arguments, the system does not lock up. I think this rules out > > > the kernel memory is fragmented theory. > > > > > > Switching cpu’s in ddb tends to lock up ddb so I have limited > > > info but here what I have from -current lockup and -current > > > with dumbsched.3 lockup. > > > > Another data point to support the idea of a missing wakeup; when > > java is in run-away state, if I send SIGSTOP followed by SIGCONT > > it dislodges it from run-away state and returns to normal operation. > > I doubt this is a missing wakeup. It is more the system is thrashing and > not making progress. The SIGSTOP causes all threads to park which means > that the thread not busy in its sched_yield() loop will finish its operation > and then on SIGCONT progress is possible. > > I need to recheck your ps output from ddb but I guess one of the threads > is stuck in a different place. That is where we need to look. > It may well be a bad interaction between SCHED_LOCK() and whatever else is > going on.
Or simply a poor userland scheduling based on sched_yield()... To me it seems there are two bugs in your report: 1/ a deadlock due to a single rwlock in sysctl(2) 2/ something unknown in java not making progress and calling sched_yield() and triggering 1/ While 1/ is well understood 2/ isn't. Why is java not making progress is what should be understood. Knowing where is the sched_yield() coming from can help.
