On Jun 14, 2023, at 12:51 PM, Vitaliy Makkoveev <[email protected]> wrote:
>
> On Tue, May 30, 2023 at 01:31:08PM +0200, Martin Pieuchot wrote:
>> So it seems the java process is holding the `sysctl_lock' for too long
>> and block all other sysctl(2). This seems wrong to me. We should come
>> up with a clever way to prevent vslocking too much memory. A single
>> lock obviously doesn't fly with that many CPUs.
>>
>
> We vslock memory to prevent context switch while doing copyin() and
> copyout(), right? This is required for avoid context switch within foreach
> loops of kernel lock protected lists. But this seems not be required for
> simple sysctl_int() calls or rwlock protected data. So sysctl_lock
> acquisition and the uvm_vslock() calls could be avoided for significant
> count of mibs and pushed deep down for the rest.
I’m back on -current testing and have some additional findings that
may help a bit. The memory leak fix had no effect on this issue. -current
behavior is as I previously described. When java trips the issue, it
goes into a state where many threads are all running at 100% cpu but
does not make forward progress. I’m going to call this state run-away java
process. Java is calling sched_yield(2) when in this state.
When java is in run-away state, a different process can trip
the next stage were processes block waiting on sysctllk indefinitely.
Top with process arguments is one, pgrep and ps -axl also trip this.
My last test on -current java was stuck in run-away state for 7 hours
45 minutes before cron daily ran and cause the lockups.
I did a test with -current + locking sched_yield() back up with the
kernel lock. The behavior changed slightly. Java still enters run-away
state occasionally but eventually does make forward progress and
complete. When java is in run-away state the sysctllk issue can still
be tripped, but if it is not tripped java eventually completes. For
about 200 invocations of a java command that usually takes 50 seconds
to complete, 4 times java entered run-away state but eventually completed:
Typically it runs like this:
0m51.16s real 5m09.37s user 0m49.96s system
The exceptions look like this:
1m11.15s real 5m35.88s user 13m20.47s system
27m18.93s real 31m13.19s user 754m48.41s system
13m44.44s real 19m56.11s user 501m39.73s system
19m23.72s real 24m40.97s user 629m08.16s system
Testing -current with dumbsched.3 behaves the same as -current described
above.
One other thing I observed so far is what happens when egdb is
Attached to the run-away java process. egdb stops the process
using ptrace(2) PT_ATTACH. Now if I issue a command that would
typically lock up the system like top displaying command line
arguments, the system does not lock up. I think this rules out
the kernel memory is fragmented theory.
Switching cpu’s in ddb tends to lock up ddb so I have limited
info but here what I have from -current lockup and -current
with dumbsched.3 lockup.
-current lockup 7 hours 45 min of run-away java then cron tripped it:
login: Stopped at db_enter+0x8: nop
ddb{0}> show all locks
CPU 6:
exclusive sched_lock &sched_lock r = 0 (0x1ca4aa0)
CPU 12:
exclusive mutex &pm->pm_mtx r = 0 (0x4001995e7f0)
Process 94379 (pgrep) thread 0x40018de79e0 (98675)
exclusive rwlock sysctllk r = 0 (0x1c240f8)
Process 70417 (java) thread 0x40018fdf140 (294923)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x40018fdf980 (389444)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x40018fdfc40 (20515)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x400190c5c50 (325344)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x400190c4650 (450774)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x400190c4390 (441666)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x400190c5990 (398914)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x4001910b420 (176161)
exclusive rwlock uobjlk r = 0 (0x40018f72190)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x4001910b160 (12453)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x4001910a0e0 (299519)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x4001910a920 (95241)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x40019604bf0 (281549)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 70417 (java) thread 0x40019605430 (219368)
shared rwlock vmmaplk r = 0 (0x400183a58b8)
Process 20430 (idle0) thread 0x40019965600 (355187)
exclusive kernel_lock &kernel_lock r = 0 (0x1cdfa38)
ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
51289 418052 86061 0 3 0x100090 sysctllk cron
68696 17984 86061 0 3 0x100090 sysctllk cron
57906 47164 86061 0 3 0x100090 sysctllk cron
57329 81627 86061 0 3 0x100090 sysctllk cron
7335 261208 86061 0 3 0x100090 sysctllk cron
87862 508928 86061 0 3 0x100090 sysctllk cron
83201 434704 86061 0 3 0x100090 sysctllk cron
45010 156734 86061 0 3 0x100090 sysctllk cron
20965 332634 86061 0 3 0x100090 sysctllk cron
26491 8756 86061 0 3 0x100090 sysctllk cron
94379 98675 52394 0 2 0x2 pgrep
52394 126132 45519 0 3 0x10008a sigsusp ksh
45519 206473 19571 0 3 0x10008a sigsusp ksh
19571 522648 65500 0 3 0x10008a sigsusp sh
65500 25815 8286 0 3 0x10008a sigsusp sh
8286 268463 86061 0 3 0x100090 piperd cron
70417 168721 71557 55 3 0x200083 fsleep java
70417 13950 71557 55 3 0x4200083 fsleep java
70417 502030 71557 55 7 0x4200003 java
70417 230060 71557 55 2 0x4200003 java
70417 341173 71557 55 2 0x4200003 java
70417 294923 71557 55 7 0x4200003 java
70417 389444 71557 55 7 0x4200003 java
70417 177348 71557 55 3 0x4200083 fsleep java
70417 9385 71557 55 2 0x4200003 java
70417 20515 71557 55 7 0x4200003 java
70417 145345 71557 55 3 0x4200083 fsleep java
70417 164798 71557 55 2 0x4200003 java
70417 311721 71557 55 3 0x4200083 fsleep java
70417 325344 71557 55 7 0x4200003 java
70417 450774 71557 55 2 0x4200003 java
70417 402679 71557 55 2 0x4200003 java
70417 441666 71557 55 7 0x4200003 java
70417 63472 71557 55 3 0x4200083 fsleep java
70417 398914 71557 55 7 0x4200003 java
70417 313943 71557 55 3 0x4200083 fsleep java
70417 295761 71557 55 3 0x4200083 fsleep java
70417 202862 71557 55 7 0x4200003 java
70417 402672 71557 55 2 0x4200003 java
70417 139718 71557 55 7 0x4200003 java
70417 176161 71557 55 7 0x4200003 java
70417 180366 71557 55 2 0x4200003 java
70417 508023 71557 55 7 0x4200003 java
70417 491109 71557 55 7 0x4200003 java
70417 12453 71557 55 7 0x4200003 java
70417 252579 71557 55 2 0x4200003 java
70417 146951 71557 55 2 0x4200003 java
70417 299519 71557 55 7 0x4200003 java
70417 99530 71557 55 2 0x4200003 java
70417 95241 71557 55 7 0x4200003 java
70417 129511 71557 55 3 0x4200083 fsleep java
70417 186515 71557 55 3 0x4200083 fsleep java
70417 419185 71557 55 2 0x4200003 java
70417 519651 71557 55 2 0x4200003 java
70417 281549 71557 55 7 0x4200003 java
70417 398760 71557 55 7 0x4200003 java
70417 397086 71557 55 3 0x4200083 fsleep java
70417 30289 71557 55 3 0x4200083 fsleep java
70417 228759 71557 55 7 0x4200003 java
70417 219368 71557 55 7 0x4200003 java
70417 327537 71557 55 3 0x4200083 fsleep java
70417 77530 71557 55 3 0x4200083 fsleep java
70417 409186 71557 55 3 0x4200083 fsleep java
70417 323740 71557 55 3 0x4200083 fsleep java
70417 217331 71557 55 3 0x4200083 fsleep java
70417 148838 71557 55 3 0x4200083 fsleep java
70417 493426 71557 55 3 0x4200083 fsleep java
70417 426834 71557 55 3 0x4200083 fsleep java
70417 371111 71557 55 3 0x4200083 fsleep java
70417 479772 71557 55 3 0x4200083 fsleep java
70417 177616 71557 55 3 0x4200083 fsleep java
70417 198266 71557 55 3 0x4200083 fsleep java
70417 160268 71557 55 3 0x4200083 fsleep java
70417 507189 71557 55 3 0x4200083 fsleep java
70417 163501 71557 55 3 0x4200083 fsleep java
70417 143106 71557 55 3 0x4200083 fsleep java
70417 72839 71557 55 3 0x4200083 fsleep java
70417 27480 71557 55 3 0x4200083 fsleep java
70417 271014 71557 55 3 0x4200083 fsleep java
70417 430403 71557 55 3 0x4200083 fsleep java
70417 484308 71557 55 3 0x4200083 fsleep java
70417 66557 71557 55 3 0x4200083 fsleep java
70417 368986 71557 55 3 0x4200083 fsleep java
70417 79694 71557 55 3 0x4200083 fsleep java
70417 263769 71557 55 2 0x4200003 java
71557 313437 54815 1000 3 0x10008b sigsusp sh
38823 473078 1 1000 3 0x1000b0 kqread ssh-agent
54815 179485 85124 1000 3 0x10008b sigsusp ksh
85124 174124 82992 1000 3 0x98 kqread sshd
82992 505962 19117 0 3 0x82 kqread sshd
94226 105636 34562 1000 3 0x100083 sysctllk top
45181 241106 1 1000 3 0x1000b0 kqread ssh-agent
34562 27078 35337 1000 3 0x10008b sigsusp ksh
35337 208560 99583 1000 3 0x98 kqread sshd
99583 298485 19117 0 3 0x82 kqread sshd
66757 202899 1 1000 3 0x1000b0 kqread ssh-agent
14076 422317 65186 1000 3 0x100083 ttyin ksh
65186 473504 69676 1000 3 0x98 kqread sshd
69676 241944 19117 0 3 0x82 kqread sshd
23096 457141 1 0 3 0x100083 ttyin getty
86061 33803 1 0 3 0x100098 kqread cron
5132 232368 1 99 3 0x1100090 kqread sndiod
87193 153492 1 110 3 0x100090 kqread sndiod
76688 357361 51636 95 3 0x1100092 kqread smtpd
88490 417242 51636 103 3 0x1100092 kqread smtpd
7527 264480 51636 95 3 0x1100092 kqread smtpd
83722 493013 51636 95 3 0x100092 kqread smtpd
18935 371029 51636 95 3 0x1100092 kqread smtpd
38021 196456 51636 95 3 0x1100092 kqread smtpd
51636 466931 1 0 3 0x100080 kqread smtpd
19117 115784 1 0 3 0x88 kqread sshd
59327 195859 595 0 3 0x80 nfsd nfsd
3775 506117 595 0 3 0x80 nfsd nfsd
71373 470169 595 0 3 0x80 nfsd nfsd
55013 265586 595 0 3 0x80 nfsd nfsd
595 146319 1 0 3 0x80 netcon nfsd
6239 131614 87208 0 3 0x80 kqread mountd
87208 116254 1 0 3 0x100080 kqread mountd
31615 209009 1 28 3 0x1100090 kqread portmap
9418 483286 1 0 3 0x100080 kqread ntpd
67774 457179 43435 83 3 0x100092 kqread ntpd
43435 200485 1 83 3 0x1100092 sysctllk ntpd
23724 476637 40297 74 3 0x1100092 bpf pflogd
40297 194081 1 0 3 0x80 netio pflogd
85000 507820 99353 73 3 0x1100090 kqread syslogd
99353 311791 1 0 3 0x100082 netio syslogd
78963 330574 1 0 3 0x100080 kqread resolvd
65579 392676 99143 77 3 0x100092 kqread dhcpleased
59810 341186 99143 77 3 0x100092 sysctllk dhcpleased
99143 123967 1 0 3 0x80 kqread dhcpleased
37546 265338 60569 115 3 0x100092 kqread slaacd
75464 520417 60569 115 3 0x100092 kqread slaacd
60569 96744 1 0 3 0x100080 kqread slaacd
66229 200910 0 0 3 0x14200 bored smr
55338 55765 0 0 3 0x14200 pgzero zerothread
18382 107837 0 0 3 0x14200 aiodoned aiodoned
71381 37537 0 0 3 0x14200 syncer update
13189 63145 0 0 3 0x14200 cleaner cleaner
58679 163470 0 0 3 0x14200 reaper reaper
8491 352290 0 0 3 0x14200 pgdaemon pagedaemon
42991 271700 0 0 3 0x14200 usbtsk usbtask
99703 278524 0 0 3 0x14200 usbatsk usbatsk
44097 41543 0 0 3 0x14200 bored sensors
29515 395831 0 0 7 0x40014200 idle63
59879 233122 0 0 7 0x40014200 idle62
24479 472135 0 0 7 0x40014200 idle61
32724 180485 0 0 7 0x40014200 idle60
84522 352102 0 0 7 0x40014200 idle59
88468 28468 0 0 7 0x40014200 idle58
9730 521314 0 0 7 0x40014200 idle57
88126 143658 0 0 7 0x40014200 idle56
62088 480688 0 0 7 0x40014200 idle55
49888 301613 0 0 7 0x40014200 idle54
91026 435639 0 0 7 0x40014200 idle53
83946 31842 0 0 7 0x40014200 idle52
42400 517472 0 0 7 0x40014200 idle51
78935 250217 0 0 7 0x40014200 idle50
86793 491694 0 0 7 0x40014200 idle49
85326 138207 0 0 7 0x40014200 idle48
45276 98532 0 0 7 0x40014200 idle47
15671 124998 0 0 7 0x40014200 idle46
4442 468098 0 0 7 0x40014200 idle45
92628 140541 0 0 3 0x40014200 idle44
20098 1145 0 0 7 0x40014200 idle43
94172 136638 0 0 7 0x40014200 idle42
88998 189223 0 0 7 0x40014200 idle41
29884 315017 0 0 7 0x40014200 idle40
44014 403200 0 0 7 0x40014200 idle39
22125 330402 0 0 7 0x40014200 idle38
21689 142353 0 0 7 0x40014200 idle37
69531 201013 0 0 7 0x40014200 idle36
71325 26153 0 0 7 0x40014200 idle35
83336 185241 0 0 7 0x40014200 idle34
71152 302638 0 0 7 0x40014200 idle33
7298 350142 0 0 3 0x40014200 idle32
19260 236429 0 0 7 0x40014200 idle31
93685 90232 0 0 7 0x40014200 idle30
94248 66415 0 0 7 0x40014200 idle29
24474 416674 0 0 3 0x40014200 idle28
98540 391643 0 0 7 0x40014200 idle27
10250 18831 0 0 7 0x40014200 idle26
95467 39821 0 0 3 0x40014200 idle25
1906 132445 0 0 3 0x40014200 idle24
14522 4049 0 0 7 0x40014200 idle23
55130 270348 0 0 7 0x40014200 idle22
53053 246249 0 0 7 0x40014200 idle21
74782 36823 0 0 7 0x40014200 idle20
59141 169699 0 0 3 0x40014200 idle19
93213 266561 0 0 7 0x40014200 idle18
71418 222712 0 0 7 0x40014200 idle17
17525 407410 0 0 3 0x40014200 idle16
37850 475698 0 0 3 0x40014200 idle15
52754 297462 0 0 3 0x40014200 idle14
36038 64823 0 0 3 0x40014200 idle13
8981 422888 0 0 3 0x40014200 idle12
21373 172943 0 0 7 0x40014200 idle11
72002 225983 0 0 7 0x40014200 idle10
98855 57927 0 0 7 0x40014200 idle9
11446 464080 0 0 3 0x40014200 idle8
68075 154944 0 0 3 0x40014200 idle7
27722 23851 0 0 3 0x40014200 idle6
23981 50539 0 0 3 0x40014200 idle5
25972 501167 0 0 3 0x40014200 idle4
57037 170331 0 0 3 0x40014200 idle3
2065 172225 0 0 3 0x40014200 idle2
60861 163397 0 0 3 0x40014200 idle1
50838 149027 0 0 3 0x14200 bored softnet3
66043 220781 0 0 3 0x14200 bored softnet2
46438 302233 0 0 3 0x14200 bored softnet1
27174 73769 0 0 3 0x14200 bored softnet0
22895 298543 0 0 3 0x14200 bored systqmp
83001 263628 0 0 3 0x14200 bored systq
62839 447694 0 0 3 0x40014200 bored softclock
*20430 355187 0 0 7 0x40014200 idle0
16760 433746 0 0 3 0x14200 kmalloc kmthread
1 508830 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{0}> ps/o
TID PID UID PRFLAGS PFLAGS CPU COMMAND
502030 70417 55 0x200003 0x4000000 6 java
294923 70417 55 0x200003 0x4000000 5 java
389444 70417 55 0x200003 0x4000000 7 java
20515 70417 55 0x200003 0x4000000 15 java
325344 70417 55 0x200003 0x4000000 44 java
441666 70417 55 0x200003 0x4000000 1 java
398914 70417 55 0x200003 0x4000000 16 java
202862 70417 55 0x200003 0x4000000 8 java
139718 70417 55 0x200003 0x4000000 4 java
176161 70417 55 0x200003 0x4000000 12 java
508023 70417 55 0x200003 0x4000000 14 java
491109 70417 55 0x200003 0x4000000 25 java
12453 70417 55 0x200003 0x4000000 28 java
299519 70417 55 0x200003 0x4000000 13 java
95241 70417 55 0x200003 0x4000000 24 java
281549 70417 55 0x200003 0x4000000 2 java
398760 70417 55 0x200003 0x4000000 3 java
228759 70417 55 0x200003 0x4000000 19 java
219368 70417 55 0x200003 0x4000000 32 java
ddb{0}> machine ddbcpu 5
<<<ddb locked>>>
-current with dumbsched.3 lockup by cron
login: Stopped at db_enter+0x8: nop
ddb{0}> show all locks
CPU 1:
exclusive sched_lock &sched_lock r = 0 (0x1c97ae8)
Process 60231 (pgrep) thread 0x40019027c30 (139536)
exclusive rwlock sysctllk r = 0 (0x1c1b128)
Process 65958 (java) thread 0x400194baeb0 (301842)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400187c1150 (80639)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400187c08e0 (130285)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400187c0bb0 (372964)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400187c0340 (114483)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400187c1c90 (370824)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400187c1420 (291903)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40019706bd0 (192156)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eec5c0 (504607)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eecb60 (464875)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eec2f0 (286485)
exclusive kernel_lock &kernel_lock r = 0 (0x1cd0430)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eec890 (14503)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eed6a0 (474246)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eece30 (13168)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eedc40 (382848)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eec020 (438667)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018eed970 (156481)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e54320 (457654)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e54e60 (435923)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e55400 (192918)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e559a0 (419962)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e55c70 (39251)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e54b90 (496191)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e54050 (405221)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x40018e55130 (186762)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c2e70 (261075)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c2600 (223229)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c2060 (456341)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c28d0 (31372)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c39b0 (112927)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c3410 (299897)
exclusive rwlock uobjlk r = 0 (0x40018e52060)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c2330 (455167)
shared rwlock vmmaplk r = 0 (0x40019704678)
Process 65958 (java) thread 0x400190c36e0 (164561)
shared rwlock vmmaplk r = 0 (0x40019704678)
ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
93903 102019 2381 0 3 0x100090 sysctllk cron
2846 152512 2381 0 3 0x100090 sysctllk cron
10813 127849 2381 0 3 0x100090 sysctllk cron
3486 389348 2381 0 3 0x100090 sysctllk cron
74120 491228 2381 0 3 0x100090 sysctllk cron
58089 370399 2381 0 3 0x100090 sysctllk cron
60231 139536 62752 0 3 0x2 vmmaplk pgrep
62752 448959 75065 0 3 0x10008a sigsusp ksh
75065 226858 60722 0 3 0x10008a sigsusp ksh
60722 61207 52294 0 3 0x10008a sigsusp sh
52294 387320 71832 0 3 0x10008a sigsusp sh
71832 213690 2381 0 3 0x100090 piperd cron
65958 7230 45531 55 3 0x200083 fsleep java
65958 192531 45531 55 3 0x4200083 fsleep java
65958 301842 45531 55 7 0x4200003 java
65958 80639 45531 55 7 0x4200003 java
65958 130285 45531 55 7 0x4200003 java
65958 52542 45531 55 3 0x4200083 fsleep java
65958 372964 45531 55 7 0x4200003 java
65958 147182 45531 55 3 0x4200083 fsleep java
65958 114483 45531 55 7 0x4200003 java
65958 370824 45531 55 7 0x4200003 java
65958 291903 45531 55 7 0x4200003 java
65958 192156 45531 55 7 0x4200003 java
65958 504607 45531 55 7 0x4200003 java
65958 41811 45531 55 3 0x4200083 fsleep java
65958 464875 45531 55 3 0x4200003 uobjlk java
*65958 286485 45531 55 7 0x4200003 java
65958 14503 45531 55 7 0x4200003 java
65958 474246 45531 55 7 0x4200003 java
65958 13168 45531 55 7 0x4200003 java
65958 382848 45531 55 7 0x4200003 java
65958 128100 45531 55 3 0x4200083 fsleep java
65958 438667 45531 55 7 0x4200003 java
65958 156481 45531 55 7 0x4200003 java
65958 457654 45531 55 7 0x4200003 java
65958 213184 45531 55 7 0x4200003 java
65958 435923 45531 55 7 0x4200003 java
65958 192918 45531 55 7 0x4200003 java
65958 419962 45531 55 2 0x4200003 java
65958 375949 45531 55 3 0x4200083 fsleep java
65958 39251 45531 55 7 0x4200003 java
65958 496191 45531 55 7 0x4200003 java
65958 405221 45531 55 7 0x4200003 java
65958 27776 45531 55 3 0x4200083 fsleep java
65958 186762 45531 55 2 0x4200003 java
65958 261075 45531 55 7 0x4200003 java
65958 223229 45531 55 7 0x4200003 java
65958 456341 45531 55 7 0x4200003 java
65958 31372 45531 55 7 0x4200003 java
65958 112927 45531 55 7 0x4200003 java
65958 299897 45531 55 7 0x4200003 java
65958 150241 45531 55 3 0x4200083 fsleep java
65958 209097 45531 55 3 0x4200083 fsleep java
65958 455167 45531 55 7 0x4200003 java
65958 138810 45531 55 3 0x4200083 fsleep java
65958 164561 45531 55 7 0x4200003 java
65958 324079 45531 55 3 0x4200083 fsleep java
65958 212246 45531 55 3 0x4200083 fsleep java
65958 135053 45531 55 3 0x4200083 fsleep java
65958 136209 45531 55 3 0x4200083 fsleep java
65958 241168 45531 55 3 0x4200083 fsleep java
65958 25033 45531 55 3 0x4200083 fsleep java
65958 265142 45531 55 3 0x4200083 fsleep java
65958 271376 45531 55 3 0x4200083 fsleep java
65958 507881 45531 55 3 0x4200083 fsleep java
65958 299826 45531 55 3 0x4200083 fsleep java
65958 216677 45531 55 3 0x4200083 fsleep java
65958 391531 45531 55 3 0x4200083 fsleep java
65958 91320 45531 55 3 0x4200083 fsleep java
65958 83315 45531 55 3 0x4200083 fsleep java
65958 210770 45531 55 3 0x4200083 fsleep java
65958 215295 45531 55 3 0x4200083 fsleep java
65958 210392 45531 55 3 0x4200083 fsleep java
65958 350031 45531 55 3 0x4200083 fsleep java
65958 169175 45531 55 3 0x4200083 fsleep java
65958 463217 45531 55 3 0x4200083 fsleep java
65958 242928 45531 55 3 0x4200083 fsleep java
65958 53076 45531 55 3 0x4200083 fsleep java
65958 489978 45531 55 3 0x4200083 fsleep java
65958 339429 45531 55 7 0x4200003 java
45531 229729 80664 1000 3 0x10008b sigsusp sh
48802 195515 1 1000 3 0x1000b0 kqread ssh-agent
4440 454765 95357 1000 3 0x100083 ttyin ksh
95357 172382 48777 1000 3 0x98 kqread sshd
70875 345373 1 1000 3 0x1000b0 kqread ssh-agent
80664 101441 81706 1000 3 0x10008b sigsusp ksh
81706 60091 81193 1000 3 0x98 kqread sshd
81193 108230 82274 0 3 0x82 kqread sshd
70046 506217 96291 1000 3 0x100083 sysctllk top
86364 130483 1 1000 3 0x1000b0 kqread ssh-agent
96291 505127 71169 1000 3 0x10008b sigsusp ksh
71169 137587 52426 1000 3 0x98 kqread sshd
52426 76557 82274 0 3 0x82 kqread sshd
48777 67113 82274 0 3 0x82 kqread sshd
74626 521740 1 0 3 0x100083 ttyin getty
2381 40188 1 0 3 0x100098 kqread cron
39949 142081 1 99 3 0x1100090 kqread sndiod
99537 194538 1 110 3 0x100090 kqread sndiod
74845 68171 71956 95 3 0x1100092 kqread smtpd
63896 495712 71956 103 3 0x1100092 kqread smtpd
68722 3502 71956 95 3 0x1100092 kqread smtpd
69693 227928 71956 95 3 0x100092 kqread smtpd
68799 261017 71956 95 3 0x1100092 kqread smtpd
88708 519105 71956 95 3 0x1100092 kqread smtpd
71956 502628 1 0 3 0x100080 kqread smtpd
82274 225740 1 0 3 0x88 kqread sshd
73894 96984 75584 0 3 0x80 nfsd nfsd
16655 123345 75584 0 3 0x80 nfsd nfsd
86188 520974 75584 0 3 0x80 nfsd nfsd
46810 494909 75584 0 3 0x80 nfsd nfsd
75584 208502 1 0 3 0x80 netcon nfsd
26786 222700 50391 0 3 0x80 kqread mountd
50391 512874 1 0 3 0x100080 kqread mountd
16225 135536 1 28 3 0x1100090 kqread portmap
13738 74857 1 0 3 0x100080 kqread ntpd
54592 488470 79129 83 3 0x100092 kqread ntpd
79129 61531 1 83 3 0x1100092 sysctllk ntpd
87803 470330 87425 74 3 0x1100092 bpf pflogd
87425 237375 1 0 3 0x80 netio pflogd
81331 500693 52507 73 3 0x1100090 kqread syslogd
52507 4057 1 0 3 0x100082 netio syslogd
8291 56979 1 0 3 0x100080 kqread resolvd
23383 153324 9390 77 3 0x100092 kqread dhcpleased
37258 147216 9390 77 3 0x100092 sysctllk dhcpleased
9390 175267 1 0 3 0x80 kqread dhcpleased
25590 342469 41177 115 3 0x100092 kqread slaacd
61357 393483 41177 115 3 0x100092 kqread slaacd
41177 339577 1 0 3 0x100080 kqread slaacd
12197 470283 0 0 3 0x14200 bored smr
77832 80715 0 0 3 0x14200 pgzero zerothread
24848 144815 0 0 3 0x14200 aiodoned aiodoned
84782 162845 0 0 3 0x14200 syncer update
62736 362379 0 0 3 0x14200 cleaner cleaner
77489 152573 0 0 3 0x14200 reaper reaper
81116 98229 0 0 3 0x14200 pgdaemon pagedaemon
89226 176649 0 0 3 0x14200 usbtsk usbtask
72519 481877 0 0 3 0x14200 usbatsk usbatsk
85670 516841 0 0 3 0x14200 bored sensors
98189 52134 0 0 7 0x40014200 idle63
99260 88385 0 0 3 0x40014200 idle62
46445 341827 0 0 3 0x40014200 idle61
94364 522577 0 0 3 0x40014200 idle60
14769 274448 0 0 3 0x40014200 idle59
81632 201907 0 0 3 0x40014200 idle58
26470 251736 0 0 3 0x40014200 idle57
34847 227613 0 0 3 0x40014200 idle56
59706 117732 0 0 7 0x40014200 idle55
97065 448673 0 0 7 0x40014200 idle54
90538 423266 0 0 7 0x40014200 idle53
40550 278262 0 0 7 0x40014200 idle52
37783 367165 0 0 7 0x40014200 idle51
26201 99138 0 0 3 0x40014200 idle50
5997 415312 0 0 3 0x40014200 idle49
51672 171284 0 0 3 0x40014200 idle48
21498 239237 0 0 3 0x40014200 idle47
5162 58072 0 0 7 0x40014200 idle46
35391 81257 0 0 7 0x40014200 idle45
13717 175738 0 0 3 0x40014200 idle44
22878 425620 0 0 7 0x40014200 idle43
42648 154481 0 0 3 0x40014200 idle42
56379 144823 0 0 7 0x40014200 idle41
92699 168282 0 0 7 0x40014200 idle40
6077 405289 0 0 7 0x40014200 idle39
85591 131257 0 0 7 0x40014200 idle38
52318 223053 0 0 3 0x40014200 idle37
18267 161512 0 0 3 0x40014200 idle36
77653 70785 0 0 7 0x40014200 idle35
53498 284796 0 0 7 0x40014200 idle34
98386 491140 0 0 7 0x40014200 idle33
58847 69465 0 0 7 0x40014200 idle32
72244 66385 0 0 7 0x40014200 idle31
55149 418920 0 0 3 0x40014200 idle30
29482 386405 0 0 7 0x40014200 idle29
7945 294845 0 0 3 0x40014200 idle28
77046 162724 0 0 7 0x40014200 idle27
95985 175324 0 0 7 0x40014200 idle26
18649 373012 0 0 7 0x40014200 idle25
49823 117696 0 0 7 0x40014200 idle24
47921 174878 0 0 3 0x40014200 idle23
14079 99854 0 0 7 0x40014200 idle22
55898 93973 0 0 7 0x40014200 idle21
58267 281434 0 0 7 0x40014200 idle20
76147 134880 0 0 3 0x40014200 idle19
11370 82585 0 0 3 0x40014200 idle18
56274 350307 0 0 7 0x40014200 idle17
50547 101022 0 0 3 0x40014200 idle16
93420 280840 0 0 3 0x40014200 idle15
88316 188059 0 0 7 0x40014200 idle14
57916 283360 0 0 3 0x40014200 idle13
15079 476571 0 0 3 0x40014200 idle12
2884 93886 0 0 7 0x40014200 idle11
65531 34898 0 0 3 0x40014200 idle10
80833 47701 0 0 3 0x40014200 idle9
51420 1696 0 0 3 0x40014200 idle8
91655 293964 0 0 3 0x40014200 idle7
75770 490286 0 0 7 0x40014200 idle6
57207 57170 0 0 3 0x40014200 idle5
42335 487067 0 0 3 0x40014200 idle4
60599 456740 0 0 7 0x40014200 idle3
43120 54537 0 0 7 0x40014200 idle2
5551 505482 0 0 3 0x40014200 idle1
85621 237388 0 0 3 0x14200 bored softnet3
67718 73745 0 0 3 0x14200 bored softnet2
52477 123744 0 0 3 0x14200 bored softnet1
94812 204004 0 0 3 0x14200 bored softnet0
37170 237307 0 0 3 0x14200 bored systqmp
79978 484491 0 0 3 0x14200 bored systq
90234 164151 0 0 3 0x40014200 bored softclock
2208 386559 0 0 3 0x40014200 idle0
69952 114523 0 0 3 0x14200 kmalloc kmthread
1 46716 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{0}> ps /o
TID PID UID PRFLAGS PFLAGS CPU COMMAND
301842 65958 55 0x200003 0x4000000 15 java
80639 65958 55 0x200003 0x4000000 62 java
130285 65958 55 0x200003 0x4000000 23 java
372964 65958 55 0x200003 0x4000000 1 java
114483 65958 55 0x200003 0x4000000 13 java
370824 65958 55 0x200003 0x4000000 9 java
291903 65958 55 0x200003 0x4000000 59 java
192156 65958 55 0x200003 0x4000000 57 java
504607 65958 55 0x200003 0x4000000 56 java
*286485 65958 55 0x200003 0x4000000 0K java
14503 65958 55 0x200003 0x4000000 36 java
474246 65958 55 0x200003 0x4000000 19 java
13168 65958 55 0x200003 0x4000000 16 java
382848 65958 55 0x200003 0x4000000 42 java
438667 65958 55 0x200003 0x4000000 60 java
156481 65958 55 0x200003 0x4000000 18 java
457654 65958 55 0x200003 0x4000000 10 java
213184 65958 55 0x200003 0x4000000 48 java
435923 65958 55 0x200003 0x4000000 61 java
192918 65958 55 0x200003 0x4000000 37 java
39251 65958 55 0x200003 0x4000000 5 java
496191 65958 55 0x200003 0x4000000 30 java
405221 65958 55 0x200003 0x4000000 12 java
261075 65958 55 0x200003 0x4000000 28 java
223229 65958 55 0x200003 0x4000000 47 java
456341 65958 55 0x200003 0x4000000 58 java
31372 65958 55 0x200003 0x4000000 49 java
112927 65958 55 0x200003 0x4000000 8 java
299897 65958 55 0x200003 0x4000000 7 java
455167 65958 55 0x200003 0x4000000 44 java
164561 65958 55 0x200003 0x4000000 4 java
339429 65958 55 0x200003 0x4000000 50 java
ddb{0}> tr
vcons_softintr(400199b2800, 8, 4001995e300, 40019704580, ffffffffffffffff,
40000c22330) at vcons_softintr+0x3c
intr_handler(2017ec8, 400199a8a00, 469ea907, 2, 1c00, 0) at intr_handler+0x50
sparc_intr_retry(1c972e0, 8, 17c4560, 0, 0, 19b0210) at sparc_intr_retry+0x5c
sleep_finish(0, 0, 20, 19957b0, 0, 0) at sleep_finish+0x90
rw_enter(40018e52050, 1, 1c03000, 0, 0, 0) at rw_enter+0x298
uvm_fault_lower_lookup(400e45d9cd0, 400e45d9d08, 400e45d9bd0, 0, 0, 4) at
uvm_fault_lower_lookup+0x2c
uvm_fault_lower(400192474a0, 400e45d9d08, 400e45d9bd0, 0, 1c0c000, 1c970d8) at
uvm_fault_lower+0x3c
uvm_fault(0, 400e45d9cd0, 0, 400192474a0, 10267e0, 3) at uvm_fault+0x1bc
text_access_fault(400e45d9ed0, 9, 39ac742254, 0, 0, 0) at
text_access_fault+0x114
sun4v_texttrap(3976ba0340, 52a, 3987677868, 398767786c, 0, 19b0210) at
sun4v_texttrap+0x1fc
ddb{0}> machine ddbcpu 15
<<< ddb lockup >>>
