Tim Stewart <t...@stoo.org> writes:
> Martin Pieuchot <m...@openbsd.org> writes:
>
>> On 11/10/17(Wed) 17:01, Martin Pieuchot wrote:
>>> OpenBSD 6.2 includes nice performance and latency improvements due to
>>> the work done in the Network Stack in the previous years. However as
>>> soon as IPsec is enabled, all network related processing are affected.
>>> In other words you cannot profit from the last MP work in the Network
>>> stack if you use IPsec.
>
> By "IPsec is enabled," do you mean as soon as an AS appears in the
> kernel? Or is it some other condition?
>
>>> During the last 6 months I hoped that somebody else would look at the
>>> IPsec stack and tell us what needs to be done. This didn't happen.
>>>
>>> Now that 6.2 is released, we cannot afford to continue to parallelize
>>> the Network Stack if some of our users and testers still run it under
>>> KERNEL_LOCK().
>>>
>>> So I did an audit of the IPsec stack and came with the small diff below.
>>> This diff doesn't remove the KERNEL_LOCK() (yet), but add some asserts
>>> to make sure that the global data structure are all accessed while
>>> holding the NET_LOCK().
>>
>> Here's the diff to stop grabbing the KERNEL_LOCK(), please test and
>> report back.
>
> I have several iked tunnels that are normally active. I applied both
> patches last night against 6.2-stable, rebooted, and my router was hung
> when I woke up this morning. Unfortunately I wasn't really set up to
> capture crash information, and there are no dumps in /var/crash/.
>
> I don't have much experience with capturing OpenBSD kernel panics. I've
> set up screen on another system so that I'll have a log of serial
> console activity (this is an apu2c4) and have set ddb.console=1. I will
> also reboot with bsd.gdb this time. Is there anything else I should to
> increase my chances of getting a useful dump?
I reproduced the lockup again. There was no output on the console. The
router stopped forwarding traffic and I found that the root shell I had
open on the console was not respond. I sent a BREAK and was dropped
into ddb.
Following is the output of `show panic' a `trace' for each CPU, and a
`ps':
#
# Stopped at db_enter+0x5
[/home/src-cvs/sys/arch/amd64/amd64/db_interface.c
ddb{0}> show panic
the kernel did not panic
ddb{0}> trace
db_enter(ffff8000220f98f8,10,ffff8000220f9888,302,8,ffffffff8122bdd5) at
db_enter+0x5 [/home/src-cvs/sys/arch/amd64/amd64/db_interface.c:402]
comintr(ffffffff81ab42b0,ffff8000000cf000,0,1,ffff8000000ce480,ffffffff81b225a8)
at comintr+0x18f [/home/src-cvs/sys/dev/ic/com.c:1091]
intr_handler(ffff8000220f9be0,ffff8000000ce400,7,9,ffff8000000ce480,ffff80000007af00)
at intr_handler+0x5e [/home/src-cvs/sys/arch/amd64/amd64/intr.c:534]
Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xcc
--- interrupt ---
end of kernel
end trace frame: 0x7b91c905c700, count: -4
0x4c48246c894c5024:
ddb{0}> machine ddbcpu 1
Stopped at x86_ipi_db+0x5
[/home/src-cvs/sys/arch/amd64/amd64/db_interface.c:356]: popq %rbp
ddb{1}> trace
x86_ipi_db(ffffffff8111dd22,10,ffff8000220bf4a8,246,8,ffffffff8122bdc5) at
x86_ipi_db+0x5 [/home/src-cvs/sys/arch/amd64/amd64/db_interface.c:356]
x86_ipi_handler(ffff8000000716b8,1388,ffff800000014300,ffff800000071000,0,0) at
x86_ipi_handler+0x6b [/home/src-cvs/sys/arch/amd64/amd64/ipi.c:106]
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
end of kernel
end trace frame: 0x7bbf6405c75250cc, count: -3
0x41cb8c419c524153:
ddb{1}> machine ddbcpu 2
Stopped at x86_ipi_db+0x5
[/home/src-cvs/sys/arch/amd64/amd64/db_interface.c:356]: popq %rbp
ddb{2}> trace
x86_ipi_db(ffffffff8111dd22,10,ffff8000220c57f8,246,8,ffffffff8122bdc5) at
x86_ipi_db+0x5 [/home/src-cvs/sys/arch/amd64/amd64/db_interface.c:356]
x86_ipi_handler(ffff8000000726b8,1388,ffff800000014400,ffff800000072000,0,0) at
x86_ipi_handler+0x6b [/home/src-cvs/sys/arch/amd64/amd64/ipi.c:106]
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
end of kernel
end trace frame: 0x7bbf6405c75250cc, count: -3
0x41cb8c419c524153:
ddb{2}> machine ddbcpu 3
Stopped at x86_ipi_db+0x5
[/home/src-cvs/sys/arch/amd64/amd64/db_interface.c:356]: popq %rbp
ddb{3}> trace
x86_ipi_db(ffff80002222f4d0,10,ffff80002222f438,246,8,ffffffff8122bdc5) at
x86_ipi_db+0x5 [/home/src-cvs/sys/arch/amd64/amd64/db_interface.c:356]
x86_ipi_handler(ffff80002221cd70,0,ffff800000074000,ffff80002211e2f0,c,1080be)
at x86_ipi_handler+0x6b [/home/src-cvs/sys/arch/amd64/amd64/ipi.c:106]
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
end of kernel
end trace frame: 0x7bbf6405c75250cc, count: -3
0x41cb8c419c524153:
ddb{3}> machine ddbcpu 0
ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
37136 487145 7535 0 3 0x83 poll htop
39965 344785 95745 0 3 0x100083 ttyin ksh
95745 351284 1 0 3 0x100089 select mosh-server
7535 104948 16008 0 3 0x10008b pause ksh
16008 466890 1 0 3 0x100089 select mosh-server
16654 508938 1 0 7 0x100083 ksh
25087 268098 1 0 3 0x100098 poll cron
51059 456262 19550 95 3 0x100092 kqread smtpd
31686 324677 19550 103 3 0x100092 kqread smtpd
22353 53606 19550 95 3 0x100092 kqread smtpd
74951 331678 19550 95 3 0x100092 kqread smtpd
6646 464953 19550 95 3 0x100092 kqread smtpd
53434 362190 19550 95 3 0x100092 kqread smtpd
19550 373371 1 0 3 0x100080 kqread smtpd
55359 293810 1 77 3 0x100090 poll dhcpd
87242 21606 1 0 3 0x80 select sshd
51388 431451 16457 101 3 0x100090 kqread iked
82712 282286 16457 101 3 0x100090 kqread iked
23782 274751 16457 101 3 0x100090 kqread iked
16457 361627 1 0 3 0x100080 kqread iked
27648 102361 95070 83 3 0x100092 poll ntpd
95070 109537 11620 83 3 0x100092 poll ntpd
11620 111885 1 0 3 0x100080 poll ntpd
23169 65903 1850 74 3 0x100092 bpf pflogd
1850 375622 1 0 3 0x80 netio pflogd
99139 498941 51182 73 3 0x100090 kqread syslogd
51182 102487 1 0 3 0x100082 netio syslogd
68119 374605 56610 115 3 0x100092 kqread slaacd
43776 523076 56610 115 3 0x100092 kqread slaacd
56610 322424 1 0 3 0x80 kqread slaacd
43137 394569 0 0 3 0x14200 pgzero zerothread
53305 334341 0 0 3 0x14200 aiodoned aiodoned
63144 303615 0 0 3 0x14200 syncer update
88096 373677 0 0 3 0x14200 cleaner cleaner
15543 49276 0 0 3 0x14200 reaper reaper
12120 286967 0 0 3 0x14200 pgdaemon pagedaemon
*49727 296965 0 0 7 0x14200 crynlk
14971 159136 0 0 3 0x14200 bored crypto
98920 485420 0 0 3 0x14200 bored sensors
51586 110870 0 0 3 0x14200 mmctsk sdmmc0
2729 497936 0 0 3 0x14200 usbtsk usbtask
95166 187161 0 0 3 0x14200 usbatsk usbatsk
80816 453279 0 0 3 0x40014200 acpi0 acpi0
48742 108334 0 0 3 0x40014200 idle3
85015 501132 0 0 7 0x40014200 idle2
20772 82189 0 0 7 0x40014200 idle1
92184 75718 0 0 3 0x14200 bored softnet
59622 302812 0 0 3 0x14200 bored systqmp
33785 72401 0 0 3 0x14200 bored systq
99411 496758 0 0 3 0x40014200 bored softclock
16653 396432 0 0 3 0x40014200 idle0
90556 360065 0 0 3 0x14200 bored sbar
1 390464 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
-TimS
--
Tim Stewart
-----------
Mail: t...@stoo.org
Matrix: @tim:stoo.org
