On 03-07-19 18:07, Willem de Bruijn wrote:
On Wed, Jul 3, 2019 at 7:08 AM Frank de Brabander <debraban...@gmail.com> wrote:
In commit 5cd8d46e a fix was applied for data corruption in
tpacket_snd. A selftest was added in commit 358be656 which
validates this fix.
Unfortunately this bug still persists, although since this fix less
likely to trigger. This bug was initially observed using a PACKET_MMAP
application, but can also be seen by tweaking the kernel selftest.
By tweaking the selftest txring_overwrite.c to run
as an infinite loop, the data corruption will still trigger. It
seems to occur faster by generating interrupts (e.g. by plugging
in USB devices). Tested with kernel version 5.2-RC7.
Cause for this bug is still unclear.
The cause of the original bug is well understood.
The issue you report I expect is due to background traffic. And more
about the test than the kernel implementation.
Can you reproduce the issue when running the modified test in a
network namespace (./in_netns.sh ./txring_overwrite)?
I observe the issue report outside that, but not inside. That implies
that what we're observing is random background traffic. The modified
test then drops the unexpected packet because it mismatches on length.
As a result the next read (the test always sends two packets, then
reads both) will report a data mismatch. Because it is reading the
first test packet, but expecting the second. Output with a bit more
data:
count: 200
count: 300
count: 400
count: 500
read: 90B != 100B
wrong pattern: 0x61 != 0x62
count: 600
count: 700
count: 800
read: 90B != 100B
wrong pattern: 0x61 != 0x62
count: 900
read: 90B != 100B
wrong pattern: 0x61 != 0x62
Notice the clear pattern.
This does not trigger inside a network namespace, which is how
kselftest invokes txring_override (from run_afpackettests).
I'm also able to reproduce the issue inside a network namespace.
I've added the extra logging, as seen in your output, for
mismatches on length. Running the test without ./in_netns.sh
is indeed as you describe:
read error: 66 != 100
wrong pattern: 0x61 != 0x62
read error: 66 != 100
wrong pattern: 0x61 != 0x62
read error: 74 != 100
read error: 66 != 100
wrong pattern: 0x53 != 0x61
wrong pattern: 0x53 != 0x62
read error: 66 != 100
read error: 66 != 100
read error: 66 != 100
wrong pattern: 0x61 != 0x62
read error: 95 != 100
read error: 95 != 100
wrong pattern: 0xffffffbe != 0x61
wrong pattern: 0x61 != 0x62
read error: 66 != 100
But even when running the test with ./in_netns.sh it shows
"wrong pattern", this time without length mismatches:
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
As already mentioned, it seems to trigger mainly (only ?) when
an USB device is connected. The PC I'm testing this on has an
USB hub with many ports and connected devices. When connecting
this USB hub, the amount of "wrong pattern" errors that are
shown seems to correlate to the amount of new devices
that the kernel should detect. Connecting in a single USB device
also triggers the error, but not on every attempt.
Unfortunately have not found any other way to force the
error to trigger. E.g. running stress-ng to generate CPU load or
timer interrupts does not seem to have any impact.
