Hi Michael. I have some new findings. I installed NetBSD in a KVM hypervisor in a Japanese datacenter this time. NetBSD can reach very high speeds compared to previous attempts (40 MB/s), and even though it still collapses here and there, it is much more stable. It makes me think that the US datacenter has a "dirtier" network path, or is applying some kind of traffic shaping, or both. Linux has no issues on the same node, and FreeBSD/OpenBSD have some trouble but not as much as NetBSD. I can only imagine they just have TCP stacks that are less sensitive to packet loss.
It's worth noting that UDP transfers show 0% loss every time. So maybe it's not loss specifically but something else that affects TCP transfers. Another interesting finding is that, at least for me, all operating systems suffer negatively from VirtIO NIC. They all improve their speeds when switching to E1000, which goes against what I often hear about VirtIO being faster due to paravirtualization. I made a general overview here: https://u.omaera.org/e1000.png All operating systems simply do better with E1000, but VirtIO caps NetBSD in particular to 1 MB/s even when tuned, so while other OSes do "okay" with VirtIO, NetBSD has serious trouble with it. On 3/7/26 03:41, Michael van Elst wrote: > It would be interesting to see how that affects your setup. I tried your patchset. Sadly I don't think I saw much improvement. Here's a worst case measurement (xpl files inside): https://u.omaera.org/xpl_us2.tar.gz The Japanese server does much better: almost no drops, often stable 40 MB/s (!). I was planning to move to that server anyway so this works for me. I will publish these results to my blog soon. === I found a new issue though: receiving files with SCP from the server->client is fast, but when sending files to the server it only sends 255KB and then it stalls for about half a minute, then sends the rest at a slow pace (around 200-300 KB/s). If I send it using legacy SCP (-O flag) the initial stall doesn't happen, but it's still slow. The server has big recv buffers and it has no trouble downloading (downloads using ftp(1) are very fast), it's just SCP. These are my tunables: net.inet6.tcp6.recvbuf_inc=131052 net.inet6.tcp6.sendbuf_inc=131052 net.inet6.tcp6.recvbuf_max=5042880 net.inet6.tcp6.sendbuf_max=5042880 net.inet6.tcp6.congctl.selected=cubic net.inet.tcp.recvbuf_inc=131052 net.inet.tcp.sendbuf_inc=131052 net.inet.tcp.recvbuf_max=5042880 net.inet.tcp.sendbuf_max=5042880 net.inet.tcp.congctl.selected=cubic kern.sbmax=5042880 Do you have an idea of what could be happening? Thanks for all the help. - z411
