Sorry I forgot including "ifconfig" output: lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768 index 5 priority 0 llprio 3 groups: lo inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000
cnmac0: flags=808843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF4> mtu 1500 lladdr a8:28:dc:cc:2e:6f index 1 priority 0 llprio 3 groups: egress media: Ethernet autoselect (1000baseT full-duplex,master) status: active inet 73.xx.xx.xx netmask 0xfffffe00 broadcast 73.xx.xx.255 cnmac1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 78:8a:20:46:a8:c1 index 2 priority 0 llprio 3 media: Ethernet autoselect (1000baseT full-duplex) status: active cnmac2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 lladdr 78:8a:20:46:a8:c2 index 3 priority 0 llprio 3 media: Ethernet autoselect (none) status: no carrier enc0: flags=0<> index 4 priority 0 llprio 3 groups: enc status: active bridge0: flags=41<UP,RUNNING> index 6 llprio 3 groups: bridge priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp cnmac2 flags=7<LEARNING,DISCOVER,BLOCKNONIP> port 3 ifpriority 0 ifcost 0 cnmac1 flags=7<LEARNING,DISCOVER,BLOCKNONIP> port 2 ifpriority 0 ifcost 0 vether0 flags=7<LEARNING,DISCOVER,BLOCKNONIP> port 7 ifpriority 0 ifcost 0 vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 lladdr fe:e1:ba:d0:c8:a9 index 7 priority 0 llprio 3 groups: vether media: Ethernet autoselect status: active inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255 pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33136 index 8 priority 0 llprio 3 groups: pflog On Sun, Oct 4, 2020 at 2:22 PM Amarendra Godbole <[email protected]> wrote: > > Hi misc@ > > I recently introduced an OpenBSD firewall inline and noticed a > reduction in overall download speeds. I am trying to understand why > this may be so. The firewall is Ubiquiti ERL running 6.7 release. > Internet connection is Comcast xfinity via cable modem, plan 200 > Mbits/s down and 10 Mbits/s up. Details follow: > > 1. config #1: MacBook - Linksys WRT1200AC - xfinity cable modem > (speed: ~210 Mbits/s down, 6 Mbits/s up) > 2. config #2: MacBook - Linksys WRT1200AC - Ubiquiti ERL - xfinity > cable modem (speed: ~90 MBits down, 6 Mbits/s up) > 3. config #3 (Line speed): MacBook wired to cable modem (~230 Mbits/s > down, ~8 Mbits/s up). > > Linksys is running latest OpenWrt, and speed tests were run on MacBook > connected wired to Linksys. It was difficult to try tcpbench since the > setup was cumbersome, and iperf3 public servers end up being busy more > often than not (and threads on misc@ indicated iperf3 wasn't as > reliable either). Test numbers come from speedtest.net and > speed.cloudflare.com. While I realize this speed test is hardly > accurate, I have tried to maintain the same configuration (no ERL and > inline ERL) to obtain relative numbers. > > I am trying to understand the reduction from 210 Mbits/s down to 90 > Mbits/s down between config #1 and config #2 above. The slowdown is > not noticeable to family, so this is more of my intellectual curiosity > than screams over a buffering video! :-) > > Relevant system information (dmesg, etc.) below. All sysctl values > attached as sysctl.txt I gathered it by reading similar threads on > misc@. If I missed anything, please let me know. Thanks in advance. > > -Amarendra > > > dmesg: > > Copyright (c) 1982, 1986, 1989, 1991, 1993 > The Regents of the University of California. All rights reserved. > Copyright (c) 1995-2020 OpenBSD. All rights reserved. https://www.OpenBSD.org > OpenBSD 6.7 (GENERIC.MP) #134: Thu May 7 16:05:06 MDT 2020 > [email protected]:/usr/src/sys/arch/octeon/compile/GENERIC.MP > real mem = 536870912 (512MB) > avail mem = 506740736 (483MB) > mainbus0 at root: board 20002 rev 2.18 > cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation > cpu0: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way > cpu1 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation > cpu1: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way > clock0 at mainbus0: int 5 > octcrypto0 at mainbus0 > iobus0 at mainbus0 > simplebus0 at iobus0: "soc" > octciu0 at simplebus0 > octsmi0 at simplebus0 > octpip0 at simplebus0 > octgmx0 at octpip0 interface 0 > cnmac0 at octgmx0: RGMII, address 78:8a:20:46:a8:c0 > atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2 > cnmac1 at octgmx0: RGMII, address 78:8a:20:46:a8:c1 > atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2 > cnmac2 at octgmx0: RGMII, address 78:8a:20:46:a8:c2 > atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2 > com0 at simplebus0: ns16550a, 64 byte fifo > com0: console > dwctwo0 at iobus0 base 0x1180068000000 irq 56 > usb0 at dwctwo0: USB revision 2.0 > uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev > 2.00/1.00 addr 1 > octrng0 at iobus0 base 0x1400000000000 irq 0 > /dev/ksyms: Symbol table not valid. > umass0 at uhub0 port 1 configuration 1 interface 0 "Lexar USB Flash > Drive" rev 2.10/11.00 addr 2 > umass0: using SCSI over Bulk-Only > scsibus0 at umass0: 2 targets, initiator 0 > sd0 at scsibus0 targ 1 lun 0: <Lexar, USB Flash Drive, 1100> removable > serial.21c40cd1719080003000 > sd0: 30526MB, 512 bytes/sector, 62517248 sectors > vscsi0 at root > scsibus1 at vscsi0: 256 targets > softraid0 at root > scsibus2 at softraid0: 256 targets > boot device: sd0 > root on sd0a (2124441bc835a462.a) swap on sd0b dump on sd0b > WARNING: No TOD clock, believing file system. > WARNING: CHECK AND RESET THE DATE! > > > pftcl -s: > > match in all scrub (no-df random-id max-mss 1440) > block drop in quick on ! cnmac0 inet from xx.xx.xx.xx/23 to any > block drop in quick inet from xx.xx.xx.xx to any > block drop all > pass out quick on egress inet from (vether0:network) to any flags S/SA > nat-to (egress) round-robin > pass out quick inet all flags S/SA > pass in on vether0 inet all flags S/SA > pass in on cnmac1 inet all flags S/SA > pass in on cnmac2 inet all flags S/SA > > pfctl -si: > > Status: Enabled for 3 days 18:14:11 Debug: err > > Interface Stats for egress IPv4 IPv6 > > Bytes In 64537867779 0 > Bytes Out 7005140381 0 > Packets In > Passed 56024370 0 > Blocked 43050 0 > Packets Out > Passed 22271061 0 > Blocked 0 0 > > State Table Total Rate > current entries 847 > half-open tcp 23 > searches 158989755 489.4/s > inserts 1095307 3.4/s > removals 1094460 3.4/s > Counters > match 1343178 4.1/s > bad-offset 0 0.0/s > fragment 0 0.0/s > short 1 0.0/s > normalize 0 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 41 0.0/s > ip-option 26164 0.1/s > proto-cksum 0 0.0/s > state-mismatch 10758 0.0/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > translate 0 0.0/s > no-route 0 0.0/s > > pfctl -s memory: > > states hard limit 100000 > src-nodes hard limit 10000 > frags hard limit 16384 > tables hard limit 1000 > table-entries hard limit 200000 > pktdelay-pkts hard limit 10000 > > The netlivlocks value keeps on increasing regularly: > kern.netlivelocks=57911 > > netstat -m: > > 1009 mbufs in use: > 917 mbufs allocated to data > 5 mbufs allocated to packet headers > 87 mbufs allocated to socket names and addresses > 801/7256 mbuf 2048 byte clusters in use (current/peak) > 0/15 mbuf 2112 byte clusters in use (current/peak) > 0/24 mbuf 4096 byte clusters in use (current/peak) > 0/8 mbuf 8192 byte clusters in use (current/peak) > 0/0 mbuf 9216 byte clusters in use (current/peak) > 0/0 mbuf 12288 byte clusters in use (current/peak) > 0/0 mbuf 16384 byte clusters in use (current/peak) > 0/8 mbuf 65536 byte clusters in use (current/peak) > 6512/17088/131072 Kbytes allocated to network (current/peak/max) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines > > netstat -i: > > Name Mtu Network Address Ipkts Ifail Opkts Ofail > Colls > lo0 32768 <Link> 198 0 198 0 > 0 > lo0 32768 localhost/1 localhost 198 0 198 0 > 0 > lo0 32768 fe80::%lo0/ fe80::1%lo0 198 0 198 0 > 0 > lo0 32768 127/8 localhost 198 0 198 0 > 0 > cnmac0 1600 <Link> a8:28:dc:cc:2e:6f 56088774 0 22283491 2688 > 0 > cnmac0 1600 73.231.60/2 c-73-231-60-128.h 56088774 0 22283491 2688 > 0 > cnmac1 1600 <Link> 78:8a:20:46:a8:c1 23646497 4 56569853 48 > 0 > cnmac2 1600 <Link> 78:8a:20:46:a8:c2 14823 0 226198 226198 > 0 > enc0* 0 <Link> 0 0 0 0 > 0 > bridge0 1500 <Link> 23187238 0 57022219 0 > 0 > vether0 32768 <Link> fe:e1:ba:d0:c8:a9 23056709 0 56795991 0 > 0 > vether0 32768 192.168.10/ 192.168.10.1 23056709 0 56795991 0 > 0 > pflog0 33136 <Link> 0 0 26171 0 > 0
