On Mon, Oct 5, 2020 at 12:22 PM Scott Seekamp <[email protected]> wrote: > > I had a similar speed drop on an Edge Router 4. I don’t know if it’s the same > situation on the Lite, but I believe it’s expected due to hardware > acceleration support (or lack of) and single core performance on the pf side.
I read somewhere that this drop can happen even with the factory OS - the routing is handled by an ASIC (which is how they push near-gigabit forwarding speeds) but if you do any sort of filtering, it falls back to software routing. Since the ASIC is black box voodoo, OpenBSD will always use the CPU for routing. > > Scott > > > On Oct 4, 2020, at 17:24, Amarendra Godbole <[email protected]> > > wrote: > > > > Sorry I forgot including "ifconfig" output: > > > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768 > > index 5 priority 0 llprio 3 > > groups: lo > > inet6 ::1 prefixlen 128 > > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 > > inet 127.0.0.1 netmask 0xff000000 > > > > cnmac0: flags=808843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF4> mtu > > 1500 > > lladdr a8:28:dc:cc:2e:6f > > index 1 priority 0 llprio 3 > > groups: egress > > media: Ethernet autoselect (1000baseT full-duplex,master) > > status: active > > inet 73.xx.xx.xx netmask 0xfffffe00 broadcast 73.xx.xx.255 > > > > cnmac1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > > mtu 1500 > > lladdr 78:8a:20:46:a8:c1 > > index 2 priority 0 llprio 3 > > media: Ethernet autoselect (1000baseT full-duplex) > > status: active > > > > cnmac2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > > mtu 1500 > > lladdr 78:8a:20:46:a8:c2 > > index 3 priority 0 llprio 3 > > media: Ethernet autoselect (none) > > status: no carrier > > enc0: flags=0<> > > index 4 priority 0 llprio 3 > > groups: enc > > status: active > > > > bridge0: flags=41<UP,RUNNING> > > index 6 llprio 3 > > groups: bridge > > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp > > cnmac2 flags=7<LEARNING,DISCOVER,BLOCKNONIP> > > port 3 ifpriority 0 ifcost 0 > > cnmac1 flags=7<LEARNING,DISCOVER,BLOCKNONIP> > > port 2 ifpriority 0 ifcost 0 > > vether0 flags=7<LEARNING,DISCOVER,BLOCKNONIP> > > port 7 ifpriority 0 ifcost 0 > > > > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 > > lladdr fe:e1:ba:d0:c8:a9 > > index 7 priority 0 llprio 3 > > groups: vether > > media: Ethernet autoselect > > status: active > > inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255 > > > > pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33136 > > index 8 priority 0 llprio 3 > > groups: pflog > > > >> On Sun, Oct 4, 2020 at 2:22 PM Amarendra Godbole > >> <[email protected]> wrote: > >> > >> Hi misc@ > >> > >> I recently introduced an OpenBSD firewall inline and noticed a > >> reduction in overall download speeds. I am trying to understand why > >> this may be so. The firewall is Ubiquiti ERL running 6.7 release. > >> Internet connection is Comcast xfinity via cable modem, plan 200 > >> Mbits/s down and 10 Mbits/s up. Details follow: > >> > >> 1. config #1: MacBook - Linksys WRT1200AC - xfinity cable modem > >> (speed: ~210 Mbits/s down, 6 Mbits/s up) > >> 2. config #2: MacBook - Linksys WRT1200AC - Ubiquiti ERL - xfinity > >> cable modem (speed: ~90 MBits down, 6 Mbits/s up) > >> 3. config #3 (Line speed): MacBook wired to cable modem (~230 Mbits/s > >> down, ~8 Mbits/s up). > >> > >> Linksys is running latest OpenWrt, and speed tests were run on MacBook > >> connected wired to Linksys. It was difficult to try tcpbench since the > >> setup was cumbersome, and iperf3 public servers end up being busy more > >> often than not (and threads on misc@ indicated iperf3 wasn't as > >> reliable either). Test numbers come from speedtest.net and > >> speed.cloudflare.com. While I realize this speed test is hardly > >> accurate, I have tried to maintain the same configuration (no ERL and > >> inline ERL) to obtain relative numbers. > >> > >> I am trying to understand the reduction from 210 Mbits/s down to 90 > >> Mbits/s down between config #1 and config #2 above. The slowdown is > >> not noticeable to family, so this is more of my intellectual curiosity > >> than screams over a buffering video! :-) > >> > >> Relevant system information (dmesg, etc.) below. All sysctl values > >> attached as sysctl.txt I gathered it by reading similar threads on > >> misc@. If I missed anything, please let me know. Thanks in advance. > >> > >> -Amarendra > >> > >> > >> dmesg: > >> > >> Copyright (c) 1982, 1986, 1989, 1991, 1993 > >> The Regents of the University of California. All rights reserved. > >> Copyright (c) 1995-2020 OpenBSD. All rights reserved. > >> https://www.OpenBSD.org > >> OpenBSD 6.7 (GENERIC.MP) #134: Thu May 7 16:05:06 MDT 2020 > >> [email protected]:/usr/src/sys/arch/octeon/compile/GENERIC.MP > >> real mem = 536870912 (512MB) > >> avail mem = 506740736 (483MB) > >> mainbus0 at root: board 20002 rev 2.18 > >> cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation > >> cpu0: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way > >> cpu1 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation > >> cpu1: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way > >> clock0 at mainbus0: int 5 > >> octcrypto0 at mainbus0 > >> iobus0 at mainbus0 > >> simplebus0 at iobus0: "soc" > >> octciu0 at simplebus0 > >> octsmi0 at simplebus0 > >> octpip0 at simplebus0 > >> octgmx0 at octpip0 interface 0 > >> cnmac0 at octgmx0: RGMII, address 78:8a:20:46:a8:c0 > >> atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2 > >> cnmac1 at octgmx0: RGMII, address 78:8a:20:46:a8:c1 > >> atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2 > >> cnmac2 at octgmx0: RGMII, address 78:8a:20:46:a8:c2 > >> atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2 > >> com0 at simplebus0: ns16550a, 64 byte fifo > >> com0: console > >> dwctwo0 at iobus0 base 0x1180068000000 irq 56 > >> usb0 at dwctwo0: USB revision 2.0 > >> uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev > >> 2.00/1.00 addr 1 > >> octrng0 at iobus0 base 0x1400000000000 irq 0 > >> /dev/ksyms: Symbol table not valid. > >> umass0 at uhub0 port 1 configuration 1 interface 0 "Lexar USB Flash > >> Drive" rev 2.10/11.00 addr 2 > >> umass0: using SCSI over Bulk-Only > >> scsibus0 at umass0: 2 targets, initiator 0 > >> sd0 at scsibus0 targ 1 lun 0: <Lexar, USB Flash Drive, 1100> removable > >> serial.21c40cd1719080003000 > >> sd0: 30526MB, 512 bytes/sector, 62517248 sectors > >> vscsi0 at root > >> scsibus1 at vscsi0: 256 targets > >> softraid0 at root > >> scsibus2 at softraid0: 256 targets > >> boot device: sd0 > >> root on sd0a (2124441bc835a462.a) swap on sd0b dump on sd0b > >> WARNING: No TOD clock, believing file system. > >> WARNING: CHECK AND RESET THE DATE! > >> > >> > >> pftcl -s: > >> > >> match in all scrub (no-df random-id max-mss 1440) > >> block drop in quick on ! cnmac0 inet from xx.xx.xx.xx/23 to any > >> block drop in quick inet from xx.xx.xx.xx to any > >> block drop all > >> pass out quick on egress inet from (vether0:network) to any flags S/SA > >> nat-to (egress) round-robin > >> pass out quick inet all flags S/SA > >> pass in on vether0 inet all flags S/SA > >> pass in on cnmac1 inet all flags S/SA > >> pass in on cnmac2 inet all flags S/SA > >> > >> pfctl -si: > >> > >> Status: Enabled for 3 days 18:14:11 Debug: err > >> > >> Interface Stats for egress IPv4 IPv6 > >> > >> Bytes In 64537867779 0 > >> Bytes Out 7005140381 0 > >> Packets In > >> Passed 56024370 0 > >> Blocked 43050 0 > >> Packets Out > >> Passed 22271061 0 > >> Blocked 0 0 > >> > >> State Table Total Rate > >> current entries 847 > >> half-open tcp 23 > >> searches 158989755 489.4/s > >> inserts 1095307 3.4/s > >> removals 1094460 3.4/s > >> Counters > >> match 1343178 4.1/s > >> bad-offset 0 0.0/s > >> fragment 0 0.0/s > >> short 1 0.0/s > >> normalize 0 0.0/s > >> memory 0 0.0/s > >> bad-timestamp 0 0.0/s > >> congestion 41 0.0/s > >> ip-option 26164 0.1/s > >> proto-cksum 0 0.0/s > >> state-mismatch 10758 0.0/s > >> state-insert 0 0.0/s > >> state-limit 0 0.0/s > >> src-limit 0 0.0/s > >> synproxy 0 0.0/s > >> translate 0 0.0/s > >> no-route 0 0.0/s > >> > >> pfctl -s memory: > >> > >> states hard limit 100000 > >> src-nodes hard limit 10000 > >> frags hard limit 16384 > >> tables hard limit 1000 > >> table-entries hard limit 200000 > >> pktdelay-pkts hard limit 10000 > >> > >> The netlivlocks value keeps on increasing regularly: > >> kern.netlivelocks=57911 > >> > >> netstat -m: > >> > >> 1009 mbufs in use: > >> 917 mbufs allocated to data > >> 5 mbufs allocated to packet headers > >> 87 mbufs allocated to socket names and addresses > >> 801/7256 mbuf 2048 byte clusters in use (current/peak) > >> 0/15 mbuf 2112 byte clusters in use (current/peak) > >> 0/24 mbuf 4096 byte clusters in use (current/peak) > >> 0/8 mbuf 8192 byte clusters in use (current/peak) > >> 0/0 mbuf 9216 byte clusters in use (current/peak) > >> 0/0 mbuf 12288 byte clusters in use (current/peak) > >> 0/0 mbuf 16384 byte clusters in use (current/peak) > >> 0/8 mbuf 65536 byte clusters in use (current/peak) > >> 6512/17088/131072 Kbytes allocated to network (current/peak/max) > >> 0 requests for memory denied > >> 0 requests for memory delayed > >> 0 calls to protocol drain routines > >> > >> netstat -i: > >> > >> Name Mtu Network Address Ipkts Ifail Opkts Ofail > >> Colls > >> lo0 32768 <Link> 198 0 198 0 > >> 0 > >> lo0 32768 localhost/1 localhost 198 0 198 0 > >> 0 > >> lo0 32768 fe80::%lo0/ fe80::1%lo0 198 0 198 0 > >> 0 > >> lo0 32768 127/8 localhost 198 0 198 0 > >> 0 > >> cnmac0 1600 <Link> a8:28:dc:cc:2e:6f 56088774 0 22283491 2688 > >> 0 > >> cnmac0 1600 73.231.60/2 c-73-231-60-128.h 56088774 0 22283491 2688 > >> 0 > >> cnmac1 1600 <Link> 78:8a:20:46:a8:c1 23646497 4 56569853 48 > >> 0 > >> cnmac2 1600 <Link> 78:8a:20:46:a8:c2 14823 0 226198 226198 > >> 0 > >> enc0* 0 <Link> 0 0 0 0 > >> 0 > >> bridge0 1500 <Link> 23187238 0 57022219 0 > >> 0 > >> vether0 32768 <Link> fe:e1:ba:d0:c8:a9 23056709 0 56795991 0 > >> 0 > >> vether0 32768 192.168.10/ 192.168.10.1 23056709 0 56795991 0 > >> 0 > >> pflog0 33136 <Link> 0 0 26171 0 > >> 0 > > > -- Aaron Mason - Programmer, open source addict I've taken my software vows - for beta or for worse
