Hi list,
I'm setting up a replacement for a customers' current Alteon Load
Balancers, using OpenBSD, pf, and relayd.
First of all: Thanks, guys, this is faboulous stuff! Having
experiences with Linux' LVS and stuff, this is like a very nice, fresh
breeze... I remember the Haiku that was posted when pf was born. :)
Now, the setup I have has some noteworthinesses. First, I have to
create about 600 tables (one for each VIP) that in turn redirect to
about 2,200 IPs in the backend (those are not real hosts, most of the
hosts have several inet aliases set due to Alteon config necessities).
In the meanwhile I tuned relayd.conf massively using parenting, so
that there are no unnecessary checks being done (most of them are
relatively 'expensive', e.g. 'check ssl' or 'check script', with
scripts doing RADIUS logins, etc).
The interval is set to 10 seconds, what works for me at a load of
about 4 (yes, I read the recent discussion on this). Is that 'too
high'? The machine is mostly idling, vmstat output below.
There's a problem that 'check send' does not seem to work correctly,
for instance. Thusly, I had to code some scripts that check for POP,
IMAP, or FTP banners and give an appropriate return code, the same was
done for SSL wrapped services.
When I use 'check send' [ssl] for one defined service and look at the
tcpdump, I can see that it works correctly, i.e. the request is sent
to the client and the answer of the client (FTP banner, e.g.) is
received by the OpenBSD machine, but relayd says that the check didn't
work correctly. Using shell scripts, it works like a charm, but is
expensive.
For SSL checks (using a shell script that invokes OpenSSL's s_client)
I get the following error message (watching 'relayd -v -n')
21415:error:0906D06C:PEM routines:PEM_read_bio:no start
line:/usr/src/lib/libssl/src/crypto/pem/pem_lib.c:650:Expecting:
TRUSTED CERTIFICATE
The script itself is:
POP3S_SERVER=$1
POP3S_OKAY=`echo "GET /" | openssl s_client -connect $1:995 2>/dev/null | \
awk '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/{ print }' | \
openssl x509 -text -noout | grep Issuer | wc -l`
if [ $POP3S_OKAY -eq 1 ]
then
exit 1
else
exit 0
fi
The certificate is okay, and the test also succeeds. So, this is more
a 'cosmetic' question...
Any hints?
Thank in advance and best regards,
Joe
vstat -m:
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 2761 3383 15390751 1280 102
32 308 332 4576010 640 0
64 1666 1534 1336669 320 95100
128 1148 36 697939 160 10000
256 208 224 3219368 80 15826
512 273 151 1343754 40 24349
1024 245 35 561499 20 55420
2048 1220 4 534068 10 49136
4096 25 3 18484 5 0
8192 6 1 15069580 5 0
16384 6 0 23 5 0
32768 2 0 3 5 0
65536 1 0 1 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, routetbl, sysctl, vnodes, UFS mount, dirhash, ACPI,
in_multi, exec, xform_data, VM swap, UVM amap, UVM aobj, temp
32 devbuf, pcb, routetbl, ifaddr, UFS mount, sem, dirhash, ACPI, proc,
VFS cluster, in_multi, ether_multi, xform_data, VM swap, UVM amap,
temp
64 devbuf, pcb, routetbl, vnodes, sem, dirhash, ACPI, in_multi,
pfkey data, UVM amap, NDP, temp
128 devbuf, routetbl, ifaddr, iov, vnodes, dirhash, ACPI, NFS srvsock,
ttys, inodedep, UVM amap, NDP, temp
256 devbuf, routetbl, ifaddr, sysctl, ioctlops, iov, vnodes, shm, VM map,
dirhash, file desc, NFS daemon, exec, newblk, UVM amap, temp
512 devbuf, pcb, ifaddr, ioctlops, iov, UFS mount, shm, dirhash,
file desc, proc, ttys, exec, UVM amap, temp
1024 devbuf, ioctlops, iov, mount, ACPI, ttys, exec, UVM amap, UVM aobj,
crypto data, temp
2048 devbuf, ifaddr, ioctlops, iov, namecache, UFS mount, proc, VM swap,
UVM amap, temp
4096 devbuf, ioctlops, iov, pagedep, UVM amap, memdesc, temp
8192 devbuf, iov, MSDOSFS mount, temp
16384 NFS node, namecache, UFS quota, UFS mount, ISOFS mount, inodedep,
indirdep
32768 devbuf
65536 namecache
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 2019 2324K 2324K 39322K 2090 0 0
16,32,64,128,256,512,1024,2048,4096,8192,32768
pcb 38 4K 5K 39322K 65124 0 0 16,32,64,512
routetbl 865 98K 107K 39322K 25779 0 0 16,32,64,128,256
ifaddr 72 14K 14K 39322K 73 0 0
32,128,256,512,2048
sysctl 2 1K 1K 39322K 2 0 0 16,256
ioctlops 0 0K 4K 39322K 173421 0 0
256,512,1024,2048,4096
iov 0 0K 8K 39322K 61642 0 0
128,256,512,1024,2048,4096,8192
mount 4 4K 4K 39322K 4 0 0 1024
NFS node 1 16K 16K 39322K 1 0 0 16384
vnodes 49 8K 97K 39322K 5894 0 0 16,64,128,256
namecache 3 82K 82K 39322K 3 0 0 2048,16384,65536
UFS quota 1 16K 16K 39322K 1 0 0 16384
UFS mount 17 35K 35K 39322K 17 0 0
16,32,512,2048,16384
shm 2 1K 1K 39322K 2 0 0 256,512
VM map 2 1K 1K 39322K 2 0 0 256
sem 2 1K 1K 39322K 2 0 0 32,64
dirhash 186 35K 43K 39322K 447 0 0
16,32,64,128,256,512
ACPI 701 41K 45K 39322K 2540 0 0
16,32,64,128,1024
file desc 1 1K 3K 39322K 53838 0 0 256,512
proc 12 5K 5K 39322K 12 0 0 32,512,2048
VFS cluster 0 0K 1K 39322K 34 0 0 32
NFS srvsock 1 1K 1K 39322K 1 0 0 128
NFS daemon 1 1K 1K 39322K 1 0 0 256
in_multi 43 2K 2K 39322K 43 0 0 16,32,64
ether_multi 12 1K 1K 39322K 12 0 0 32
ISOFS mount 1 16K 16K 39322K 1 0 0 16384
MSDOSFS mount 1 8K 8K 39322K 1 0 0 8192
ttys 414 259K 259K 39322K 414 0 0 128,512,1024
exec 0 0K 2K 39322K 729130 0 0 16,256,512,1024
pfkey data 1 1K 1K 39322K 2 0 0 64
xform_data 0 0K 1K 39322K 156310 0 0 16,32
pagedep 1 4K 4K 39322K 1 0 0 4096
inodedep 1 16K 17K 39322K 34 0 0 128,16384
newblk 1 1K 1K 39322K 1 0 0 256
indirdep 0 0K 16K 39322K 17 0 0 16384
VM swap 1 1K 3K 39322K 4 0 0 16,32,2048
UVM amap 2988 167K 387K 39322K 21333380 0 0
16,32,64,128,256,512,1024,2048,4096
UVM aobj 2 2K 2K 39322K 2 0 0 16,1024
memdesc 1 4K 4K 39322K 1 0 0 4096
crypto data 1 1K 1K 39322K 1 0 0 1024
NDP 11 1K 1K 39322K 13 0 0 64,128
temp 415 389K 397K 39322K 20137886 0 0
16,32,64,128,256,512,1024,2048,4096,8192
Memory Totals: In Use Free Requests
3547K 359K 42748183
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
extentpl 20 239 0 39 1 0 1 1 0 8 0
phpool 40 961 0 415 5 0 5 5 0 8 0
pmappl 80 439814 0 39 3 0 3 3 0 8 2
vmsppl 188 439814 0 39 7 0 7 7 0 8 5
vmmpepl 88 72975166 0 3692 208 0 208 208 0 179 126
vmmpekpl 88 1551605 0 14 2 0 2 2 0 8 1
aobjpl 52 1 0 1 1 0 1 1 0 8 0
amappl 44 20847488 0 2914 75 0 75 75 0 45 42
anonpl 16 33329130 0 4014 40 0 40 40 0 125 17
bufpl 144 58582 0 12224 469 9 460 469 0 8 8
mbpl 256 7496735 0 160 23 0 23 23 1 384 10
mcl2k 2048 1432711 0 49 47 0 47 47 4 3072 20
sockpl 212 353804 0 127 18 7 11 16 0 8 3
procpl 348 439825 0 50 13 0 13 13 0 8 8
processpl 24 439825 0 50 1 0 1 1 0 8 0
zombiepl 72 439775 0 0 2 0 2 2 0 8 2
ucredpl 80 105414 0 17 1 0 1 1 0 8 0
pgrppl 24 2035 0 28 1 0 1 1 0 8 0
sessionpl 48 1215 0 22 1 0 1 1 0 8 0
pcredpl 24 439825 0 50 1 0 1 1 0 8 0
lockfpl 56 166 0 2 1 0 1 1 0 8 0
filepl 88 4702237 0 122 9 0 9 9 0 8 5
fdescpl 300 439815 0 40 10 0 10 10 0 8 6
pipepl 72 666370 0 12 4 0 4 4 0 8 3
kqueuepl 192 96 0 3 1 0 1 1 0 8 0
knotepl 64 3516384 0 15 2 0 2 2 0 8 1
sigapl 316 439814 0 39 11 0 11 11 0 8 7
wdcspl 96 50646 0 0 1 0 1 1 0 8 1
namei 1024 7359254 0 0 2 0 2 2 0 8 2
vnodes 148 5927 0 5927 220 0 220 220 0 8 0
nchpl 72 2963 0 2963 53 0 53 53 0 8 0
ffsino 184 122944 0 5922 270 0 270 270 0 8 0
dino1pl 128 122944 0 5922 192 0 192 192 0 8 0
pagedeppl 68 149 0 0 1 0 1 1 0 8 1
inodedeppl 84 363 0 0 1 0 1 1 0 8 1
newblkpl 36 653 0 0 1 0 1 1 0 8 1
bmsafemappl 32 193 0 0 1 0 1 1 0 8 1
allocdirectpl 76 627 0 0 2 0 2 2 0 8 2
indirdeppl 28 22 0 0 1 0 1 1 0 8 1
allocindirpl 60 26 0 0 1 0 1 1 0 8 1
freefragpl 36 91 0 0 1 0 1 1 0 8 1
freeblkspl 168 133 0 0 1 0 1 1 0 8 1
freefilepl 28 186 0 0 1 0 1 1 0 8 1
diraddpl 32 209 0 0 1 0 1 1 0 8 1
mkdirpl 28 10 0 0 1 0 1 1 0 8 1
dirrempl 32 199 0 0 1 0 1 1 0 8 1
dirhash 1024 612 0 252 84 0 84 84 0 128 20
pfrulepl 852 4156 0 179 536 491 45 45 0 8 0
pfstatepl 216 163055 0 452 50 0 50 50 0 556 24
pfstatekeypl 72 163055 0 452 26 16 10 16 0 8 1
pfstateitempl 12 163055 0 452 3 0 3 3 0 8 1
pfpooladdrpl 68 4067 0 172 3 0 3 3 0 8 0
pfrktable 1240 11043 0 356 119 0 119 119 0 5000 0
pfrkentry 92 10894 0 541 37 24 13 13 0 8 0
pfosfpen 108 8352 0 696 140 121 19 19 0 8 0
pfosfp 28 4884 0 407 3 0 3 3 0 8 0
rtentpl 116 1659 0 75 6 0 6 6 0 8 1
tcpcbpl 400 230742 0 92 68 55 13 27 0 8 1
tcpqepl 16 19029 0 0 1 0 1 1 0 13 1
sackhlpl 20 2 0 0 1 0 1 1 0 163 1
synpl 184 1035 0 0 1 0 1 1 0 8 1
plimitpl 152 178 0 11 1 0 1 1 0 8 0
inpcbpl 224 288751 0 100 17 7 10 15 0 8 3
In use 6522K, total allocated 32912K; utilization 19.8%
dmesg:
OpenBSD 4.6-current (GENERIC) #86: Tue Jul 28 23:51:07 MDT 2009
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 2.80GHz ("GenuineIntel" 686-class) 2.81 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,xTPR
real mem = 2146795520 (2047MB)
avail mem = 2067046400 (1971MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 01/04/06, BIOS32 rev. 0 @
0xffe90, SMBIOS rev. 2.3 @ 0xfb030 (83 entries)
bios0: vendor Dell Computer Corporation version "A06" date 01/04/2006
bios0: Dell Computer Corporation PowerEdge 750
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP APIC SPCR
acpi0: wakeup devices PCI0(S5) PCI1(S5) PCI2(S5) PCI3(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 200MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins
ioapic1: misconfigured as apic 0, remapped to apid 3
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 3 (PCI1)
acpiprt2 at acpi0: bus 2 (PCI2)
acpiprt3 at acpi0: bus 1 (PCI3)
acpicpu0 at acpi0
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xec000/0x4000!
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82875P Host" rev 0x02
ppb0 at pci0 dev 3 function 0 "Intel 82875P CSA" rev 0x02
pci1 at ppb0 bus 1
em0 at pci1 dev 1 function 0 "Intel PRO/1000CT (82547GI)" rev 0x00:
apic 2 int 18 (irq 10), address 00:c0:9f:46:39:87
ppb1 at pci0 dev 28 function 0 "Intel 6300ESB PCIX" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x0a
pci3 at ppb2 bus 3
em1 at pci3 dev 2 function 0 "Intel PRO/1000MT (82541GI)" rev 0x00:
apic 2 int 21 (irq 7), address 00:c0:9f:46:39:88
xl0 at pci3 dev 3 function 0 "3Com 3c905B 100Base-TX" rev 0x64: apic 2
int 22 (irq 5), address 00:50:da:43:cf:11
bmtphy0 at xl0 phy 24: 3C905B internal PHY, rev. 0
vga1 at pci3 dev 14 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 6300ESB LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 6300ESB SATA" rev 0x02: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
wd0 at pciide0 channel 0 drive 0: <TS8GSSD25-S>
wd0: 1-sector PIO, LBA, 7627MB, 15621984 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
ichiic0 at pci0 dev 31 function 3 "Intel 6300ESB SMBus" rev 0x02: SMBus disabled
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
mtrr: Pentium Pro MTRR support
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
