hi,
do you have more details, like the tested relayd.conf lines, about the
'check send' problem?
reyk
On Wed, Aug 05, 2009 at 02:54:58PM +0200, Nice Daemon wrote:
> Hi list,
>
> I'm setting up a replacement for a customers' current Alteon Load
> Balancers, using OpenBSD, pf, and relayd.
>
> First of all: Thanks, guys, this is faboulous stuff! Having
> experiences with Linux' LVS and stuff, this is like a very nice, fresh
> breeze... I remember the Haiku that was posted when pf was born. :)
>
> Now, the setup I have has some noteworthinesses. First, I have to
> create about 600 tables (one for each VIP) that in turn redirect to
> about 2,200 IPs in the backend (those are not real hosts, most of the
> hosts have several inet aliases set due to Alteon config necessities).
>
> In the meanwhile I tuned relayd.conf massively using parenting, so
> that there are no unnecessary checks being done (most of them are
> relatively 'expensive', e.g. 'check ssl' or 'check script', with
> scripts doing RADIUS logins, etc).
>
> The interval is set to 10 seconds, what works for me at a load of
> about 4 (yes, I read the recent discussion on this). Is that 'too
> high'? The machine is mostly idling, vmstat output below.
>
> There's a problem that 'check send' does not seem to work correctly,
> for instance. Thusly, I had to code some scripts that check for POP,
> IMAP, or FTP banners and give an appropriate return code, the same was
> done for SSL wrapped services.
>
> When I use 'check send' [ssl] for one defined service and look at the
> tcpdump, I can see that it works correctly, i.e. the request is sent
> to the client and the answer of the client (FTP banner, e.g.) is
> received by the OpenBSD machine, but relayd says that the check didn't
> work correctly. Using shell scripts, it works like a charm, but is
> expensive.
>
> For SSL checks (using a shell script that invokes OpenSSL's s_client)
> I get the following error message (watching 'relayd -v -n')
>
> 21415:error:0906D06C:PEM routines:PEM_read_bio:no start
> line:/usr/src/lib/libssl/src/crypto/pem/pem_lib.c:650:Expecting:
> TRUSTED CERTIFICATE
>
> The script itself is:
>
> POP3S_SERVER=$1
>
> POP3S_OKAY=`echo "GET /" | openssl s_client -connect $1:995 2>/dev/null | \
> awk '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/{ print }' | \
> openssl x509 -text -noout | grep Issuer | wc -l`
>
> if [ $POP3S_OKAY -eq 1 ]
> then
> exit 1
> else
> exit 0
> fi
>
> The certificate is okay, and the test also succeeds. So, this is more
> a 'cosmetic' question...
>
> Any hints?
>
> Thank in advance and best regards,
>
> Joe
>
> vstat -m:
>
> Memory statistics by bucket size
> Size In Use Free Requests HighWater Couldfree
> 16 2761 3383 15390751 1280 102
> 32 308 332 4576010 640 0
> 64 1666 1534 1336669 320 95100
> 128 1148 36 697939 160 10000
> 256 208 224 3219368 80 15826
> 512 273 151 1343754 40 24349
> 1024 245 35 561499 20 55420
> 2048 1220 4 534068 10 49136
> 4096 25 3 18484 5 0
> 8192 6 1 15069580 5 0
> 16384 6 0 23 5 0
> 32768 2 0 3 5 0
> 65536 1 0 1 5 0
>
> Memory usage type by bucket size
> Size Type(s)
> 16 devbuf, pcb, routetbl, sysctl, vnodes, UFS mount, dirhash, ACPI,
> in_multi, exec, xform_data, VM swap, UVM amap, UVM aobj, temp
> 32 devbuf, pcb, routetbl, ifaddr, UFS mount, sem, dirhash, ACPI, proc,
> VFS cluster, in_multi, ether_multi, xform_data, VM swap, UVM amap,
> temp
> 64 devbuf, pcb, routetbl, vnodes, sem, dirhash, ACPI, in_multi,
> pfkey data, UVM amap, NDP, temp
> 128 devbuf, routetbl, ifaddr, iov, vnodes, dirhash, ACPI, NFS srvsock,
> ttys, inodedep, UVM amap, NDP, temp
> 256 devbuf, routetbl, ifaddr, sysctl, ioctlops, iov, vnodes, shm, VM
> map,
> dirhash, file desc, NFS daemon, exec, newblk, UVM amap, temp
> 512 devbuf, pcb, ifaddr, ioctlops, iov, UFS mount, shm, dirhash,
> file desc, proc, ttys, exec, UVM amap, temp
> 1024 devbuf, ioctlops, iov, mount, ACPI, ttys, exec, UVM amap, UVM aobj,
> crypto data, temp
> 2048 devbuf, ifaddr, ioctlops, iov, namecache, UFS mount, proc, VM swap,
> UVM amap, temp
> 4096 devbuf, ioctlops, iov, pagedep, UVM amap, memdesc, temp
> 8192 devbuf, iov, MSDOSFS mount, temp
> 16384 NFS node, namecache, UFS quota, UFS mount, ISOFS mount, inodedep,
> indirdep
> 32768 devbuf
> 65536 namecache
>
> Memory statistics by type Type Kern
> Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
> devbuf 2019 2324K 2324K 39322K 2090 0 0
> 16,32,64,128,256,512,1024,2048,4096,8192,32768
> pcb 38 4K 5K 39322K 65124 0 0 16,32,64,512
> routetbl 865 98K 107K 39322K 25779 0 0
> 16,32,64,128,256
> ifaddr 72 14K 14K 39322K 73 0 0
> 32,128,256,512,2048
> sysctl 2 1K 1K 39322K 2 0 0 16,256
> ioctlops 0 0K 4K 39322K 173421 0 0
> 256,512,1024,2048,4096
> iov 0 0K 8K 39322K 61642 0 0
> 128,256,512,1024,2048,4096,8192
> mount 4 4K 4K 39322K 4 0 0 1024
> NFS node 1 16K 16K 39322K 1 0 0 16384
> vnodes 49 8K 97K 39322K 5894 0 0 16,64,128,256
> namecache 3 82K 82K 39322K 3 0 0
> 2048,16384,65536
> UFS quota 1 16K 16K 39322K 1 0 0 16384
> UFS mount 17 35K 35K 39322K 17 0 0
> 16,32,512,2048,16384
> shm 2 1K 1K 39322K 2 0 0 256,512
> VM map 2 1K 1K 39322K 2 0 0 256
> sem 2 1K 1K 39322K 2 0 0 32,64
> dirhash 186 35K 43K 39322K 447 0 0
> 16,32,64,128,256,512
> ACPI 701 41K 45K 39322K 2540 0 0
> 16,32,64,128,1024
> file desc 1 1K 3K 39322K 53838 0 0 256,512
> proc 12 5K 5K 39322K 12 0 0 32,512,2048
> VFS cluster 0 0K 1K 39322K 34 0 0 32
> NFS srvsock 1 1K 1K 39322K 1 0 0 128
> NFS daemon 1 1K 1K 39322K 1 0 0 256
> in_multi 43 2K 2K 39322K 43 0 0 16,32,64
> ether_multi 12 1K 1K 39322K 12 0 0 32
> ISOFS mount 1 16K 16K 39322K 1 0 0 16384
> MSDOSFS mount 1 8K 8K 39322K 1 0 0 8192
> ttys 414 259K 259K 39322K 414 0 0 128,512,1024
> exec 0 0K 2K 39322K 729130 0 0
> 16,256,512,1024
> pfkey data 1 1K 1K 39322K 2 0 0 64
> xform_data 0 0K 1K 39322K 156310 0 0 16,32
> pagedep 1 4K 4K 39322K 1 0 0 4096
> inodedep 1 16K 17K 39322K 34 0 0 128,16384
> newblk 1 1K 1K 39322K 1 0 0 256
> indirdep 0 0K 16K 39322K 17 0 0 16384
> VM swap 1 1K 3K 39322K 4 0 0 16,32,2048
> UVM amap 2988 167K 387K 39322K 21333380 0 0
> 16,32,64,128,256,512,1024,2048,4096
> UVM aobj 2 2K 2K 39322K 2 0 0 16,1024
> memdesc 1 4K 4K 39322K 1 0 0 4096
> crypto data 1 1K 1K 39322K 1 0 0 1024
> NDP 11 1K 1K 39322K 13 0 0 64,128
> temp 415 389K 397K 39322K 20137886 0 0
> 16,32,64,128,256,512,1024,2048,4096,8192
>
> Memory Totals: In Use Free Requests
> 3547K 359K 42748183
> Memory resource pool statistics
> Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg
> Idle
> extentpl 20 239 0 39 1 0 1 1 0 8
> 0
> phpool 40 961 0 415 5 0 5 5 0 8
> 0
> pmappl 80 439814 0 39 3 0 3 3 0 8
> 2
> vmsppl 188 439814 0 39 7 0 7 7 0 8
> 5
> vmmpepl 88 72975166 0 3692 208 0 208 208 0 179
> 126
> vmmpekpl 88 1551605 0 14 2 0 2 2 0 8
> 1
> aobjpl 52 1 0 1 1 0 1 1 0 8
> 0
> amappl 44 20847488 0 2914 75 0 75 75 0 45
> 42
> anonpl 16 33329130 0 4014 40 0 40 40 0 125
> 17
> bufpl 144 58582 0 12224 469 9 460 469 0 8
> 8
> mbpl 256 7496735 0 160 23 0 23 23 1 384
> 10
> mcl2k 2048 1432711 0 49 47 0 47 47 4 3072
> 20
> sockpl 212 353804 0 127 18 7 11 16 0 8
> 3
> procpl 348 439825 0 50 13 0 13 13 0 8
> 8
> processpl 24 439825 0 50 1 0 1 1 0 8
> 0
> zombiepl 72 439775 0 0 2 0 2 2 0 8
> 2
> ucredpl 80 105414 0 17 1 0 1 1 0 8
> 0
> pgrppl 24 2035 0 28 1 0 1 1 0 8
> 0
> sessionpl 48 1215 0 22 1 0 1 1 0 8
> 0
> pcredpl 24 439825 0 50 1 0 1 1 0 8
> 0
> lockfpl 56 166 0 2 1 0 1 1 0 8
> 0
> filepl 88 4702237 0 122 9 0 9 9 0 8
> 5
> fdescpl 300 439815 0 40 10 0 10 10 0 8
> 6
> pipepl 72 666370 0 12 4 0 4 4 0 8
> 3
> kqueuepl 192 96 0 3 1 0 1 1 0 8
> 0
> knotepl 64 3516384 0 15 2 0 2 2 0 8
> 1
> sigapl 316 439814 0 39 11 0 11 11 0 8
> 7
> wdcspl 96 50646 0 0 1 0 1 1 0 8
> 1
> namei 1024 7359254 0 0 2 0 2 2 0 8
> 2
> vnodes 148 5927 0 5927 220 0 220 220 0 8
> 0
> nchpl 72 2963 0 2963 53 0 53 53 0 8
> 0
> ffsino 184 122944 0 5922 270 0 270 270 0 8
> 0
> dino1pl 128 122944 0 5922 192 0 192 192 0 8
> 0
> pagedeppl 68 149 0 0 1 0 1 1 0 8
> 1
> inodedeppl 84 363 0 0 1 0 1 1 0 8
> 1
> newblkpl 36 653 0 0 1 0 1 1 0 8
> 1
> bmsafemappl 32 193 0 0 1 0 1 1 0 8
> 1
> allocdirectpl 76 627 0 0 2 0 2 2 0 8
> 2
> indirdeppl 28 22 0 0 1 0 1 1 0 8
> 1
> allocindirpl 60 26 0 0 1 0 1 1 0 8
> 1
> freefragpl 36 91 0 0 1 0 1 1 0 8
> 1
> freeblkspl 168 133 0 0 1 0 1 1 0 8
> 1
> freefilepl 28 186 0 0 1 0 1 1 0 8
> 1
> diraddpl 32 209 0 0 1 0 1 1 0 8
> 1
> mkdirpl 28 10 0 0 1 0 1 1 0 8
> 1
> dirrempl 32 199 0 0 1 0 1 1 0 8
> 1
> dirhash 1024 612 0 252 84 0 84 84 0 128
> 20
> pfrulepl 852 4156 0 179 536 491 45 45 0 8
> 0
> pfstatepl 216 163055 0 452 50 0 50 50 0 556
> 24
> pfstatekeypl 72 163055 0 452 26 16 10 16 0 8
> 1
> pfstateitempl 12 163055 0 452 3 0 3 3 0 8
> 1
> pfpooladdrpl 68 4067 0 172 3 0 3 3 0 8
> 0
> pfrktable 1240 11043 0 356 119 0 119 119 0 5000
> 0
> pfrkentry 92 10894 0 541 37 24 13 13 0 8
> 0
> pfosfpen 108 8352 0 696 140 121 19 19 0 8
> 0
> pfosfp 28 4884 0 407 3 0 3 3 0 8
> 0
> rtentpl 116 1659 0 75 6 0 6 6 0 8
> 1
> tcpcbpl 400 230742 0 92 68 55 13 27 0 8
> 1
> tcpqepl 16 19029 0 0 1 0 1 1 0 13
> 1
> sackhlpl 20 2 0 0 1 0 1 1 0 163
> 1
> synpl 184 1035 0 0 1 0 1 1 0 8
> 1
> plimitpl 152 178 0 11 1 0 1 1 0 8
> 0
> inpcbpl 224 288751 0 100 17 7 10 15 0 8
> 3
>
> In use 6522K, total allocated 32912K; utilization 19.8%
>
> dmesg:
>
> OpenBSD 4.6-current (GENERIC) #86: Tue Jul 28 23:51:07 MDT 2009
> [email protected]:/usr/src/sys/arch/i386/compile/GENERIC
> cpu0: Intel(R) Pentium(R) 4 CPU 2.80GHz ("GenuineIntel" 686-class) 2.81 GHz
> cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,xTPR
> real mem = 2146795520 (2047MB)
> avail mem = 2067046400 (1971MB)
> mainbus0 at root
> bios0 at mainbus0: AT/286+ BIOS, date 01/04/06, BIOS32 rev. 0 @
> 0xffe90, SMBIOS rev. 2.3 @ 0xfb030 (83 entries)
> bios0: vendor Dell Computer Corporation version "A06" date 01/04/2006
> bios0: Dell Computer Corporation PowerEdge 750
> acpi0 at bios0: rev 0
> acpi0: tables DSDT FACP APIC SPCR
> acpi0: wakeup devices PCI0(S5) PCI1(S5) PCI2(S5) PCI3(S5)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: apic clock running at 200MHz
> cpu at mainbus0: not configured
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
> ioapic0: misconfigured as apic 0, remapped to apid 2
> ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins
> ioapic1: misconfigured as apic 0, remapped to apid 3
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 3 (PCI1)
> acpiprt2 at acpi0: bus 2 (PCI2)
> acpiprt3 at acpi0: bus 1 (PCI3)
> acpicpu0 at acpi0
> bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xec000/0x4000!
> pci0 at mainbus0 bus 0: configuration mode 1 (bios)
> pchb0 at pci0 dev 0 function 0 "Intel 82875P Host" rev 0x02
> ppb0 at pci0 dev 3 function 0 "Intel 82875P CSA" rev 0x02
> pci1 at ppb0 bus 1
> em0 at pci1 dev 1 function 0 "Intel PRO/1000CT (82547GI)" rev 0x00:
> apic 2 int 18 (irq 10), address 00:c0:9f:46:39:87
> ppb1 at pci0 dev 28 function 0 "Intel 6300ESB PCIX" rev 0x02
> pci2 at ppb1 bus 2
> ppb2 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x0a
> pci3 at ppb2 bus 3
> em1 at pci3 dev 2 function 0 "Intel PRO/1000MT (82541GI)" rev 0x00:
> apic 2 int 21 (irq 7), address 00:c0:9f:46:39:88
> xl0 at pci3 dev 3 function 0 "3Com 3c905B 100Base-TX" rev 0x64: apic 2
> int 22 (irq 5), address 00:50:da:43:cf:11
> bmtphy0 at xl0 phy 24: 3C905B internal PHY, rev. 0
> vga1 at pci3 dev 14 function 0 "ATI Rage XL" rev 0x27
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ichpcib0 at pci0 dev 31 function 0 "Intel 6300ESB LPC" rev 0x02
> pciide0 at pci0 dev 31 function 2 "Intel 6300ESB SATA" rev 0x02: DMA,
> channel 0 configured to compatibility, channel 1 configured to
> compatibility
> wd0 at pciide0 channel 0 drive 0: <TS8GSSD25-S>
> wd0: 1-sector PIO, LBA, 7627MB, 15621984 sectors
> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
> ichiic0 at pci0 dev 31 function 3 "Intel 6300ESB SMBus" rev 0x02: SMBus
> disabled
> isa0 at ichpcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: <PC speaker>
> spkr0 at pcppi0
> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> mtrr: Pentium Pro MTRR support
> softraid0 at root
> root on wd0a swap on wd0b dump on wd0b
