Hello,
I have 2 openbsd systems (OpenBSD 5.3 (GENERIC.MP) #62: Tue Mar 12 18:21:20
MDT 2013) in active-backup configuration.
During boot of OS, pfsync bulk takes 13 minutes to sync with master server
no matter how many states there are to sync.
I've reproduced this problem in my testing environment. Even with no
traffic (only 6 states in PF State Table) pfsync bulk took 13 minutes.
Logs from operation:
Aug 2 09:47:02 pfw1 /bsd: root on wd0a (89609dda54f2ae25.a) swap on wd0b
dump on wd0b
Aug 2 09:47:02 pfw1 /bsd: carp: carp10 demoted group carp by 1 to 129
(carpdev)
Aug 2 09:47:02 pfw1 /bsd: carp: pfsync0 demoted group carp by 32 to 161
(pfsync init)
Aug 2 09:47:02 pfw1 /bsd: carp: pfsync0 demoted group pfsync by 32 to 32
(pfsync init)
Aug 2 09:47:02 pfw1 /bsd: carp: pfsync0 demoted group carp by 1 to 162
(pfsync bulk start)
Aug 2 09:47:02 pfw1 /bsd: carp: pfsync0 demoted group pfsync by 1 to 33
(pfsync bulk start)
Aug 2 09:47:01 pfw1 savecore: no core dump
Aug 2 09:47:06 pfw1 /bsd: carp10: state transition: INIT -> BACKUP
Aug 2 09:47:06 pfw1 /bsd: carp: carp10 demoted group carp by -1 to 33
(carpdev)
Aug 2 10:00:01 pfw1 newsyslog[16429]: logfile turned over
tail: /var/log/messages has been replaced, reopening.
Aug 2 10:00:01 pfw1 newsyslog[16429]: logfile turned over
Aug 2 10:00:01 pfw1 syslogd: restart
Aug 2 10:00:24 pfw1 /bsd: carp: pfsync0 demoted group carp by -1 to 32
(pfsync bulk done)
Aug 2 10:00:24 pfw1 /bsd: carp: pfsync0 demoted group pfsync by -1 to 32
(pfsync bulk done)
Aug 2 10:00:24 pfw1 /bsd: carp: pfsync0 demoted group carp by -32 to 0
(pfsync init)
Aug 2 10:00:24 pfw1 /bsd: carp: pfsync0 demoted group pfsync by -32 to 0
(pfsync init)
my pf.conf:
set state-policy if-bound
set block-policy drop
set limit states 400000
set limit src-nodes 20000
set skip on lo
pass quick log proto pfsync keep state (no-sync)
pass quick log proto carp
pass quick log
block quick log
I`ve started to commented out line by line from pf.conf and I`ve noticed
that pfsync bulk time depends on "limit states" value.
So when I changed it to default (10000 states) pfsync bulk took 16 seconds.
When it is set to 10000 it takes 4 minutes to perform pfsync bulk.
10000 states:
Aug 2 11:30:52 pfw1 /bsd: carp: pfsync0 demoted group carp by 32 to 161
(pfsync init)
Aug 2 11:30:52 pfw1 /bsd: carp: pfsync0 demoted group pfsync by 32 to 32
(pfsync init)
Aug 2 11:30:52 pfw1 /bsd: carp: pfsync0 demoted group carp by 1 to 162
(pfsync bulk start)
Aug 2 11:30:52 pfw1 /bsd: carp: pfsync0 demoted group pfsync by 1 to 33
(pfsync bulk start)
Aug 2 11:30:51 pfw1 savecore: no core dump
Aug 2 11:30:53 pfw1 /bsd: carp10: state transition: INIT -> BACKUP
Aug 2 11:30:53 pfw1 /bsd: carp: carp10 demoted group carp by -1 to 161
(carpdev)
Aug 2 11:31:08 pfw1 /bsd: carp10: state transition: BACKUP -> MASTER
Aug 2 11:31:08 pfw1 /bsd: carp10: state transition: MASTER -> BACKUP
Aug 2 11:31:14 pfw1 /bsd: carp: pfsync0 demoted group carp by -1 to 32
(pfsync bulk done)
Aug 2 11:31:14 pfw1 /bsd: carp: pfsync0 demoted group pfsync by -1 to 32
(pfsync bulk done)
Aug 2 11:31:14 pfw1 /bsd: carp: pfsync0 demoted group carp by -32 to 0
(pfsync init)
Aug 2 11:31:14 pfw1 /bsd: carp: pfsync0 demoted group pfsync by -32 to 0
(pfsync init)
100000 states:
Aug 2 11:35:38 pfw1 /bsd: carp: pfsync0 demoted group carp by 32 to 161
(pfsync init)
Aug 2 11:35:38 pfw1 /bsd: carp: pfsync0 demoted group pfsync by 32 to 32
(pfsync init)
Aug 2 11:35:38 pfw1 /bsd: carp: pfsync0 demoted group carp by 1 to 162
(pfsync bulk start)
Aug 2 11:35:38 pfw1 /bsd: carp: pfsync0 demoted group pfsync by 1 to 33
(pfsync bulk start)
Aug 2 11:35:37 pfw1 savecore: no core dump
Aug 2 11:35:39 pfw1 /bsd: carp10: state transition: INIT -> BACKUP
Aug 2 11:35:39 pfw1 /bsd: carp: carp10 demoted group carp by -1 to 161
(carpdev)
Aug 2 11:39:00 pfw1 /bsd: carp: pfsync0 demoted group carp by -1 to 32
(pfsync bulk done)
Aug 2 11:39:00 pfw1 /bsd: carp: pfsync0 demoted group pfsync by -1 to 32
(pfsync bulk done)
Aug 2 11:39:00 pfw1 /bsd: carp: pfsync0 demoted group carp by -32 to 0
(pfsync init)
Aug 2 11:39:00 pfw1 /bsd: carp: pfsync0 demoted group pfsync by -32 to 0
(pfsync init)
Why is that happening?
My testing environment:
Machine: 2x Sun Fire X2200 M2 with Quad Core Processor
Network cables:
machine1 nfe0 - crossover cabel - nfe0 machine2 (pfsync)
machine1 nfe1 - crossover cabel - nfe1 machine2 (carp)
machine1 bge0 - my lan connection, cable to switch
machine2 bge0 - my lan connection, cable to switch (same switch that the
one line above.
Network configuration Machine1
/etc/hostname.nfe0
inet 172.31.255.9 255.255.255.252 NONE description "LAN"
-inet6
up
/etc/hostname.nfe1
up description WAN
-inet6
/etc/hostname.carp10
carpdev nfe1 vhid 3 pass sosecret advskew 0 advbase 5 -inet6
inet 10.10.5.1 255.255.255.0
/etc/hostname.pfsync0
up syncdev nfe0
/etc/hostname.bge0
inet 192.168.50.58 255.255.255.0 NONE description "LAN WAW"
-inet6
up
!route add -inet 192.168.30.99 192.168.50.1
!route add -inet 192.168.30.186 192.168.50.1
Network configuration Machine2
/etc/hostname.nfe0
inet 172.31.255.10 255.255.255.252 NONE description "LAN"
-inet6
up
/etc/hostname.nfe1
up description WAN
-inet6
/etc/hostname.carp10
carpdev nfe1 vhid 3 pass sosecret advskew 200 advbase 5 -inet6
inet 10.10.5.1 255.255.255.0
/etc/hostname.pfsync0
up syncdev nfe0
/etc/hostname.bge0
inet 192.168.50.66 255.255.255.0 NONE description "LAN WAW"
-inet6
up
!route add -inet 192.168.30.99 192.168.50.1
!route add -inet 192.168.30.186 192.168.50.1
Common configuration:
netstat -m
282 mbufs in use:
274 mbufs allocated to data
2 mbufs allocated to packet headers
6 mbufs allocated to socket names and addresses
273/318/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
940 Kbytes allocated to network (65% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
netstat -nr -f inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio
Iface
10.10.5/24 link#7 C 0 0 - 4
carp10
127/8 127.0.0.1 UGRS 0 0 33152 8 lo0
127.0.0.1 127.0.0.1 UH 1 32 33152 4 lo0
172.31.255.8/30 link#1 UC 0 0 - 4
nfe0
192.168.30.99 192.168.50.1 UGHS 1 751 - 8
bge0
192.168.30.189 192.168.50.1 UGHS 0 0 - 8
bge0
192.168.50/24 link#3 UC 1 0 - 4
bge0
192.168.50.1 00:1b:21:a0:a4:c6 UHLc 2 0 - 4
bge0
224/4 127.0.0.1 URS 0 0 33152 8 lo0
PF:
FILTER RULES:
@0 pass log quick proto pfsync all keep state (no-sync, if-bound)
[ Evaluations: 22 Packets: 585 Bytes: 162952 States: 2
]
[ Inserted: uid 0 pid 10754 State Creations: 2 ]
@1 pass log quick proto carp all keep state (if-bound)
[ Evaluations: 20 Packets: 274 Bytes: 15344 States: 1
]
[ Inserted: uid 0 pid 10754 State Creations: 1 ]
@2 pass log quick all flags S/SA keep state (if-bound)
[ Evaluations: 19 Packets: 767 Bytes: 207539 States: 4
]
[ Inserted: uid 0 pid 10754 State Creations: 25 ]
@3 block drop log quick all
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
[ Inserted: uid 0 pid 10754 State Creations: 0 ]
No queue in use
STATES:
nfe0 pfsync 172.31.255.9 -> 224.0.0.240 SINGLE:NO_TRAFFIC
age 00:11:30, expires in 00:00:28, 287:0 pkts, 79792:0 bytes, rule 0
id: 51fb7d6600000000 creatorid: 82f57a4b
nfe0 pfsync 224.0.0.240 <- 172.31.255.10 NO_TRAFFIC:SINGLE
age 00:11:28, expires in 00:00:27, 298:0 pkts, 83160:0 bytes, rule 0
id: 51fb7d6600000001 creatorid: 82f57a4b
nfe1 carp 224.0.0.18 <- 10.10.5.1 NO_TRAFFIC:SINGLE
age 00:11:25, expires in 00:00:26, 274:0 pkts, 15344:0 bytes, rule 1
id: 51fb7d6600000002 creatorid: 82f57a4b
nfe1 carp 10.10.5.1 -> 224.0.0.18 SINGLE:NO_TRAFFIC
age 00:15:58, expires in 00:00:27, 0:0 pkts, 0:0 bytes
id: 51fb7c4800000004 creatorid: 82f57a4b
bge0 udp 192.168.50.255:631 <- 192.168.50.1:631 NO_TRAFFIC:SINGLE
age 00:11:06, expires in 00:00:18, 66:0 pkts, 15070:0 bytes, rule 2
id: 51fb7d6600000003 creatorid: 82f57a4b
bge0 udp 192.168.60.255:631 <- 192.168.60.1:631 NO_TRAFFIC:SINGLE
age 00:11:06, expires in 00:00:18, 66:0 pkts, 15070:0 bytes, rule 2
id: 51fb7d6600000004 creatorid: 82f57a4b
bge0 tcp 192.168.50.58:22 <- 192.168.30.99:46107
ESTABLISHED:ESTABLISHED
[941657145 + 91648] wscale 3 [3525060236 + 17376] wscale 7
age 00:10:01, expires in 24:00:00, 324:291 pkts, 24789:150253 bytes,
rule 2
id: 51fb7d6600000006 creatorid: 82f57a4b
bge0 tcp 192.168.50.66:22 <- 192.168.30.99:35527
ESTABLISHED:ESTABLISHED
[2687422329 + 42368] wscale 3 [1652551938 + 17376] wscale 7
age 00:27:56, expires in 23:57:15, 0:0 pkts, 0:0 bytes
id: 51fb61010000006f creatorid: d39dd2a9
bge0 udp 192.168.60.255:138 <- 192.168.60.74:138 NO_TRAFFIC:SINGLE
age 00:00:22, expires in 00:00:38, 1:0 pkts, 229:0 bytes, rule 2
id: 51fb7d6600000010 creatorid: 82f57a4b
INFO:
Status: Enabled for 0 days 00:11:32 Debug: err
Hostid: 0x82f57a4b
Checksum: 0xb754ca6491198bb3eda3ecd1e170c3a7
State Table Total Rate
current entries 9
searches 2263 3.3/s
inserts 19 0.0/s
removals 10 0.0/s
Source Tracking Table
current entries 0
searches 0 0.0/s
inserts 0 0.0/s
removals 0 0.0/s
Counters
match 22 0.0/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 5 0.0/s
proto-cksum 0 0.0/s
state-mismatch 0 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
Limit Counters
max states per rule 0 0.0/s
max-src-states 0 0.0/s
max-src-nodes 0 0.0/s
max-src-conn 0 0.0/s
max-src-conn-rate 0 0.0/s
overload table insertion 0 0.0/s
overload flush states 0 0.0/s
TIMEOUTS:
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 30s
interval 10s
adaptive.start 60000 states
adaptive.end 120000 states
src.track 0s
LIMITS:
states hard limit 100000
src-nodes hard limit 20000
frags hard limit 1536
tables hard limit 1000
table-entries hard limit 200000
OS FINGERPRINTS:
710 fingerprints loaded
Sysctl (network related)
net.inet.ip.forwarding=0
net.inet.ip.redirect=1
net.inet.ip.ttl=64
net.inet.ip.sourceroute=0
net.inet.ip.directed-broadcast=0
net.inet.ip.portfirst=1024
net.inet.ip.portlast=30000
net.inet.ip.porthifirst=30001
net.inet.ip.porthilast=65535
net.inet.ip.maxqueue=300
net.inet.ip.encdebug=0
net.inet.ip.ipsec-expire-acquire=30
net.inet.ip.ipsec-invalid-life=60
net.inet.ip.ipsec-pfs=1
net.inet.ip.ipsec-soft-allocs=0
net.inet.ip.ipsec-allocs=0
net.inet.ip.ipsec-soft-bytes=0
net.inet.ip.ipsec-bytes=0
net.inet.ip.ipsec-timeout=86400
net.inet.ip.ipsec-soft-timeout=80000
net.inet.ip.ipsec-soft-firstuse=3600
net.inet.ip.ipsec-firstuse=7200
net.inet.ip.ipsec-enc-alg=aes
net.inet.ip.ipsec-auth-alg=hmac-sha1
net.inet.ip.mtudisc=1
net.inet.ip.mtudisctimeout=600
net.inet.ip.ipsec-comp-alg=deflate
net.inet.ip.ifq.len=0
net.inet.ip.ifq.maxlen=2048
net.inet.ip.ifq.drops=0
net.inet.ip.mforwarding=0
net.inet.ip.multipath=0
net.inet.ip.mrtproto=19
net.inet.ip.arpqueued=0
net.inet.icmp.maskrepl=0
net.inet.icmp.bmcastecho=0
net.inet.icmp.errppslimit=100
net.inet.icmp.rediraccept=0
net.inet.icmp.redirtimeout=600
net.inet.icmp.tstamprepl=1
net.inet.ipip.allow=0
net.inet.tcp.rfc1323=1
net.inet.tcp.keepinittime=150
net.inet.tcp.keepidle=14400
net.inet.tcp.keepintvl=150
net.inet.tcp.slowhz=2
net.inet.tcp.baddynamic=1,7,9,11,13,15,17,18,19,20,21,22,23,25,37,42,43,53,57
,67,68,70,77,79,80,87,88,95,101,102,103,104,105,106,107,109,110,111,113,115,1
17,119,123,129,135,137,138,139,143,152,163,164,177,178,179,191,194,199,201,20
2,204,206,210,213,220,372,389,427,433,443,445,464,465,468,512,513,514,515,521
,526,530,531,532,540,543,544,545,548,554,556,587,631,636,646,706,749,750,751,
754,760,782,871,873,888,993,995,1080,1109,1127,1433,1434,1524,1525,1723,1900,
2049,2105,2106,2108,2110,2111,2112,2120,2121,2401,2627,2983,3031,3260,3306,35
17,3689,4190,4444,4500,5002,5060,5432,5680,6000,6001,6002,6003,6004,6005,6006
,6007,6008,6009,6010,7326,8025,8026,10050,10051,16992,16993,16994,16995
net.inet.tcp.sack=1
net.inet.tcp.mssdflt=512
net.inet.tcp.rstppslimit=100
net.inet.tcp.ackonpush=0
net.inet.tcp.ecn=0
net.inet.tcp.syncachelimit=10255
net.inet.tcp.synbucketlimit=105
net.inet.tcp.rfc3390=2
net.inet.tcp.reasslimit=3072
net.inet.tcp.sackholelimit=32768
net.inet.tcp.always_keepalive=0
net.inet.udp.checksum=1
net.inet.udp.baddynamic=7,9,13,18,19,22,37,39,53,67,68,69,70,80,88,105,107,10
9,110,111,123,129,135,137,138,139,143,161,162,163,164,177,178,179,191,194,199
,201,202,204,206,210,213,220,372,389,427,445,464,468,500,512,513,514,517,518,
520,525,533,548,554,587,623,631,636,646,664,706,749,750,751,993,995,1433,1434
,1524,1525,1645,1646,1701,1723,1812,1813,1900,2049,2627,3031,3517,3689,4190,4
444,4500,5002,5060,7000,7001,7002,7003,7004,7005,7006,7007,7008,7009,8025,806
7,10050,10051,26740
net.inet.udp.recvspace=41600
net.inet.udp.sendspace=9216
net.inet.gre.allow=0
net.inet.gre.wccp=0
net.inet.esp.enable=1
net.inet.esp.udpencap=1
net.inet.esp.udpencap_port=4500
net.inet.ah.enable=1
net.inet.mobileip.allow=0
net.inet.etherip.allow=0
net.inet.ipcomp.enable=0
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=7
net.inet.divert.recvspace=65636
net.inet.divert.sendspace=65636
net.bpf.bufsize=32768
net.bpf.maxbufsize=2097152
net.mpls.ttl=255
net.mpls.ifq.len=0
net.mpls.ifq.maxlen=256
net.mpls.ifq.drops=0
net.mpls.maxloop_inkernel=16
net.mpls.mapttl_ip=1
net.mpls.mapttl_ip6=0
net.pipex.enable=0
Dmesg
OpenBSD 5.3 (GENERIC.MP) #62: Tue Mar 12 18:21:20 MDT 2013
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 32210878464 (30718MB)
avail mem = 31330938880 (29879MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfcff0 (52 entries)
bios0: vendor Sun Microsystems version "S39_3D16" date 03/17/2010
bios0: Sun Microsystems Sun Fire X2200 M2 with Quad Core Processor
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC SPCR OEMB SRAT HPET EINJ BERT ERST HEST SSDT
acpi0: wakeup devices PS2K(S4) PS2M(S4) NSMB(S4) USB0(S4) USB2(S1) NMAC(S5)
NMAD(S5) P0P1(S4) HDAC(S4) BR10(S4) BR11(S4) BR12(S4) BR1E(S5) BCM1(S5)
BR13(S4) BR14(S4) BR15(S4) SLPB(S4) PWRB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.31 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu0: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu0: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu0: AMD erratum 721 detected and fixed
cpu0: apic clock running at 200MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.10 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu1: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu1: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu1: AMD erratum 721 detected and fixed
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.10 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu2: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu2: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu2: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu2: AMD erratum 721 detected and fixed
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.10 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu3: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu3: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu3: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu3: AMD erratum 721 detected and fixed
cpu4 at mainbus0: apid 4 (application processor)
cpu4: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.10 MHz
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu4: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu4: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu4: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu4: AMD erratum 721 detected and fixed
cpu5 at mainbus0: apid 5 (application processor)
cpu5: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.10 MHz
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu5: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu5: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu5: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu5: AMD erratum 721 detected and fixed
cpu6 at mainbus0: apid 6 (application processor)
cpu6: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.10 MHz
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu6: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu6: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu6: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu6: AMD erratum 721 detected and fixed
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Quad-Core AMD Opteron(tm) Processor 2347 HE, 1900.10 MHz
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DN
OW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,ITSC
cpu7: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 2MB 64b/line 32-way L3 cache
cpu7: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu7: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu7: AMD erratum 721 detected and fixed
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 25000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 7 (BR10)
acpiprt3 at acpi0: bus -1 (BR11)
acpiprt4 at acpi0: bus 5 (BR12)
acpiprt5 at acpi0: bus 6 (BR1E)
acpiprt6 at acpi0: bus 4 (BR13)
acpiprt7 at acpi0: bus 3 (BR14)
acpiprt8 at acpi0: bus 2 (BR15)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
acpicpu2 at acpi0: PSS
acpicpu3 at acpi0: PSS
acpicpu4 at acpi0: PSS
acpicpu5 at acpi0: PSS
acpicpu6 at acpi0: PSS
acpicpu7 at acpi0: PSS
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
cpu0: 1900 MHz: speeds: 1900 1700 1400 1200 1000 MHz
pci0 at mainbus0 bus 0
"NVIDIA MCP55 Memory" rev 0xa2 at pci0 dev 0 function 0 not configured
pcib0 at pci0 dev 1 function 0 "NVIDIA MCP55 ISA" rev 0xa3
nviic0 at pci0 dev 1 function 1 "NVIDIA MCP55 SMBus" rev 0xa3
iic0 at nviic0
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem1 at iic0 addr 0x51: 1GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem2 at iic0 addr 0x52: 1GB DDR2 SDRAM registered ECC PC2-3200CL3
spdmem3 at iic0 addr 0x53: 1GB DDR2 SDRAM registered ECC PC2-3200CL3
spdmem4 at iic0 addr 0x54: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem5 at iic0 addr 0x55: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem6 at iic0 addr 0x56: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem7 at iic0 addr 0x57: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
iic1 at nviic0
spdmem8 at iic1 addr 0x50: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem9 at iic1 addr 0x51: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem10 at iic1 addr 0x52: 1GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem11 at iic1 addr 0x53: 1GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem12 at iic1 addr 0x54: 4GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem13 at iic1 addr 0x55: 4GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem14 at iic1 addr 0x56: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
spdmem15 at iic1 addr 0x57: 2GB DDR2 SDRAM registered cmd/addr parity, data
ECC PC2-5300CL5
ohci0 at pci0 dev 2 function 0 "NVIDIA MCP55 USB" rev 0xa1: apic 8 int 7,
version 1.0, legacy support
ehci0 at pci0 dev 2 function 1 "NVIDIA MCP55 USB" rev 0xa2: apic 8 int 10
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 4 function 0 "NVIDIA MCP55 IDE" rev 0xa1: DMA, channel
0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 ignored (disabled)
pciide1 at pci0 dev 5 function 0 "NVIDIA MCP55 SATA" rev 0xa3: DMA
pciide1: using apic 8 int 11 for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <Hitachi HDT725025VLA380>
wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
ppb0 at pci0 dev 6 function 0 "NVIDIA MCP55 PCI-PCI" rev 0xa2
pci1 at ppb0 bus 1
vga1 at pci1 dev 5 function 0 "ASPEED Technology AST2000" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
nfe0 at pci0 dev 8 function 0 "NVIDIA MCP55 LAN" rev 0xa3: apic 8 int 5,
address 00:23:8b:03:82:4d
eephy0 at nfe0 phy 2: 88E1149 Gigabit PHY, rev. 3
nfe1 at pci0 dev 9 function 0 "NVIDIA MCP55 LAN" rev 0xa3: apic 8 int 10,
address 00:23:8b:03:82:4e
eephy1 at nfe1 phy 3: 88E1149 Gigabit PHY, rev. 3
ppb1 at pci0 dev 10 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci2 at ppb1 bus 2
ppb2 at pci0 dev 11 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci3 at ppb2 bus 3
ppb3 at pci0 dev 12 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci4 at ppb3 bus 4
ppb4 at pci0 dev 13 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci5 at ppb4 bus 5
ppb5 at pci5 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb5
pci6 at ppb5 bus 6
bge0 at pci6 dev 4 function 0 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): apic 8 int 10, address 00:23:8b:03:82:4b
brgphy0 at bge0 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0
bge1 at pci6 dev 4 function 1 "Broadcom BCM5715" rev 0xa3, BCM5715 A3
(0x9003): apic 8 int 11, address 00:23:8b:03:82:4c
brgphy1 at bge1 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0
ppb6 at pci0 dev 15 function 0 "NVIDIA MCP55 PCIE" rev 0xa3
pci7 at ppb6 bus 7
mpi0 at pci7 dev 0 function 0 "Symbios Logic SAS1064E" rev 0x02: apic 8 int
5
scsibus0 at mpi0: 63 targets
pchb0 at pci0 dev 24 function 0 "AMD AMD64 10h HyperTransport" rev 0x00
pchb1 at pci0 dev 24 function 1 "AMD AMD64 10h Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 10h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD AMD64 10h Misc Cfg" rev 0x00
pchb3 at pci0 dev 24 function 4 "AMD AMD64 10h Link Cfg" rev 0x00
pchb4 at pci0 dev 25 function 0 "AMD AMD64 10h HyperTransport" rev 0x00
pchb5 at pci0 dev 25 function 1 "AMD AMD64 10h Address Map" rev 0x00
pchb6 at pci0 dev 25 function 2 "AMD AMD64 10h DRAM Cfg" rev 0x00
km1 at pci0 dev 25 function 3 "AMD AMD64 10h Misc Cfg" rev 0x00
pchb7 at pci0 dev 25 function 4 "AMD AMD64 10h Link Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627EHF rev 0x68
wbsio0 port 0xa78/2 not configured
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "NVIDIA OHCI root hub" rev 1.00/1.00 addr 1
mtrr: Pentium Pro MTRR support
uhidev0 at uhub1 port 2 configuration 1 interface 0 "Virtual USB MULTIMEDIA
KEYBOARD" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub1 port 2 configuration 1 interface 1 "Virtual USB MULTIMEDIA
KEYBOARD" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (89609dda54f2ae25.a) swap on wd0b dump on wd0b
carp: carp10 demoted group carp by 1 to 129 (carpdev)
carp: pfsync0 demoted group carp by 32 to 161 (pfsync init)
carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init)
carp: pfsync0 demoted group carp by 1 to 162 (pfsync bulk start)
carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start)
carp10: state transition: INIT -> BACKUP
carp: carp10 demoted group carp by -1 to 161 (carpdev)
carp: pfsync0 demoted group carp by -1 to 32 (pfsync bulk done)
carp: pfsync0 demoted group pfsync by -1 to 32 (pfsync bulk done)
carp: pfsync0 demoted group carp by -32 to 0 (pfsync init)
carp: pfsync0 demoted group pfsync by -32 to 0 (pfsync init)
ps auxww
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 1 0.0 0.0 600 432 ?? Is 11:35AM 0:01.00 /sbin/init
root 6022 0.0 0.0 584 860 ?? Is 11:35AM 0:00.02 syslogd:
[priv] (syslogd)
_syslogd 16197 0.0 0.0 616 828 ?? I 11:35AM 0:00.02
/usr/sbin/syslogd -a /var/www/dev/log -a /var/empty/dev/log
root 25112 0.0 0.0 732 524 ?? Is 11:35AM 0:00.17 pflogd:
[priv] (pflogd)
_pflogd 27504 0.0 0.0 796 392 ?? S 11:35AM 0:00.01 pflogd:
[running] -s 160 -i pflog0 -f /var/log/pflog (pflogd)
root 1059 0.0 0.0 968 1412 ?? Is 11:35AM 0:00.12
/usr/sbin/sshd
root 25974 0.0 0.0 1732 1780 ?? Ss 11:35AM 0:00.25 sendmail:
accepting connections (sendmail)
root 32392 0.0 0.0 524 884 ?? Is 11:35AM 0:00.03
/usr/sbin/inetd
_sndio 16182 0.0 0.0 420 504 ?? I<s 11:35AM 0:00.03
/usr/bin/sndiod
root 20016 0.0 0.0 636 1032 ?? Is 11:35AM 0:00.05
/usr/sbin/cron
root 30174 0.0 0.0 3432 3236 ?? Ss 11:37AM 0:00.53 sshd:
root@ttyp0 (sshd)
root 6225 0.0 0.0 812 636 p0 Ss 11:37AM 0:00.02 -ksh (ksh)
root 2156 0.0 0.0 420 260 p0 R+/3 11:49AM 0:01.65 ps -auxww
root 23113 0.0 0.0 332 900 C0 Is+ 11:35AM 0:00.02
/usr/libexec/getty std.9600 ttyC0
root 13274 0.0 0.0 512 904 C1 Is+ 11:35AM 0:00.01
/usr/libexec/getty std.9600 ttyC1
root 11787 0.0 0.0 532 912 C2 Is+ 11:35AM 0:00.01
/usr/libexec/getty std.9600 ttyC2
root 12138 0.0 0.0 524 904 C3 Is+ 11:35AM 0:00.01
/usr/libexec/getty std.9600 ttyC3
root 9642 0.0 0.0 348 908 C5 Is+ 11:35AM 0:00.01
/usr/libexec/getty std.9600 ttyC5
Tcpdump:
# tcpdump -netti nfe0
tcpdump: listening on nfe0, link-type EN10MB
tcpdump: WARNING: compensating for unaligned libpcap packets
1375437404.177855 00:23:8b:03:82:4d 01:00:5e:00:00:f0 0800 394: 172.31.255.9:
PFSYNCv6 len 360
act UPD ST COMP count 4
...
(DF) [tos 0x10]
1375437404.177886 00:1e:68:86:53:e2 01:00:5e:00:00:f0 0800 338:
172.31.255.10: PFSYNCv6 len 304
act UPD ST REQ count 2
id: 51fb7d6600000003 creatorid: 82f57a4b
id: 51fb7d6600000004 creatorid: 82f57a4b
act UPD ST COMP count 3
...
(DF) [tos 0x10]
1375437404.177895 00:23:8b:03:82:4d 01:00:5e:00:00:f0 0800 614: 172.31.255.9:
PFSYNCv6 len 580
act UPD ST REQ count 2
id: 51fb610100000024 creatorid: d39dd2a9
id: 51fb610100000025 creatorid: d39dd2a9
act UPD ST count 2
...
(DF) [tos 0x10]
1375437404.177928 00:1e:68:86:53:e2 01:00:5e:00:00:f0 0800 586:
172.31.255.10: PFSYNCv6 len 552
act UPD ST count 2
...
(DF) [tos 0x10]
1375437405.227862 00:23:8b:03:82:4d 01:00:5e:00:00:f0 0800 142: 172.31.255.9:
PFSYNCv6 len 108
act UPD ST COMP count 1
...
(DF) [tos 0x10]
1375437406.227872 00:23:8b:03:82:4d 01:00:5e:00:00:f0 0800 142: 172.31.255.9:
PFSYNCv6 len 108
act UPD ST COMP count 1
...
Regards--
Dariusz Binkul
