Hi, I got a freeBSD 8.1 polling issue on my PC. It is a dual-core Intel Pentium x86 PC (2.8GHz each core). The Ethernet interface is Broadcom NetXtreme 57xx Gigabit Ethernet interface. I set the following options (enable polling and zero-buffer copy) and rebuilt the kernel:
Code:
# To make an SMP kernel, the next two lines are needed
options SMP # Symmetric MultiProcessor Kernel
device apic # I/O APIC
options DEVICE_POLLING # Open Polling
options HZ=1000
options ZERO_COPY_SOCKETS
The following were appended to the /etc/sysctl.conf
Code:
kern.polling.enable=1
# increase BPF buffer to 10M
net.bpf.bufsize=10485760
net.bpf.maxbufsize=10485760
kern.polling.idle_poll=1
kern.polling.burst_max=1000
After installed and rebooted the system, kern.polling.enable was not found in
MIB so I had to ignore this error. Looks like kern.polling.enable is removed
from FreeBSD v8.1?
Everything looked good so build my application to received data from another HP
server. I wrote the application using libpcap-1.1.1 with BFP zero-copy turned
on (I found the #define HAVE_ZEROCOPY_BPF 1 in config.h). Attached please find
the source code of my application.
Before running the application, I set the following parameters:
Code:
ifconfig bge0 polling # This will turn on the polling of the Broadcom
driver.
Code:
sysctl -w net.bpf.bufsize=10485760
sysctl -w net.bpf.maxbufsize=10485760
sysctl -w kern.polling.idle_poll=1
sysctl -w kern.polling.burst_max=1000
sysctl -w kern.polling.each_burst=128
sysctl -w net.inet.ip.intr_queue_maxlen=256
Then I ran the application to receive data from the HP server. I ran multiple
iperf on the HP server to send around 133Mbits/s UDP load to the PC under test.
The UDP payload size was 47 bytes. The entire IP packet size is 76 bytes.
First of all, the receiving application worked well and received around 205K
packets/second without packet losing (I checked the receiving status using
pcap_stats). However, after 2 minutes, the application can not received data
any more. The packets/second is 0. I ran the ping from the PC under test and
found that the ping reporting timeout and destination unreachable (the ping
from HP to the PC also failed). Looked like the link between the HP server and
PC was broken so the application could receive data. No packet was dropped.
Then I restart the bge0 interface using: ifconfig bge0 down && ifconfig bge0 up
And then I re-ran the application and it continued receiving data. But after 1
or 2 minutes, the link broke again. I think it was my application that caused
the bge0 interface down. I started the tcpdump and it worked well without
breaking the link.
I tried to increase the kern.polling.each_burst from 128 to 500 but the
application would cause the bge0 down within 1 minute. No packet was dropped
before the link was down.
I checked the CPU usage of the PC. The sys used is around 90% (might be caused
by kern.polling.idle_poll=1), user land is 13%.
I don’t understand why the application would break the bge0.
I tried changing the parameters:
options HZ=2000
sysctl -w net.bpf.bufsize=20485760
sysctl -w net.bpf.maxbufsize=20485760
sysctl -w kern.polling.idle_poll=1
sysctl -w kern.polling.burst_max=10000
sysctl -w kern.polling.each_burst=5000
The performance was better: I got 307K packet/second (the HP server sended
around 250Mbits/s, my PC got 200Mbits/s). But after 2 minutes, the bge0 was
down again.
Could anybody have a look at this issue? How can I optimize the performance of
the polling?
Thanks,
Jin
<<cap.cpp>>
Best regards
===========================
犯强汉者，虽远必诛。
麦进 Mai Jin
Alcatel Shanghai Bell (Nanjing) Co. Ltd.
Alcatel-Net: 2735-5011
Tel: (+86)-25-8473 1240-5011
Addr: 11F, Yangtse River Tech Park.
Building No.40 of Nanchang Road,
Gulou District, Nanjing, China
Zip: 210037
[email protected]
ASB/MoAD/RDR/BSR APL
cap.cpp
Description: cap.cpp
_______________________________________________ [email protected] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "[email protected]"
