Joe,
I think you may be dealing with a PCIe fifo issue.
I have seen issues with the Intel PCIe gigabit ethernet onboard
parts when compared to PCIe slot cards and PCIX cards like the ones you
are testing. Specifically the partitioning of the controller's buffers
between rcv and xmit operations (internal to the controller chip itself)
and the controller's relationship with the PCIe buffer on the
northbridge. PCIe, being serial, has different challenges when reaching
the top end of a device's performance capabilities. In this case you are
suffering some buffer throttling.
By default the buffers are partitioned for "one size fits most"
scenario. If you know your i/o profile you can use ethtool (or modify
e1000 driver source) to repartition the controller's fifo to favor rcv
or xmit operations. This results in better performance in situations
where you know you will have heavier writes over reads or vice versa.
*OR* it is because you are using a Supermicro motherboard.. =)
--Jeff
[EMAIL PROTECTED] wrote:
Message: 1
Date: Sun, 23 Sep 2007 22:48:11 -0400
From: Joe Landman <[EMAIL PROTECTED]>
Subject: [Beowulf] Whats up with these newer Intel NICs?
To: Beowulf List <beowulf@beowulf.org>
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi folks:
Working on trying to figure out why the Intel NICs on these
motherboards we are working with are slow. Ok, slow is a relative term.
More along the lines of "not as fast as they could be" specifically
relative to a PCI-x 1000/MT adapter we plugged in.
Scenario is trying to do some load testing. I have 4 clients, all
with the same version of OS, pounding on our server (part of the load
test). Gigabit, server does channel bonding. Seeing good results. But
.... on the nodes that use this beast:
04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
Ethernet Controller (Copper) (rev 01)
Subsystem: Super Micro Computer Inc Unknown device 0000
Flags: bus master, fast devsel, latency 0, IRQ 1274
Memory at c8200000 (32-bit, non-prefetchable) [size=128K]
I/O ports at 2000 [size=32]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable+
Capabilities: [e0] Express Endpoint IRQ 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number f2-72-32-ff-ff-48-30-00
we get ~70-75 MB/s, while plugging a nice little
05:02.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet
Controller (rev 01)
Subsystem: Intel Corporation PRO/1000 MT Dual Port Server Adapter
Flags: bus master, 66MHz, medium devsel, latency 52, IRQ 28
Memory at c8340000 (64-bit, non-prefetchable) [size=128K]
Memory at c8300000 (64-bit, non-prefetchable) [size=256K]
I/O ports at 3000 [size=64]
[virtual] Expansion ROM at c2000000 [disabled] [size=256K]
Capabilities: [dc] Power Management version 2
Capabilities: [e4] PCI-X non-bridge device
into a PCI-x slot gives us 92-98 MB/s for our load test (IOzone). It
gives more than that, I am optically averaging.
Ok. So the mystery is *why*.
First I note that the first unit, which is a motherboard NIC, has
"32-bit memory" at a particular address, while the second unit, the
1000/MT card in the PCI-x slot has "64-bit memory" at a different address.
Second, and this is counter intuitive, but the motherboard gigabit unit
is on PCI-e (x4 at that!)
[ 115.246121] PCI: Setting latency timer of device 0000:04:00.0 to 64
[ 115.261547] e1000: 0000:04:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x4)
00:30:48:32:72:f2
[ 115.290791] PM: Adding info for No Bus:eth0
[ 115.290843] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[ 115.290868] ACPI: PCI Interrupt 0000:04:00.1[B] -> GSI 19 (level, low) ->
IRQ 19
[ 115.290882] PCI: Setting latency timer of device 0000:04:00.1 to 64
[ 115.306461] e1000: 0000:04:00.1: e1000_probe: (PCI Express:2.5Gb/s:Width x4)
00:30:48:32:72:f3
[ 115.342947] PM: Adding info for No Bus:eth1
[ 115.342983] e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
while the PCI-x is on, well, PCI-x. And it should be slower.
[ 115.608773] e1000: 0000:05:02.0: e1000_probe: (PCI:33MHz:64-bit)
00:04:23:9e:36:ca
[ 115.636072] PM: Adding info for No Bus:eth2
[ 115.636105] e1000: eth2: e1000_probe: Intel(R) PRO/1000 Network Connection
[ 115.636129] ACPI: PCI Interrupt 0000:05:02.1[B] -> GSI 29 (level, low) ->
IRQ 29
[ 115.902030] e1000: 0000:05:02.1: e1000_probe: (PCI:33MHz:64-bit)
00:04:23:9e:36:cb
[ 115.928619] PM: Adding info for No Bus:eth3
[ 115.928648] e1000: eth3: e1000_probe: Intel(R) PRO/1000 Network Connection
[ 115.928687] ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 24 (level, low) ->
IRQ 24
The driver is 7.3.20-k2-NAPI
[ 110.772712] Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
[ 110.772717] Copyright (c) 1999-2006 Intel Corporation.
I know 7.6.5 is out, and I installed it on one of the machines, without
any impact.
Motherboard is a Supermicro X7DVA-i I think. I am also seeing this on a
different Supermicro motherboard with dual cores. Same issue/performance.
Any thoughts?
--
Best Regards,
Jeff Johnson
Vice President
Engineering/Technology
Western Scientific, Inc
[EMAIL PROTECTED]
http://www.wsm.com
5444 Napa Street - San Diego, CA 92110
Tel 800.443.6699 +001.619.220.6580
Fax +001.619.220.6590
"Braccae tuae aperiuntur"
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
