Hi folks:
Working on trying to figure out why the Intel NICs on these
motherboards we are working with are slow. Ok, slow is a relative term.
More along the lines of "not as fast as they could be" specifically
relative to a PCI-x 1000/MT adapter we plugged in.
Scenario is trying to do some load testing. I have 4 clients, all
with the same version of OS, pounding on our server (part of the load
test). Gigabit, server does channel bonding. Seeing good results. But
.... on the nodes that use this beast:
04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit
Ethernet Controller (Copper) (rev 01)
Subsystem: Super Micro Computer Inc Unknown device 0000
Flags: bus master, fast devsel, latency 0, IRQ 1274
Memory at c8200000 (32-bit, non-prefetchable) [size=128K]
I/O ports at 2000 [size=32]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable+
Capabilities: [e0] Express Endpoint IRQ 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number
f2-72-32-ff-ff-48-30-00
we get ~70-75 MB/s, while plugging a nice little
05:02.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet
Controller (rev 01)
Subsystem: Intel Corporation PRO/1000 MT Dual Port Server
Adapter
Flags: bus master, 66MHz, medium devsel, latency 52, IRQ 28
Memory at c8340000 (64-bit, non-prefetchable) [size=128K]
Memory at c8300000 (64-bit, non-prefetchable) [size=256K]
I/O ports at 3000 [size=64]
[virtual] Expansion ROM at c2000000 [disabled] [size=256K]
Capabilities: [dc] Power Management version 2
Capabilities: [e4] PCI-X non-bridge device
into a PCI-x slot gives us 92-98 MB/s for our load test (IOzone). It
gives more than that, I am optically averaging.
Ok. So the mystery is *why*.
First I note that the first unit, which is a motherboard NIC, has
"32-bit memory" at a particular address, while the second unit, the
1000/MT card in the PCI-x slot has "64-bit memory" at a different
address.
Second, and this is counter intuitive, but the motherboard gigabit unit
is on PCI-e (x4 at that!)
[ 115.246121] PCI: Setting latency timer of device 0000:04:00.0 to 64
[ 115.261547] e1000: 0000:04:00.0: e1000_probe: (PCI
Express:2.5Gb/s:Width x4) 00:30:48:32:72:f2
[ 115.290791] PM: Adding info for No Bus:eth0
[ 115.290843] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network
Connection
[ 115.290868] ACPI: PCI Interrupt 0000:04:00.1[B] -> GSI 19 (level,
low) -> IRQ 19
[ 115.290882] PCI: Setting latency timer of device 0000:04:00.1 to 64
[ 115.306461] e1000: 0000:04:00.1: e1000_probe: (PCI
Express:2.5Gb/s:Width x4) 00:30:48:32:72:f3
[ 115.342947] PM: Adding info for No Bus:eth1
[ 115.342983] e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network
Connection
while the PCI-x is on, well, PCI-x. And it should be slower.
[ 115.608773] e1000: 0000:05:02.0: e1000_probe: (PCI:33MHz:64-bit)
00:04:23:9e:36:ca
[ 115.636072] PM: Adding info for No Bus:eth2
[ 115.636105] e1000: eth2: e1000_probe: Intel(R) PRO/1000 Network
Connection
[ 115.636129] ACPI: PCI Interrupt 0000:05:02.1[B] -> GSI 29 (level,
low) -> IRQ 29
[ 115.902030] e1000: 0000:05:02.1: e1000_probe: (PCI:33MHz:64-bit)
00:04:23:9e:36:cb
[ 115.928619] PM: Adding info for No Bus:eth3
[ 115.928648] e1000: eth3: e1000_probe: Intel(R) PRO/1000 Network
Connection
[ 115.928687] ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 24 (level,
low) -> IRQ 24
The driver is 7.3.20-k2-NAPI
[ 110.772712] Intel(R) PRO/1000 Network Driver - version
7.3.20-k2-NAPI
[ 110.772717] Copyright (c) 1999-2006 Intel Corporation.
I know 7.6.5 is out, and I installed it on one of the machines, without
any impact.
Motherboard is a Supermicro X7DVA-i I think. I am also seeing this on a
different Supermicro motherboard with dual cores. Same
issue/performance.
Any thoughts?
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: [EMAIL PROTECTED]
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
!DSPAM:46f725f6227831804284693!
