I've been trying to get the fiber interface of the vf610-zii-dev-rev-c
board working with net-next to no avail. This platform utilizes a
Cotsworks SFF attached to port 9 of a Marvell 88E6390X.
I have fiber link up on port 9 and am able to send packets from the
management CPU of the switch through the switch and out port 9 through
the fiber interface to a fiber link partner successfully. I'm also
able to send packets from that fiber link partner back (to the point
of the switches SERDES) and am seeing the fiber ports SERDES RX
counters increment for each packet transmitted by the link partner.
The switches port 9 MAC is not showing any RX counters incrementing
though and I do not receive the frames at the management CPU.
Because the SERDES RX counters are incrementing while the MAC RX
counters are not incrementing, it seems to me that the issue is
between the MAC and SERDES. This is odd to me given that TX works
fine.
In support of debugging this issue, I've applied an ethtool patch
which allows decoding the 88E6390X SERDES registers from userspace.
Looking at the register dump, nothing obvious sticks out to me though.
I'm not sure what the right next steps are and would appreciate any
theories on what to try/test to root cause this issue.
Below is an ethtool register dump and an ethtool statistics dump from
when the link is up and I have done some attempts at pinging a remote
host over fiber.
root@(none):~$ ethtool -d sff2
88E6390X Switch Port Registers
------------------------------
00: Port Status 0xce49
Transmit Pause Enable bit 1
Receive Pause Enable bit 1
802.3 PHY Detected 0
Link Status Up
Duplex Full
Speed 1000 Mbps
Duplex Fixed 0
EEE Enabled 1
Transmitter Paused 0
Flow Control 0
Config Mode 0x9
01: Physical Control 0x203e
RGMII Receive Timing Control Default
RGMII Transmit Timing Control Default
Force Speed 1
Alternate Speed Mode Normal
MII PHY Mode MAC
EEE force value 0
Force EEE 0
Link's Forced value Up
Force Link 1
Duplex's Forced value Full
Force Duplex 1
Force Speed 1000 Mbps
02: Flow Control 0x0100
03: Switch Identifier 0x0a11
04: Port Control 0x0433
Source Address Filtering controls Disabled
Egress Mode Unmodified
Ingress & Egress Header Mode 0
IGMP and MLD Snooping 1
Frame Mode Normal
VLAN Tunnel 0
TagIfBoth 0
Initial Priority assignment Tag & IP Priority
Egress Flooding mode No unknown DA
Port State Forwarding
05: Port Control 1 0x0000
Message Port 0
LAG Port 0
VTU Page 0
LAG ID 0
FID[11:4] 0x000
06: Port Base VLAN Map (Header) 0x0400
FID[3:0] 0x000
Force Mapping 0
VLANTable 10
07: Default VLAN ID & Priority 0x0000
Default Priority 0x0
Force to use Default VID 0
Default VLAN Identifier 0
08: Port Control 2 0x0080
Force good FCS in the frame 0
Allow bad FCS 0
Jumbo Mode 1522
802.1QMode Disabled
Discard Tagged Frames 0
Discard Untagged Frames 0
Map using DA hits 1
ARP Mirror enable 0
Egress Monitor Source Port 0
Ingress Monitor Source Port 0
Allow VID of Zero 0
Default Queue Priority 0x0
09: Egress Rate Control 0x0001
10: Egress Rate Control 2 0x0000
11: Port Association Vector 0x0200
12: Port ATU Control 0x0000
13: Override 0x0000
14: Policy Control 0x0000
15: Port Ether Type 0x9100
16: Reserved 0x0000
17: Reserved 0x0000
18: Reserved 0x0000
19: Reserved 0x0000
20: Reserved 0x0000
21: Reserved 0x0000
22: LED Control 0x0033
23: IP Priority Mapping Table 0x0000
24: IEEE Priority Mapping Table 0x3e07
25: Port Control 3 0x0000
26: Reserved 0x0000
27: Queue Counters 0x8000
28: Queue Control 0x0000
29: Reserved 0x0000
30: Cut Through Control 0x0000
31: Debug Counters 0x0000
88E6390X Switch Port SERDES Registers
-------------------------------------
f000: Global Clock Configuration 1 0x0000
f001: Global Clock Configuration 2 0x0002
f002: Port Operational Configuration 0x0003
f00a: FIFO and CRC Int Enable 0x0000
f00b: FIFO and CRC Int Status 0x0000
f00c: PPM FIFO Control 1 0x0000
f00d: PPM FIFO Control 2 0x0000
f00e: PPM FIFO Status 0x0000
f010: Packet Generation Control 1 0x0501
f011: Packet Generation Control 2 0x0000
f012: Initial Payload 0-1/Packet Generation 0x0000
f013: Initial Payload 2-3/Packet Generation 0x0000
f016: Packet Generation Length 0x0000
f017: Packet Generation Burst Sequence 0x0000
f018: Packet Generation IPG 0x0002
f019: Packet Gen_Chkr Clock Control 0x0000
f01a: Transmit Powerdown Delay 0x0032
f01b: Transmit Packet Counter [15:0] 0x0000
f01c: Transmit Packet Counter [31:16] 0x0000
f01d: Transmit Packet Counter [47:32] 0x0000
f01e: Transmit Byte Counter [15:0] 0x0000
f01f: Transmit Byte Counter [31:16] 0x0000
f020: Transmit Byte Counter [47:32] 0x0000
f021: Receive Packet Counter [15:0] 0x0000
f022: Receive Packet Counter [31:16] 0x0000
f023: Receive Packet Counter [47:32] 0x0000
f024: Receive Byte Count [15:0] 0x0000
f025: Receive Byte Count [31:16] 0x0000
f026: Receive Byte Count [47:32] 0x0000
f027: Receive Packet Error Count [15:0] 0x0000
f028: Receive Packet Error Count [31:16] 0x0000
f029: Receive Packet Error Count [47:32] 0x0000
f030: PRBS Control 0x0200
f031: PRBS Symbol Tx Counter [15:0] 0x0000
f032: PRBS Symbol Tx Counter [31:16] 0x0000
f033: PRBS Symbol Tx Counter [47:32] 0x0000
f034: PRBS Symbol Rx Counter [15:0] 0x0000
f035: PRBS Symbol Rx Counter [31:16] 0x0000
f036: PRBS Symbol RX Counter [47:32] 0x0000
f037: PRBS Error Count [15:0] 0x0000
f038: PRBS Error Count [31:16] 0x0000
f039: PRBS Error Count [47:32] 0x0000
2000: 1000BASE-X/SGMII Control Register 0x1140
Reset 0
Loopback 0
SGMII Speed 1000 Mbps
Autoneg Enable 1
Power down 0
Isolate 0
Restart Autonet 0
Duplex Full
2001: 1000BASE-X/SGMII Status Register 0x016d
Autoneg Complete 1
Remote Fault 0
Link Status Up
2002: PHY Identifier 0x0141
2003: PHY Identifier 0x0c00
2004: SGMII (Media side) Auto-Negotiation Ad 0x00a0
Link Status Down
Duplex Half
SGMII Speed 10 Mbps
Transmit Pause 0
Receive Pause 0
Fibre/Copper Fibre
EEE mode 0
Clock stopped during LPI 1
2005: SGMII (Media side) Link Partner Abilit 0x40a0
Acknowledge 1
2006: 1000BASE-X Auto-Negotiation Expansion 0x0007
2007: 1000BASE-X Next Page Transmit Register 0x2801
2008: 1000BASE-X Link Partner Next Page Regi 0x0000
200f: Extended Status Register 0x8000
a000: 1000BASE-X Timer Mode Select Register 0x2000
a001: 1000BASE-X Interrupt Enable Register 0x0600
Speed Changed 0
Duplex Changed 0
Page Received 0
Autoneg Complete 0
Link Up->Link Down 1
Link Down->Link Up 1
Symbol Error 0
False Carrier 0
a002: 1000BASE-X Interrupt Status Register 0x0000
Speed Changed 0
Duplex Changed 0
Page Received 0
Autoneg Complete 0
Link Up->Link Down 0
Link Down->Link Up 0
Symbol Error 0
False Carrier 0
a003: 1000BASE-X PHY Specific Status 0xac2c
Speed 1000 Mbps
Duplex Full
Page Received 0
Speed/Duplex Resolved 1
Link Up
Sync 1
Energy Detect 0
Transmit Pause 0
Receive Pause 0
1000: 10GBASE-X4 PCS Control 1 0x2040
1001: 10GBASE-X4 PCS Status 1 0x0082
1002: PCS Device Identifier 1 0x0141
1003: PCS Device Identifier 2 0x0c00
1004: PCS Speed Ability 0x0001
1005: PCS Devices In Package 1 0x009a
1006: PCS Devices In Package 2 0x4000
1007: Reserved 0x0001
1008: 10GBASE-X4 PCS Status 2 0x8402
100e: PCS Package Identifier 1 0x0141
100f: PCS Package Identifier 2 0x0c00
1014: PCS EEE Capability Register 0x0000
1018: 10GBase-X Lane Status 0x0c01
1019: 10GBase-X Test Control 0x0000
9000: 10GBase-X Control 0x0001
9001: 10GBase-X Interrupt Enable 1 0x0000
9002: 10GBase-X Interrupt Enable 2 0x0000
9003: 10GBase-X Interrupt Status 1 0x0000
9004: 10GBase-X Interrupt Status 2 0x00e1
9006: 10GBase-X Real Time Status 0x0011
9010: 10GBase-X Random Sequence Control 0x0000
9011: 10GBase-X Jitter Packet Transmit Count 0x0000
9012: 10GBase-X Jitter Packet Transmit Count 0x0000
9013: 10GBase-X Jitter Packet Received Count 0x0000
9014: 10GBase-X Jitter Packet Received Count 0x0000
9015: 10GBase-X Jitter Packet Error Counter 0x0000
9016: 10GBase-X Jitter Packet Error Counter 0x0000
1020: 10GBASE-R PCS Status 1 0x0000
1021: 10GBASE-R PCS Status 2 0x0000
1022: 10GBASE-R PCS Test Pattern Seed A 0 0x0000
1023: 10GBASE-R PCS Test Pattern Seed A 1 0x0000
1024: 10GBASE-R PCS Test Pattern Seed A 2 0x0000
1025: 10GBASE-R PCS Test Pattern Seed A 3 0x0000
1026: 10GBASE-R PCS Test Pattern Seed B 0 0x0000
1027: 10GBASE-R PCS Test Pattern Seed B 1 0x0000
1028: 10GBASE-R PCS Test Pattern Seed B 2 0x0000
1029: 10GBASE-R PCS Test Pattern Seed B 3 0x0000
102a: 10GBASE-R PCS Test Pattern Control 0x0000
102b: 10GBASE-R PCS Test Error Counter 0x0000
root@(none):~$ ethtool -S sff2
NIC statistics:
tx_packets: 3
tx_bytes: 126
rx_packets: 0
rx_bytes: 0
in_good_octets: 0
in_bad_octets: 0
in_unicast: 0
in_broadcasts: 0
in_multicasts: 0
in_pause: 0
in_undersize: 0
in_fragments: 0
in_oversize: 0
in_jabber: 0
in_rx_error: 0
in_fcs_error: 0
out_octets: 192
out_unicast: 0
out_broadcasts: 3
out_multicasts: 0
out_pause: 0
excessive: 0
collisions: 0
deferred: 0
single: 0
multiple: 0
out_fcs_error: 0
late: 0
hist_64bytes: 3
hist_65_127bytes: 0
hist_128_255bytes: 0
hist_256_511bytes: 0
hist_512_1023bytes: 0
hist_1024_max_bytes: 0
in_discards: 0
in_filtered: 0
in_accepted: 0
in_bad_accepted: 0
in_good_avb_class_a: 0
in_good_avb_class_b: 0
in_bad_avb_class_a: 0
in_bad_avb_class_b: 0
tcam_counter_0: 0
tcam_counter_1: 0
tcam_counter_2: 0
tcam_counter_3: 0
in_da_unknown: 0
in_management: 0
out_queue_0: 3
out_queue_1: 0
out_queue_2: 0
out_queue_3: 0
out_queue_4: 0
out_queue_5: 0
out_queue_6: 0
out_queue_7: 0
out_cut_through: 0
out_octets_a: 0
out_octets_b: 0
out_management: 3
serdes_rx_pkts: 6
serdes_rx_bytes: 384
serdes_rx_pkts_error: 0
atu_member_violation: 0
atu_miss_violation: 0
atu_full_violation: 0
vtu_member_violation: 0
vtu_miss_violation: 0
