Intel X540-T2 on Debian 12 won't negotiate 10Gbps on both ports

[citricut]

Dear Intel community

I'm running a homelab ProxMox 8 node on a HP Prodesk 600 G6 with Intel i7-10700 CPU. I've plugged an Intel X540-T2 dual 10G card to it to use pfSense as homelab firewall.
Port 0 is connected to my ISP router's 10G port (Zyxel AX7501-B0), the other port is connected to a switch with a TP Link 10G RJ45 SFP+ transceiver. Card is connected to the PCIe x16 slot since I don't have a graphics card.
The situation I have is that only one of the ports would negotiate 10G, with 1 other running at 1G. If I test the ports individually, each of them can link at 10G but when both are plugged, typically port 0 will run at 1G and port 1 at 10G.

So far I have tried to update to the latest ixgbe driver and enable IOMMU groups to make sure they don't share one. I'm running out of ideas therefore recurring to the community for some hints on what to try next. Any help is much appreciated.

Best regards!

These are some outputs:

ethtool:

root@pve:~# ethtool  enp1s0f0
Settings for enp1s0f0:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        MDI-X: Unknown
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
root@pve:~# ethtool  enp1s0f1
Settings for enp1s0f1:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        MDI-X: Unknown
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

ethtool -i:

Code:

root@pve:~# ethtool -i enp1s0f1
driver: ixgbe
version: 6.5.11-7-pve
firmware-version: 0x8000052f
expansion-rom-version:
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@pve:~# ethtool -i enp1s0f0
driver: ixgbe
version: 6.5.11-7-pve
firmware-version: 0x8000052f
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

dmesg

Code:

[    1.293509] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    1.293510] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    1.293558] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[    1.595779] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
[    1.679543] ixgbe 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    1.704009] ixgbe 0000:01:00.0: MAC: 3, PHY: 0, PBA No: 000000-000
[    1.704012] ixgbe 0000:01:00.0: 6c:92:bf:5e:c8:e3
[    1.850976] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[    1.851054] ixgbe 0000:01:00.1: enabling device (0000 -> 0002)
[    2.127312] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
[    2.211424] ixgbe 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    2.235610] ixgbe 0000:01:00.1: MAC: 3, PHY: 0, PBA No: 000000-000
[    2.235612] ixgbe 0000:01:00.1: 6c:92:bf:5e:c8:e4
[    2.382946] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[    2.412958] ixgbe 0000:01:00.0 enp1s0f0: renamed from eth0
[    2.456883] ixgbe 0000:01:00.1 enp1s0f1: renamed from eth1
[    8.868295] ixgbe 0000:01:00.0 enp1s0f0: entered allmulticast mode
[    8.868315] ixgbe 0000:01:00.0 enp1s0f0: entered promiscuous mode
[    9.129915] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0
[    9.285110] ixgbe 0000:01:00.1 enp1s0f1: entered allmulticast mode
[    9.285160] ixgbe 0000:01:00.1 enp1s0f1: entered promiscuous mode
[    9.360594] ixgbe 0000:01:00.1: registered PHC device on enp1s0f1
[   14.203023] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[   31.754225] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 1 Gbps, Flow Control: RX/TX

Disconnect cable from port1, bring port0 down and up, connects at 10Gbps:

[87550.031832] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
[87653.861191] ixgbe 0000:01:00.0: removed PHC on enp1s0f0
[87654.549426] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0
[87660.450977] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Connect cable port1, port0 goes down and reconnects at 1 Gbps, port1 connects at 10Gbps
[87697.818005] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Down
[87698.875198] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[87720.303569] ixgbe 0000:01:00.0 enp1s0f0: NIC Link is Up 1 Gbps, Flow Control: RX/TX
[87756.762677] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Down
[87757.803543] ixgbe 0000:01:00.1 enp1s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

modinfo latest driver:

filename:       /lib/modules/6.5.11-7-pve/updates/dkms/ixgbe.ko
version:        5.19.9
license:        GPL
description:    Intel(R) 10GbE PCI Express Linux Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
srcversion:     3AA6B39C75EB10BF0E7186C
alias:          pci:v00008086d000015E5sv*sd*bc*sc*i*
alias:          pci:v00008086d000015E4sv*sd*bc*sc*i*
alias:          pci:v00008086d000015CEsv*sd*bc*sc*i*
alias:          pci:v00008086d000015CCsv*sd*bc*sc*i*
alias:          pci:v00008086d000015CAsv*sd*bc*sc*i*
alias:          pci:v00008086d000015C8sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C7sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C6sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C4sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C3sv*sd*bc*sc*i*
alias:          pci:v00008086d000015C2sv*sd*bc*sc*i*
alias:          pci:v00008086d000015AEsv*sd*bc*sc*i*
alias:          pci:v00008086d000015ADsv*sd*bc*sc*i*
alias:          pci:v00008086d000015ACsv*sd*bc*sc*i*
alias:          pci:v00008086d000015ABsv*sd*bc*sc*i*
alias:          pci:v00008086d000015B0sv*sd*bc*sc*i*
alias:          pci:v00008086d000015AAsv*sd*bc*sc*i*
alias:          pci:v00008086d000015D1sv*sd*bc*sc*i*
alias:          pci:v00008086d00001563sv*sd*bc*sc*i*
alias:          pci:v00008086d00001560sv*sd*bc*sc*i*
alias:          pci:v00008086d00001558sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Asv*sd*bc*sc*i*
alias:          pci:v00008086d00001557sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias:          pci:v00008086d00001528sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F8sv*sd*bc*sc*i*
alias:          pci:v00008086d0000151Csv*sd*bc*sc*i*
alias:          pci:v00008086d00001529sv*sd*bc*sc*i*
alias:          pci:v00008086d0000152Asv*sd*bc*sc*i*
alias:          pci:v00008086d000010F9sv*sd*bc*sc*i*
alias:          pci:v00008086d00001514sv*sd*bc*sc*i*
alias:          pci:v00008086d00001507sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FBsv*sd*bc*sc*i*
alias:          pci:v00008086d00001517sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FCsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F7sv*sd*bc*sc*i*
alias:          pci:v00008086d00001508sv*sd*bc*sc*i*
alias:          pci:v00008086d000010DBsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F4sv*sd*bc*sc*i*
alias:          pci:v00008086d000010E1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*
alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*
alias:          pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        dca
retpoline:      Y
name:           ixgbe
vermagic:       6.5.11-7-pve SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         DKMS module signing key
sig_key:        06:6B:99:06:96:23:9F:1F:B5:08:9E:38:21:4D:14:14:AA:3F:77:07
sig_hashalgo:   sha512
signature:      13:2A:72:4D:BE:01:B8:8D:22:EF:40:82:A0:32:FA:65:A9:7F:BC:46:
                ED:4D:16:87:B9:D4:79:1F:93:1A:7C:B1:D0:71:62:AF:62:44:21:EE:
                AD:36:B8:A1:0E:FF:5D:11:98:1E:F4:12:36:68:87:59:93:4E:DD:B7:
                36:C6:0D:94:1D:FF:B7:FE:F4:9E:88:85:90:83:70:77:F5:4A:36:53:
                31:68:10:0C:A1:62:20:FC:D7:82:52:8E:58:A5:21:94:59:54:30:E7:
                61:C8:A6:18:FA:95:0B:4C:3A:BC:35:4D:68:2A:A2:FB:F3:8A:E9:CB:
                C1:59:28:58:DD:BA:B3:7C:0C:A4:18:50:D2:62:71:25:FE:78:26:60:
                18:5F:2C:58:3B:E4:F6:56:BA:43:44:CD:4E:99:68:6D:02:DB:F2:4F:
                FA:AF:77:C4:97:5C:0C:F8:AC:D0:51:CD:B0:AC:AE:1E:8A:93:0D:5D:
                A1:CC:BD:25:77:07:EC:CB:C3:21:0B:56:19:52:7A:2A:14:C2:A9:A9:
                D3:17:32:5F:64:99:45:67:35:D5:8B:3A:B5:3A:4C:F6:1A:69:AE:BE:
                D6:EC:41:66:CA:C5:44:06:D9:BB:02:79:C8:86:E6:55:A7:5C:FD:9B:
                0E:DA:25:3B:7B:56:CF:32:BA:2E:E9:4D:DB:08:8A:9E
parm:           IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm:           InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
parm:           MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm:           DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm:           RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm:           VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable (1 queue) 2-16 enable (default=8) (array of int)
parm:           max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm:           VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)
parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
parm:           LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm:           LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm:           LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm:           LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm:           LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm:           FdirPballoc:Flow Director packet buffer allocation level:
                        1 = 8k hash filters or 2k perfect filters
                        2 = 16k hash filters or 4k perfect filters
                        3 = 32k hash filters or 8k perfect filters (array of int)
parm:           AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm:           FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm:           MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int)
parm:           LRO:Large Receive Offload (0,1), default 0 = off (array of int)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)
parm:           dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)
parm:           vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)

IOMMU groups:

IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation CometLake-S GT2 [UHD Graphics 630] [8086:9bc5] (rev 05)
IOMMU group 10 01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU group 11 01:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU group 12 02:00.0 Non-Volatile memory controller [0108]: SK hynix Gold P31/PC711 NVMe Solid State Drive [1c5c:174a]
IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation 10th Gen Core Processor Host Bridge/DRAM Registers [8086:9b43] (rev 05)
IOMMU group 2 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
IOMMU group 3 00:12.0 Signal processing controller [1180]: Intel Corporation Comet Lake PCH Thermal Controller [8086:06f9]
IOMMU group 4 00:14.0 USB controller [0c03]: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller [8086:06ed]
IOMMU group 4 00:14.2 RAM memory [0500]: Intel Corporation Comet Lake PCH Shared SRAM [8086:06ef]
IOMMU group 5 00:14.3 Network controller [0280]: Intel Corporation Comet Lake PCH CNVi WiFi [8086:06f0]
IOMMU group 6 00:16.0 Communication controller [0780]: Intel Corporation Comet Lake HECI Controller [8086:06e0]
IOMMU group 6 00:16.3 Serial controller [0700]: Intel Corporation Comet Lake Keyboard and Text (KT) Redirection [8086:06e3]
IOMMU group 7 00:17.0 SATA controller [0106]: Intel Corporation Comet Lake SATA AHCI Controller [8086:06d2]
IOMMU group 8 00:1b.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #21 [8086:06ac] (rev f0)
IOMMU group 9 00:1f.0 ISA bridge [0601]: Intel Corporation Q470 Chipset LPC/eSPI Controller [8086:0687]
IOMMU group 9 00:1f.3 Audio device [0403]: Intel Corporation Comet Lake PCH cAVS [8086:06c8]
IOMMU group 9 00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH SMBus Controller [8086:06a3]
IOMMU group 9 00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH SPI Controller [8086:06a4]
IOMMU group 9 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (11) I219-LM [8086:0d4c]

lspci:

01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

[Sreelakshmi1]

Hi citricut,

Thank you for posting in Intel community forum.

Please allow us sometime to check the details and keep you posted.

Regards,

Sreelakshmi 

[IntelSupport]

Hi citricut,

Good day! I hope this message finds you well.

Thank you for bringing this to our attention. We understand the importance of the matter and would like to assure you that we are actively looking into it. Please allow us some time to conduct a thorough review and analysis.

Regards,

Salman

[IntelSupport]

Hi Citricut,

Thank you for your patience!

To further diagnose the problem, could you please provide the following details:

1. Switch Model and Cable Type:

• What model of switch are you connecting to?
  • Could you specify the type of cable used for the connections?

1. ethtool Outputs:

• Please provide the output of the command "ethtool -k <interface>" for the relevant network interface.

1. ixgbe Driver Loading:

• Were there any error outputs or messages when the ixgbe driver was loaded?

1. In-Kernel Driver and Kernel Version:

• Have you tried using the in-kernel driver?
  • Was kernel version 5.5.3 also used on Fedora 29?

1. LAN LEDs Status:

• When connected, were the LAN LEDs displaying green for 10Gbps or orange for 1Gbps on the link speed LED?

Your prompt response with this information will be greatly appreciated as it will assist us in further diagnosing and resolving the issue.

Thank you for your cooperation.

Best regards,

Salman

[IntelSupport]

Hi Citricut,

Good day!

I hope this email finds you well.

This is a follow-up regarding our previous communication.

Your prompt response is greatly appreciated.

Regards,

Salman

[citricut]

Hi

Thanks for following up. I’ve traced back the issue to the switch, a Horaco managed switch with a 10Gbps SFP+ adapter. After changing the switch both ports connect at 10Gb.
I cannot explain this behaviour though, since the port connected to the switch was properly linking at 10Gb.

You can mark this as solved.

Best regards!

[IntelSupport]

Hi Citricut,

I hope this email finds you well!

I just wanted to extend a heartfelt thank you for confirming the details. It's greatly appreciated, and with that, we're officially closing the case. Your cooperation has been invaluable.

Remember, we're always here to assist you further, so please don't hesitate to reach out if you need anything else. Your satisfaction is our priority.

Thank you once again for choosing Intel products and services.

Warm regards,

Salman