- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello!
I have a problem with Intel XL710, after 22 Gbps of traffic some cores of processor loads 100% IRQ and traffic goes down.
There are latast firmware and driver:
# ethtool -i ens2
driver: i40e
version: 2.24.6
firmware-version: 9.40 0x8000ecc0 1.3429.0
expansion-rom-version:
bus-info: 0000:d8:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
I read this guide
And made all possible configurations. But nothing changes
382 set_irq_affinity local ens2
384 set_irq_affinity all ens2
387 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
389 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 5000 tx-usecs 20000
391 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 10000 tx-usecs 20000
393 ethtool -g ens2
394 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 84 tx-usecs 84
396 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 62 tx-usecs 62
398 ethtool -S ens2 | grep drop
399 ethtool -S ens2 | grep drop
400 ethtool -S ens2 | grep drop
401 ethtool -S ens2 | grep drop
402 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 336 tx-usecs 84
403 ethtool -S ens2 | grep drop
404 ethtool -S ens2 | grep drop
406 ethtool -S ens2 | grep drop
407 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 672 tx-usecs 84
408 ethtool -S ens2 | grep drop
409 ethtool -S ens2 | grep drop
411 ethtool -S ens2 | grep drop
412 ethtool -S ens2 | grep drop
425 ethtool -S ens2 | grep drop
426 ethtool -S ens2 | grep drop
427 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 8400 tx-usecs 840
428 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 840
430 ethtool -S ens2 | grep drop
431 ethtool -S ens2 | grep drop
432 ethtool -S ens2 | grep drop
433 ethtool -S ens2 | grep drop
434 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1680
435 ethtool -S ens2 | grep drop
436 ethtool -S ens2 | grep drop
439 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 3200 tx-usecs 3200
469 ethtool -a ens2
472 ethtool ens2
473 ethtool -i ens2
475 ethtool -i ens2
476 ethtool ens2
482 ethtool -C ens2 adaptive-rx on
484 ethtool -c ens2
486 ethtool -C ens2 adaptive-tx on
487 ethtool -c ens2
492 history | grep ens2
494 ethtool -m ens2
499 ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1600
501 history | grep ens2
Server configuration:
80 cores Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
RAM 960 GB
8 x SAMSUNG MZQLB7T6HMLA-000AZ NVME disks
Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
What can be done with the settings of this network card to solve the problem?
On another server with a similar configuration, but a different network cards, everything is fine
Server configuration:
88 cores Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
RAM 512 GB
8 x SAMSUNG MZQLB7T6HMLA-00007 NVME disks
4 x 82599ES 10-Gigabit SFI/SFP+ Network Connection
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi AndriiV,
Thank you for reaching out to the Intel community. We sincerely apologize for the delay in our response.
We acknowledge receipt of your concern and want to assure you that assisting you is our top priority. To effectively troubleshoot the issue you're facing, we require some additional information from your end.
Could you kindly provide the following details?
- System Model:
- Operating System:
- Is the Ethernet card embedded on the board?
- Was the Ethernet card working before? Have any changes been made recently?
Your prompt response with this information will greatly assist us in diagnosing and resolving the issue as quickly as possible.
We look forward to hearing from you soon.
Best Regards,
Simon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Simon!
1. Supermicro SuperServer 1029U-TRTP2
2. Debian 10.13
# uname -a
Linux 15224 4.19.0-26-amd64 #1 SMP Debian 4.19.304-1 (2024-01-09) x86_64 GNU/Linux
3. No
4. No, I am using XL710 card for a first time
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi AndriiV,
Thank you for sharing the requested details.
As you mentioned, there is another server with a similar configuration, but with different network cards. Could you please share the server motherboard and the network card details?
We look forward to hearing from you soon.
Best Regards,
Simon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Simon!
Supermicro SuperServer 1028U-TR4+
Motherboard: X10DRU-i+
Debian 10.13
# uname -a
Linux 157540 4.19.0-25-amd64 #1 SMP Debian 4.19.289-2 (2023-08-08) x86_64 GNU/Linux
4x10Gbps network cards:
87:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
87:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Andrii,
Good day!
Thank you for providing the requested details.
We are currently investigating this inquiry with our resources, and we will provide you with an update at the earliest opportunity.
Thank you for your patience and understanding.
Best regards,
Simon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Andrii,
Good Day!
Thank you for your patience as we work to resolve the issue you've encountered. We at Intel are fully committed to ensuring your satisfaction.
To address your concern, please refer to the following link for detailed instructions: Intel Ethernet 700 Series Linux Performance Tuning Guide - Section 4.1.
Your understanding and cooperation are greatly appreciated.
Best regards,
Simon
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Simon!
As you can see my first message in this topic. I already read this guide and made all tunnings. I tried
382 set_irq_affinity local ens2
384 set_irq_affinity all ens2
With "local" configuration 100% IRQ on some cores at 19Gbps yet. You can see atop screen after
set_irq_affinity local ens2
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I think the problem is that the traffic is unevenly distributed across the queues
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Andrii,
Good Day!
Firstly, I'd like to extend our apologies for any inconvenience caused by our previous communication. Your detailed explanation of the issue and server configuration was immensely helpful, and we appreciate your patience and cooperation.
I'm writing to inform you that we've escalated the matter to our next level team for further investigation and resolution. Rest assured, we're prioritizing this issue to ensure a swift resolution.
We will keep you updated on the progress and provide any necessary information as soon as it becomes available.
Once again, thank you for your understanding and cooperation throughout this process.
Best regards,
Simon
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page