Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4813 Discussions

Intel SR-IOV XL710 VF doesn't work after VM reboot

KPoku
Beginner
4,662 Views

I'm using Ubuntu 16.04.3 LTS with KVM virtualization. Also i'm using

Intel XL710 card with SR-IOV feature.

Here are the outputs:

/lib/modprobe.d$ modinfo i40e

filename:      /lib/modules/4.4.0-96-generic/updates/drivers/net/ethernet/intel/i40e/i40e.ko

version:       2.4.10

license:       GPL

description:   Intel(R) 40-10 Gigabit Ethernet Connection Network Driver

author:        Intel Corporation, <e1000-devel@lists.sourceforge.net>

srcversion:    3977C21019A3C4865FF253A

alias:         pci:v00008086d0000158Bsv*sd*bc*sc*i*

alias:         pci:v00008086d0000158Asv*sd*bc*sc*i*

alias:         pci:v00008086d000037D3sv*sd*bc*sc*i*

alias:         pci:v00008086d000037D2sv*sd*bc*sc*i*

alias:         pci:v00008086d000037D1sv*sd*bc*sc*i*

alias:         pci:v00008086d000037D0sv*sd*bc*sc*i*

alias:         pci:v00008086d000037CFsv*sd*bc*sc*i*

alias:         pci:v00008086d000037CEsv*sd*bc*sc*i*

alias:         pci:v00008086d00001588sv*sd*bc*sc*i*

alias:         pci:v00008086d00001587sv*sd*bc*sc*i*

alias:         pci:v00008086d00001589sv*sd*bc*sc*i*

alias:         pci:v00008086d00001586sv*sd*bc*sc*i*

alias:         pci:v00008086d00001585sv*sd*bc*sc*i*

alias:         pci:v00008086d00001584sv*sd*bc*sc*i*

alias:         pci:v00008086d00001583sv*sd*bc*sc*i*

alias:         pci:v00008086d00001581sv*sd*bc*sc*i*

alias:         pci:v00008086d00001580sv*sd*bc*sc*i*

alias:         pci:v00008086d00001574sv*sd*bc*sc*i*

alias:         pci:v00008086d00001572sv*sd*bc*sc*i*

depends:       ptp,vxlan

vermagic:      4.4.0-96-generic SMP mod_unload modversions

parm:          debug:Debug level (0=none,...,16=all) (int)

 

 cat /etc/rc.local

#!/bin/sh -e

#

# rc.local

#

#

# By default this script does nothing.

 

 

echo 8 > /sys/class/net/enp6s0f1/device/sriov_numvfs

echo 8 > /sys/class/net/enp6s0f0/device/sriov_numvfs

 

ip link set enp6s0f0 vf 0 mac aa:bb:cc:aa:00:02 vlan 12

ip link set enp6s0f0 vf 1 mac aa:bb:cc:aa:01:02 vlan 32

ip link set enp6s0f0 vf 2 mac aa:ff:ff:aa:00:12 vlan 22

ip link set enp6s0f0 vf 3 mac aa:ff:ff:aa:01:12 vlan 42

 

ip link set enp6s0f1 vf 0 mac aa:bb:cc:bb:00:02 vlan 52

ip link set enp6s0f1 vf 1 mac aa:bb:cc:bb:01:02 vlan 72

ip link set enp6s0f1 vf 2 mac aa:ff:ff:bb:00:12 vlan 62

ip link set enp6s0f1 vf 3 mac aa:ff:ff:bb:01:12 vlan 82

 

exit 0

 

Driver i40e is also blacklisted as suggested in Intel SR-IOV configuration guide.

Everything works ok, but in case i reboot the VM, some of the virtual functions won't work. I need to reboot VM a few times (sometimes i need just one reboot, sometimes like three reboots) and virtual functions start working properly.

 

I was able only to find this in logs on KVM host (/var/log/kern.log) when VF doesn't work:

Jan 8 09:56:23 ri-cgn-kvm-2 kernel: [15787510.840948] i40e 0000:06:00.0: VF 1 failed opcode 3, retval: -5

Jan 8 09:56:23 ri-cgn-kvm-2 kernel: [15787510.841010] i40e 0000:06:00.0: Invalid message from VF 1, opcode 3, len 4

Jan 8 09:57:08 ri-cgn-kvm-2 kernel: [15787555.936159] i40e 0000:06:00.1: VF 0 failed opcode 10, retval: -5

Jan 8 09:57:08 ri-cgn-kvm-2 kernel: [15787555.936222] i40e 0000:06:00.1: Invalid message from VF 0, opcode 10, len 44

Jan 8 09:57:08 ri-cgn-kvm-2 kernel: [15787555.976198] i40e 0000:06:00.1: VF 0 failed opcode 11, retval: -10

Jan 8 10:05:14 ri-cgn-kvm-2 kernel: [15788042.047074] i40e 0000:06:00.0: VF 0 failed opcode 3, retval: -5

Jan 8 10:05:14 ri-cgn-kvm-2 kernel: [15788042.047138] i40e 0000:06:00.0: Invalid message from VF 0, opcode 3, len 4

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.950375] i40e 0000:06:00.0: VF is not trusted, switch the VF to trusted to add more functionality

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.951291] i40e 0000:06:00.1: VF is not trusted, switch the VF to trusted to add more functionality

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.951300] i40e 0000:06:00.1: VF 1 failed opcode 10, retval: -1

Jan 8 10:21:14 ri-cgn-kvm-2 kernel: [15789001.977115] i40e 0000:06:00.0: VF 1 failed opcode 10, retval: -1

Jan 8 10:37:25 ri-cgn-kvm-2 kernel: [15789972.835652] i40e 0000:06:00.0: VF 1 failed opcode 11, retval: -10

Jan 8 10:37:27 ri-cgn-kvm-2 kernel: [15789975.074702] i40e 0000:06:00.1: VF 1 failed opcode 11, retval: -10

 

What is causing this issue?

 

Regards,

Kresimir

 

0 Kudos
1 Solution
Daniel_D_Intel1
Employee
2,820 Views
Hello KPoku, Thank you for the information. The kernel is not providing the latest i40e drivers. Please check if updating to the latest i40e drivers resolves the issue. Please check with Cisco if there is a firmware update available for the XL710, as this is an OEM Cisco version. They might also be able to provide the compatibility of drivers with the latest firmware. You can find the latest drivers here: https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux- Let us know if the drivers and firmware update solve this issue. If you have any other questions please inform us. Best regards, Daniel D Intel Customer Support

View solution in original post

0 Kudos
11 Replies
Daniel_D_Intel1
Employee
2,820 Views
Hello KPoku, Thank you for posting in Intel Ethernet Communities. Please let us know if this is a retail Intel version of the XL710 adapter or an OEM by providing the markings from the product label. Please include ethtool -i <interface> output to confirm the firmware, and ethtool -k <interface> to provide the features on the NIC. Let us know if you have any other questions. Best regards, Daniel D Intel Customer Support
0 Kudos
KPoku
Beginner
2,820 Views

Hi Daniel,

 

here are outputs of ethtool command:

ethtool -i enp1s0f0

driver: igb

version: 5.3.0-k

firmware-version: 1.63, 0x80000e74, 0.385.8

expansion-rom-version:

bus-info: 0000:01:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

 

ethtool -k enp1s0f0

Features for enp1s0f0:

rx-checksumming: on

tx-checksumming: on

       tx-checksum-ipv4: on

       tx-checksum-ip-generic: off [fixed]

       tx-checksum-ipv6: on

       tx-checksum-fcoe-crc: off [fixed]

       tx-checksum-sctp: on

scatter-gather: on

       tx-scatter-gather: on

       tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

       tx-tcp-segmentation: on

       tx-tcp-ecn-segmentation: off [fixed]

       tx-tcp6-segmentation: on

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

busy-poll: off [fixed]

hw-tc-offload: off [fixed]

 

 

ethtool -i enp1s0f1

driver: igb

version: 5.3.0-k

firmware-version: 1.63, 0x80000e74, 0.385.8

expansion-rom-version:

bus-info: 0000:01:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

 

ethtool -k enp1s0f1

Features for enp1s0f1:

rx-checksumming: on

tx-checksumming: on

       tx-checksum-ipv4: on

       tx-checksum-ip-generic: off [fixed]

       tx-checksum-ipv6: on

       tx-checksum-fcoe-crc: off [fixed]

       tx-checksum-sctp: on

scatter-gather: on

       tx-scatter-gather: on

       tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

       tx-tcp-segmentation: on

       tx-tcp-ecn-segmentation: off [fixed]

       tx-tcp6-segmentation: on

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-segmentation: off [fixed]

tx-udp_tnl-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: off

loopback: off [fixed]

rx-fcs: off [fixed]

rx-all: off

tx-vlan-stag-hw-insert: off [fixed]

rx-vlan-stag-hw-parse: off [fixed]

rx-vlan-stag-filter: off [fixed]

l2-fwd-offload: off [fixed]

busy-poll: off [fixed]

hw-tc-offload: off [fixed]

 

This Intel cards were ordered and provided with Cisco. I'm running CISCO UCS C220M4S server.

 

Regards,

Kresimir

 

0 Kudos
Daniel_D_Intel1
Employee
2,821 Views
Hello KPoku, Thank you for the information. The kernel is not providing the latest i40e drivers. Please check if updating to the latest i40e drivers resolves the issue. Please check with Cisco if there is a firmware update available for the XL710, as this is an OEM Cisco version. They might also be able to provide the compatibility of drivers with the latest firmware. You can find the latest drivers here: https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux- Let us know if the drivers and firmware update solve this issue. If you have any other questions please inform us. Best regards, Daniel D Intel Customer Support
0 Kudos
Daniel_D_Intel1
Employee
2,820 Views
Hello KPoku, Please let us know if you are still having any issues after updating to the latest i40e driver and checking if there is a firmware update from Cisco. If you have any other questions please do not hesitate to ask. Best regards, Daniel D Intel Customer Support
0 Kudos
Daniel_D_Intel1
Employee
2,820 Views
Hello KPoku, Please let us know if you still need assistance. If you have any other questions please do not hesitate to ask. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation
0 Kudos
KPoku
Beginner
2,820 Views

Hi Daneil,

 

i've installed latest drivers, v.2.7.26, one host and didn't experience issue with VF.

I'll be testing for next couple of days to see will issue appear.

 

Regards,

Kresimir

0 Kudos
Daniel_D_Intel1
Employee
2,820 Views
Hello KPoku, Thank you for the reply. Please let us know if the issue is still seen in your testing. If you have any other questions please ask. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation
0 Kudos
Daniel_D_Intel1
Employee
2,821 Views
Hello KPoku, After testing the latest driver for some time are you still getting the expected VF performance? Please let us know if you need additional assistance, or have any questions. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation
0 Kudos
Daniel_D_Intel1
Employee
2,821 Views
Hello KPoku, Please let us know if you have any questions. If the driver update solved the issue please inform us. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation
0 Kudos
Mike_Intel
Moderator
2,821 Views
Hello KPoku, Please tell us If the driver update solved the issue. Kindly tell us if you have any questions. Best regards, Michael L. Intel Customer Support Under contract to Intel Corporation
0 Kudos
KPoku
Beginner
2,821 Views

Hi Michael,

 

for now i didn't have any issues with latest drivers, so i can say that issue is solved.

Thank you for the assistance.

 

Regards,

Kresimir

0 Kudos
Reply