Performance Degradation in CPU v4 vs CPU v3

[aAsra]

We are running IOR benchmarking on a Lustre filesystem and we see performance degradation when upgrading from cpu v3 to cpu v4;

We are seeing cpus overload under IOR benchmark, this is occurring after cpu v3 to v4 upgrade

As a result, we see performance degradation after the upgrade..

can someone take a look at the syslog to see if this is a known issue:

Syslog messages and snapshot of the cpus utilizations are below:

kernel:NMI watchdog: BUG: soft lockup - CPU# 54 stuck for 22s! [ll_ost_io01_107:24645]

Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 3 stuck for 23s! [ptlrpcd_00_13:7854]

Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 10 stuck for 23s! [ptlrpcd_00_07:7848]

Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 46 stuck for 22s! [ll_ost_io01_047:49948]

Message from syslogd@OSS1-A212 at Aug 21 15:53:49 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 48 stuck for 22s! [ll_ost_io01_092:24578]

Message from syslogd@OSS1-A212 at Aug 21 15:53:53 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 4 stuck for 22s! [ptlrpcd_00_19:7860]

Message from syslogd@OSS1-A212 at Aug 21 15:53:53 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 27 stuck for 23s! [ll_ost_io01_086:22338]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 0 stuck for 22s! [ptlrpcd_00_08:7849]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 12 stuck for 22s! [ptlrpcd_00_14:7855]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 15 stuck for 22s! [ll_ost_io01_008:50182]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 17 stuck for 22s! [kiblnd_sd_01_02:7835]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 18 stuck for 22s! [kiblnd_sd_01_01:7834]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 20 stuck for 22s! [ll_ost_io01_016:44925]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 21 stuck for 22s! [ll_ost_io01_076:22069]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 22 stuck for 22s! [kiblnd_sd_01_00:7833]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 23 stuck for 22s! [ll_ost_io01_094:24592]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 24 stuck for 22s! [kiblnd_sd_01_03:7836]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 25 stuck for 22s! [ll_ost_io01_106:24643]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 26 stuck for 22s! [ll_ost_io01_064:33777]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 36 stuck for 22s! [kiblnd_sd_00_00:7829]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 43 stuck for 22s! [ll_ost_io01_031:42636]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 44 stuck for 22s! [ll_ost_io01_109:24689]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 49 stuck for 22s! [ll_ost_io01_046:50178]

Message from syslogd@OSS1-A212 at Aug 21 15:53:57 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 53 stuck for 22s! [ll_ost_io01_029:37763]

Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 8 stuck for 22s! [ptlrpcd_00_09:7850]

Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 16 stuck for 22s! [ll_ost_io01_098:24613]

Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 32 stuck for 22s! [ptlrpcd_00_12:7853]

Message from syslogd@OSS1-A212 at Aug 21 15:54:01 ...

kernel:NMI watchdog: BUG: soft lockup - CPU# 42 stuck for 22s! [ll_ost_io01_024:39085]

Message from syslogd@OSS...

[AlHill]

What are the processor model numbers?

What motherboard and BIOS are you using?

What operating system are you using?

Doc

[aAsra]

Hi Doc,

Operating System: Centos 7.3

cpu info:

ndor_id : GenuineIntelcpu family : 6model : 79model name : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

Motheboard and Bios info:

description: Rack Mount Chassis

product: X10DRH (085915D9)

vendor: Supermicro

version: 123456789

serial: 123456789

width: 64 bits

capabilities: smbios-3.0 dmi-3.0 smp vsyscall32

configuration: boot=normal chassis=rackmount family=SMC X10 sku=085915D9 uuid=00000000-0000-0000-0000-002590F9F7AE

*-core

description: Motherboard

product: X10DRH-IT

vendor: Supermicro

physical id: 0

version: 1.00

serial: OM14BS004942

slot: Default string

*-firmware

description: BIOS

vendor: American Megatrends Inc.

physical id: 0

version: 2.0

date: 12/17/2015

size: 64KiB

capacity: 8128KiB

capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi

thanks,

Abe

[idata]

Hello aasraoui,

I know that you are expecting support here but please be aware that your best contact support is with your system manufacturer (Supermicro). I know that this is an Intel processor but your system manufacturer is more familiar with your PC features, specifications and how all the components interact with each other. At this point for you to get the proper support you can contact here:http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRH-iT.cfm http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRH-iT.cfmNOTE: This link is being offered for your convenience and should not be viewed as an endorsement by Intel of the content, products, or services offered there. Regards, Ivan U.

[GNiet]

Hi Abe

Were you able to find a root cause or resolution? we are experiencing the same issue with the v4 procs. Our environment has over 50 FC630's running with the v3 proc's with no issues, we added 30 FC630's running with the v4 proc's into the same environment and most of them are having CPU Softlocks issues when running under a heavy load.

Operating System: Ubuntu 14.04

Hardware Info: Dell PowerEdge FC630

CPU Info:

vendor_id : GenuineIntel

cpu family : 6

model : 79

model name : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

stepping : 1

microcode : 0xb00001f

BIOS:

BIOS Information

Vendor: Dell Inc.

Version: 2.4.2

Release Date: 01/09/2017

Address: 0xF0000

Runtime Size: 64 kB

ROM Size: 16384 kB

Characteristics:

ISA is supported

PCI is supported

PNP is supported

BIOS is upgradeable

BIOS shadowing is allowed

Boot from CD is supported

Selectable boot is supported

EDD is supported

Japanese floppy for Toshiba 1.2 MB is supported (int 13h)

5.25"/360 kB floppy services are supported (int 13h)

5.25"/1.2 MB floppy services are supported (int 13h)

3.5"/720 kB floppy services are supported (int 13h)

8042 keyboard services are supported (int 9h)

Serial services are supported (int 14h)

CGA/mono video services are supported (int 10h)

ACPI is supported

USB legacy is supported

BIOS boot specification is supported

Function key-initiated network boot is supported

Targeted content distribution is supported

UEFI is supported

BIOS Revision: 2.4