Intel X540-T2 small packet performance tuning

[WAnge1]

Hello,

I wonder what small-packet throughput is possible with this setup if proper configured:

OS: SLES12-SP2

Kernel: 4.4.74-92.32-default

NIC: Intel X540-T2 Dual Port 10GBaseT

Server: Lenovo System x3550 M5, 12 CPU-Cores "Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz"

smp_affinity: optimized, all 12 cores are in use by the NICs IRQs

This is a loadbalancer for DNS packets. Mainly small (~80 bytes) UDP packets.

Loadbalancer is "ipvs" in DR mode. This means the kernel does only a re-routing (MAC-rewrite) of the packet and push it back on the network (same adapter).

With this setup the server is able to process ~700kpps. At this load, all CPU cores are 100% busy (softirq).

My questions to you are, do you think this can be improved? Have we missed something?

If we direct all traffic to one single cpu core, the server is able to process already ~350kpps! Why does the system scale this bad if 12 cores are in use?

I'm grateful for every tip

Thanks

Winfried

[idata]

Hi Gnaba,

Thank you for posting at Wired Communities. Just to double check are you saying with 12 cores CPU, you can only process 700 kpps of packets but with only one single core cpu, it can process 350 Kpps already? Have you check with Loadbalancer the application vendor for the configuration? Thanks,sharon

[WAnge1]

Hi sharon,

Have you check with Loadbalancer the application vendor for the configuration?

Yes, now I have

Long story short, the kernel module "ip_vs" (loadbalancer) was wrong configured.

options ip_vs conn_tab_bits=20

did the trick!

Now we're able to process >1Mpps with 60% CPU idle. That's great!

Thanks a lot!

Winfried

[idata]

Hi Gnaba,

Thank you for the update and glad to know the configuration did the trick and issue is fixed. regards,sharon