Intel processors and 40Gbps NICs

[MFerr8]

I am trying to achieve a 40Gbps, single TCP flow, throughput out of a40Gbps NIC. To prevent any disk latency, I have created a RAM disk of 80GB on the server and on the client. Then I have created a 40GB file in this RAM disk that is also the root of my ftp server. When I do the download of this 40GB file on the client, the speed never goes over about 12Gbps while I would expect to saturate the NIC at 40Gbps. Doing some further analysis, I have done a copy of the file residing in the RAM disk to the same RAM disk, only changing the name. The result is that it takes almost the same time it takes to send the file via the 40Gbps NIC to the client. My conclusion was that the memory sub-system can not write in such speed. However, if I start a second download at the same time, pinning the download to a different CPU, then the speed just doubles. And if I start a third download it triples.

Based on this observation, I believe that the problem is that every core on a CPU must have a defined memory access time slot to prevent starvation of memory access and also to prevent a single core to "block" the entire system from working properly. However, this memory slot is not long enough to produce the 40Gbps that I would like to have on a single stream. To test my theory, I have removed the second system CPU (socket) and disabled all cores, except 2 ones. I did the same test and found that the in memory file copy would gain over 40% speed.

All this said, I "think" that the best way to achieve a very high throughput in such scenario should be by disabling as much as possible all the cores so the remaining ones would have a longer time slot to write on RAM.

My questions: Am I my missing something or this is the way it is? I believe that the time slot control for memory bus access is strictly controlled by Intel CPU. If it is not, can you give me some hints on how to tailor it?

Thanks in advance for your feedback.

Moacir

[idata]

Hi Moacir,

Thank you for the post. What is the 40 Gbps NIC you refer here? What is your system and operating system used? Thanks,wb

[MFerr8]

The main specs of the servers I am using is:

Huawei H2288H V3 server

Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz CPU

4 x 32GB DDR4 2133MHz RAM

Chelsio T580-LP-CR NIC

Red Hat Linux Enterprise version 7.3 OS

[idata]

HI Moacir,

Thank you for the information provided. It is best you can contact Huawei support for them to further assist you as this involve the entire system. Chelsio is a third party NIC which we cannot comment on this product. With regards to the CPU inquiry, I can forward your inquiry to the CPU support. Rgds,wb

[MFerr8]

I am already in contact with Huawei R&D and I also have contacted Chelsio trying to pin-point where limitation is.

If you can, please forward the case to the CPU support. My goal is to find what is the best Intel CPU and chip-set for such scenario, while also learning the limits of the hardware I am currently using.

Thank you for your support.

Moacir

[idata]

Hi Moacir,

Thank you for the information. I forwarded your inquiry to processor support. Thanks,wb

[MFerr8]

So far I got no response... Has this case been sent to the processor? How can I get an update on it.

Thanks,

Moacir

[idata]

Hello Moacir,

Thank you for contacting the Intel community. Please bear in mind that your best support is with your system manufacturer as they are more familiar with your system and how all the components interact with each other. Intel sells products to various OEM's (Original Equipment Manufacturers), who then integrate them, with other system components (power supplies, disk drives, add in cards, software, etc). These components may be customized to the OEM's specifications to make the entire system compatible and to work as it should. This Intel® processor supports Max Memory Bandwidth 76.8 GB/s but all this depends on your system configuration you can see the processor specs here:https://ark.intel.com/products/91767/Intel-Xeon-Processor-E5-2650-v4-30M-Cache-2_20-GHz https://ark.intel.com/products/91767/Intel-Xeon-Processor-E5-2650-v4-30M-Cache-2_20-GHz You can also try updating your motherboard BIOS with your system manufacturer to see if that helps with this issue. You should be able to get support about your system here:http://e.huawei.com/en/products/cloud-computing-dc/servers/rh-series/rh2288-v3 http://e.huawei.com/en/products/cloud-computing-dc/servers/rh-series/rh2288-v3 Regards, Ivan.

[MFerr8]

Ivan,

Thank you for your response.

It is more like that I am on a "ping-pong" game as the Huawei R&D (Research and Development) says it is an Intel CPU design problem and, in turn, you (Intel) says it may be a "make" problem. Anyway, as long as I know Huawei R&D is watching this thread and maybe they jump in to clarify who got the problem, if any.

This said, my customer is driving a Very Big Data environment and wiling to be able to download files, over a single flow, at 40Gpbs. I may be taking the wrong path as I am using RAM memory to emulate disk, willing that it would be the best solution for a disk-subsystem able to deliver such expected 40Gbs throughput (please see my initial post). If the "make" I am using now has a problem to achieve this goal, what make (Intel or Intel partner) could deliver such performance? What would be the Intel advise for such performance?

Moacir

[idata]

Hello Moacir,

I have sent you a private message.Regards,Ivan.