Only 1 UPI is working

[WeixingSun]

Hi Intel Experts,

We have a 2-socket server of 5318Y, each CPU has a PCIe-switch, and 4 GPUs attached to the PCIe-switch. as image attached : arch1.jpg

Now we encountered an issue of UPI bandwidth when doing DMA:

the DMA between 2 cards under different CPUs limited to about 15GB/s, like card0--card4

but the DMA between 2 cards under the same CPU looks good, like card0--card3

and we looked at the UPI data from PCM seems only 1 UPI is getting utilized and reached 99% other 2 UPIs leaving 0%.

Please help to identify where is the issue, and how to resolve it.

Thanks !

[Mike_Intel]

Hello WeixingSun,

Thank you for posting in Intel community Forum.

Base on your inquiry, we have specific forum for these issues or inquiries. I will be transferring this thread for faster response.

If you have questions, please let us know.

Best regards,

Michael L.

Intel Customer Support Technician

[WeixingSun]

Thank you Michael, appreciate it so much!