PCIe stalls

[jsch]

With a large fpga FIFO we include FIFO depth in data streaming to a PC and
see periods of time that the fifo is not being read out on the PCIe (PC) side,
i.e. for every write to the FIFO, the depth increases by one. After a history
of seemingly random PCIe stalls of wildly varying times, which would produce
fifo overflow at our desired transfer rates, we now have more capable hardware
systems and see only very specific and repeatable effects. Specifically, within
about 100KB of the stream start, there's a roughly 5 microsecond stall and then,
about 6.5 MB later, a stall of around 90 microseconds resulting in fifo usage
of 10000+    (512b fifo width and 7.04 GB/s). After these two stalls,
and fifo recovery, the following 24TB transfer completes with the fifo depth remaining
below 20. Because this effect is seen on a variety of PC's, Windows, Linux, as well as
other fpga's, I don't know that hardware details are significant.

Is this a familiar phenomenon?

If I can't do anything about this, it will require additional board-level hardware
and additional gateware complexity to buffer unpredictable amounts of data
(due to use in various systems).

So any information from anyone dealing with this, or even being aware of it,
I would truly appreciate. I have not found any mention of this in all my web searching.

[wchiah]

Hi,

Can I know which device that you are using ?
Which example design are you try to run ? AVST, AVMM or MCDMA ?

Meanwhile, 

The phenomenon you're experiencing does sound unusual, especially with the specific and repeatable nature of the stalls.  I can offer some suggestions that might help diagnose or mitigate the problem:

1. Check for PCIe Flow Control: Ensure that the PCIe link is not experiencing flow control issues. If the PCIe link is not able to accept data at the rate it's being sent, it can cause stalls.

2. Investigate Buffer Management: Look into how the FIFO is managed and if there are any conditions under which it might stall. For example, if there are buffer thresholds that trigger certain behaviors, they could be causing these stalls.

3. Consider PCIe Latency: PCIe has inherent latency, and depending on how data is being read from the FIFO on the PC side, it's possible that the stalls are related to PCIe transaction latency.

4. Monitor PCIe Performance Counters: Use PCIe performance counters to monitor the link for errors or anomalies that might indicate issues with the PCIe communication.

5. Check for Software Delays: Ensure that there are no software delays on the PC side that might cause it to temporarily stop reading from the FIFO.

6. Consider Using Scatter-Gather DMA: If possible, consider using Scatter-Gather DMA to transfer data to the PC. This can offload some of the data transfer responsibilities from the CPU and may help alleviate stalls.

7. Update Firmware and Drivers: Ensure that the firmware and drivers for both the FPGA board and the PC are up to date, as older versions might have bugs or limitations that could cause these stalls.

Regards,

Wincent_Intel

[wchiah]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

p/s: If any answer from the community or Intel Support is helpful, please feel free to give the best answer or rate 9/10 survey.

[wchiah]

Hi

We have not hear from you and this Case is idling. It is not recommended to idle for too long.

Therefore following our support policy, I have to put this case in close status. My apologies if any inconvenience cause

Hence, This thread will be transitioned to community support.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If your support experience falls below a 9 out of 10, I kindly request the opportunity to rectify it before concluding our interaction. If the issue cannot be resolved, please inform me of the cause so that I can learn from it and strive to enhance the quality of future service experiences. 

Regards,

Wincent_Intel

p/s: If any answer from the community or Intel Support is helpful, please feel free to give the best answer or rate 9/10 survey.