FPGA Intellectual Property
PCI Express*, Networking and Connectivity, Memory Interfaces, DSP IP, and Video IP
6343 Discussions

PCIe HIP on Sandy Bridge E

Altera_Forum
Honored Contributor II
1,082 Views

Hello all, 

 

I am working on a project with SIV utilizing the PCIe 2.0 HIP as a legacy endpoint. Things have been working quite well for us at first, but recently we have run into an issue on certain platforms; specifically Sandy Bridge E. 

 

With a LeCroy PCIe Analyzer, I see that a bad TLP is being sent to the Root Complex from our FPGA. Typically it has been a undefined format and type, but I have caught it as a CplLk TLP -- something my TLP-layer logic would not send. I used SignalTap to examine all the headers I send to the HIP and cannot find any issue in our code, and none of these bad headers. After this TLP is sent, all upstream TLPs are malformed -- they almost appears to be shifted (i.e. I see 0x4A00010 in the address field, a possible CplD). 

 

I really don't know where to begin on this one. I have tried it on Sandy Bridge platforms and everything functions as expected (all under centOS). My next step is to use a reference design to see how it behaves in this system. 

 

Has anyone else run into this problem? Any suggestions from the community? Thanks in advance. 

 

Jay
0 Kudos
3 Replies
Altera_Forum
Honored Contributor II
322 Views

 

--- Quote Start ---  

Hello all, 

 

I am working on a project with SIV utilizing the PCIe 2.0 HIP as a legacy endpoint. Things have been working quite well for us at first, but recently we have run into an issue on certain platforms; specifically Sandy Bridge E. 

 

With a LeCroy PCIe Analyzer, I see that a bad TLP is being sent to the Root Complex from our FPGA. Typically it has been a undefined format and type, but I have caught it as a CplLk TLP -- something my TLP-layer logic would not send. I used SignalTap to examine all the headers I send to the HIP and cannot find any issue in our code, and none of these bad headers. After this TLP is sent, all upstream TLPs are malformed -- they almost appears to be shifted (i.e. I see 0x4A00010 in the address field, a possible CplD). 

 

I really don't know where to begin on this one. I have tried it on Sandy Bridge platforms and everything functions as expected (all under centOS). My next step is to use a reference design to see how it behaves in this system. 

 

Has anyone else run into this problem? Any suggestions from the community? Thanks in advance. 

 

Jay 

--- Quote End ---  

 

 

 

Hello Jay, 

 

I have been desperately looking for a solution for a problem we have with our custom StratixIV based PCIe boards in conjunction with Sandybridge EP motherboards. 

It seems that you experienced a similar problem back in 2012. So, my question is whether or not you were able to resolve this problem. 

 

Unfortunately I do not have a PCIe bus analyzer, however here is some more information about our findings: 

 

Our custom board utilizes the PCIe HIP as an endpoint in a Gen2x8 configuration. 

It runs flawlessly with various older motherboards, but we do have an issue with Sandybridge EP based motherboards. 

With this type of motherboards the achievable bandwidth (FPGA -> PC memory direction) drops dramatically (from greater than 20Gbit/s to less than 3Gbit/s). 

Our board is installed in a x16 slot and seems to be properly enumerated in a Gen2x8 configuration (as reported by the 'lspci' command under Linux OS). 

 

Amongst others we are using Supermicro servers for our tests. 

We found by trial and error that we can fix the problem on >this< machine by changing the PCIe port configuration in the BIOS: 

>>> PCIE port Bifurcation Control - IOUx - PCIe Port - [x16] -> [x8x8] <<< 

After changing this setting to [x8x8] our board works as expected. 

 

For our final product we are targeting 2U rackmount servers from Asus. These machines seem to use the same BIOS as the Supermicro servers (even the same version). 

However, the setup utility does NOT have an option to change the PCIe port configuration. So, for the time being we have no solution to make our boards work in the Asus machines. 

 

Any comment on this issue will be greatly appreciated. 

 

Best Regards, 

Volker
0 Kudos
Altera_Forum
Honored Contributor II
322 Views

We did figure out the problem that we were experiencing. It ended up being a logic bug in the handling of MSI-X TLPs being sent to the PCIe IP core. The Sandy-Bridge motherboard would use different DW alignment, and an empty flag was not being properly set for MSI-X TLPs. In our other systems, the bug would not show itself since the addressing (32-bit vs. 64-bit for the Sandy Bridge) didn't call for the flag. 

 

I am not sure that my experience is similar to what you are currently seeing. I found that once the core is fed a badly formed TLP from the logic, it completely breaks down and will transmit garbage. It is at least worthwhile to verify, but since you were able to correct your performance via software, it is not likely the same issue. 

 

Good luck! There are many others on these forums that are vastly more knowledgeable about PCIe and Linux that should be able to chime in and possibly help. 

 

Jay
0 Kudos
Altera_Forum
Honored Contributor II
322 Views

I agree, most likely this one is a different issues. I am pretty sure that my TLPs fed into the PCIe core are correct. 

Thanks anyway for taking the time and for your quick response. 

 

Volker
0 Kudos
Reply