Multi-jtag server configuration

[Altera_Forum]

We are starting to ramp up and deploy several development systems in our lab. The current count is now at 24 FPGA based systems, with an expected growth of 30 more by Q2/Q3 timeframe, and over 100 by EOY'15. To support this deployment, we need to move beyond 1 host (Quartus)<>1 Target (FPGA). We are using the USB Blaster cables currently (may switch to USB Blaster II if mechanical issues are resolved). In our current testing, we have deployed 4-5 target systems per jtag server. This works ok for automated programming of an FPGA and exiting, but falls flat when remote quartus A is programming FPGA 1, and remote quartus B is running a SignalTap debug session on FPGA 2 (we do not plan on having multiple quartus instances looking at 1 FPGA - not a very good support model anyways). Unfortunately, (from what we have experienced to date), jtagd (Linux) only supports 1 Quartus connection at a time (seeing and reporting multiple Blasters is not a problem). 

 One solution we were told about is to have the server run as a VM server, with each blaster controlled by an independent VM (USB pass-through). This is not desireable as it takes more IP addresses to support. Most of our quartus hosts are running in a VM, but we want to be able to dynamically allocate an FPGA target, and each user can own their own VM, requesting a target from the pool.  We need to get this functional ASAP. Any suggestions? We are willing to help with the development/testing of jtagd to support this functionality (which shouldn't be too difficult - it already exists in printer and scanner servers).  Tobin Davis Lab Manager Cloud Platform Technologies - Intel Corp.

[Altera_Forum]

Please don't be offended by the following questions, I'm asking them in a serious context (not treating you as a "newbie"). 

 First question: Why do you need the USB-Blasters?  Obvious answer: for debug tools like SignalTap II.  So how many systems do you need to debug at one time; all of them, or just a few?  Reconfiguration of FPGAs can easily be handled by any number of schemes, but of course, that depends on how your hardware is designed.  Cheers, Dave

[Altera_Forum]

First, let me state that I am a systems administrator and lab manager, not a hardware engineer. 

 Our module has a 10 pin header for the USB Blaster to plug into for programming and probing the FPGA. We currently have 24 systems with modules, and due to the configuration the module needs to be programmed while the system bios is in a loop waiting for discovery. The systems currently are not self-programming, thus the need for Quartus on a 'Host' system. We have a team of engineers doing various debug and optimizations of our base RTL, thus the reason for Signal Tap and reprogramming (which takes ~30-40 seconds vs 3-5 minutes converting to POF and programming that way). When the engineers are not programming or debugging, the systems are in a pool for automated regression testing. When they need a system, they can request one from the pool.  We are also ramping up to allow other internal (and possibly external) groups access these systems. Due to the space needed (4U rack space per FPGA based system), we need to be as conservative as possible. Hence the 1U 'JTag Server' in my diagram. Having a dedicated system running JTAG for each FPGA based system is not scalable, space or administratively.

[Altera_Forum]

--- Quote Start ---  First, let me state that I am a systems administrator and lab manager, not a hardware engineer.   --- Quote End ---   Great! Its good to know your perspective on the problem.    --- Quote Start ---   Our module has a 10 pin header for the USB Blaster to plug into for programming and probing the FPGA. We currently have 24 systems with modules, and due to the configuration the module needs to be programmed while the system bios is in a loop waiting for discovery. The systems currently are not self-programming, thus the need for Quartus on a 'Host' system. We have a team of engineers doing various debug and optimizations of our base RTL, thus the reason for Signal Tap and reprogramming (which takes ~30-40 seconds vs 3-5 minutes converting to POF and programming that way). When the engineers are not programming or debugging, the systems are in a pool for automated regression testing. When they need a system, they can request one from the pool.  We are also ramping up to allow other internal (and possibly external) groups access these systems. Due to the space needed (4U rack space per FPGA based system), we need to be as conservative as possible. Hence the 1U 'JTag Server' in my diagram. Having a dedicated system running JTAG for each FPGA based system is not scalable, space or administratively.  --- Quote End ---    Your comments regarding the BIOS imply that these boards are also PCIe boards. Is that correct? Are these custom boards, or commercially available development boards? If so, what are they? I could take a look at the board design and see what other options might work for you.  Your type of system management issue is something that needs to be solved at the hardware design level, so ultimately you might be limited by the boards your engineers have selected.  I have a system with a similar issue to yours; see p16 + p18  https://www.ovro.caltech.edu/~dwh/correlator/pdf/hawkins_jpl_2014.pdf  The 'data processing' FPGAs on these boards are dynamically reconfigured by the on-board PowerPC processor. The PowerPC is the PCI end-point and is always powered, so that the BIOS on the x86 cPCI host CPU does not see the PCI end-point 'disappear', which is what would happen if the FPGAs were the PCI end-point. In your case you sound like you're working with PCIe, but the concept is the same.  If you are working with PCIe end-points implemented as FPGAs, and those FPGAs support CvP (configuration via PCIe), then it should be possible to have the PCIe end-points configure at power-on with an appropriate set of PCIe end-point registers that will allow the FPGA core to be configured and re-configured at will. Those end-points would not require JTAG for configuration, and configuration would take a second or so.   You could have a pool of machines without JTAG connections, and another pool with JTAG connections. The JTAG pool would be used by engineers that need to use SignalTap II, while the non-JTAG pool could be used by the FPGA interface application developers, i.e., the developers working with already-debugged FPGA designs.  If these boards are custom boards, and are still being designed, then here is an option I have considered; Compact PCI-Serial (CPCI-S.0), uses a backplane that includes PCIe, Ethernet, and USB. A Compact PCI-Serial host contains a USB hub that can connect to every board in a chassis. By including a USB-Blaster circuit on every peripheral board, the CPCI-S.0 host can see every USB-Blaster in a chassis. If that host runs an Altera JTAG server, then all connections in a chassis will be visible to the outside world. This "solution" is better than having numerous USB-Blaster cables, since there are no cables to manage, they are inherently part of the chassis design. If your system is not Compact PCI-Serial, but uses a custom chassis design, then this same concept could be applied, but would depend on the chassis connectivity. If you can provide more details, I can provide more ideas :)  Cheers, Dave

[Altera_Forum]

Our modules are purely custom. Pactron is assembling them. Here's a rough overview: http://www.altera.com/end-markets/computer-storage/qpi/quick-path-interconnect.html 

 These essentially sit in the second socket of a 2 CPU Xeon server, and communicate via the QPI bus protocol (single link). As this link is established very early in the bios during power-on, we have to make custom modifications to the bios to wait for the FPGA to program and establish this link. Future designs will have the bios program the FPGA through the CPU without the need for jtag, but this is what we are currently deploying for development. We have also been shipping these to customers in small volumes, so I am sure they would also be interested in this solution. Redesign of the module is NOT an option.

[Altera_Forum]

These modules are entirely custom and designed to sit in the second cpu socket of an Intel Xeon DP server. We already have built and deployed a few hundred now. Redesign is not an option. Future platforms will have a better program flow through the processor directly and will be programmed by the bios at power on (adds ~20-30 seconds to the boot time). When we get to production stage of that design, we will no longer need jtag for everything, although we will likely need it for internal development/debug. 

 My theory for 1 server, multiple targets was that it reduces the overhead per rack cabinet. Our current production system is based on a 4U server, and the next generation will allow for 1U or 2U server (same motherboard for 1U/2U). Each server already uses 2 IP addresses (1 OS, 1 Remote Management). Having a third for each system (jtag server) is not easy without moving internally to a new subnet. Networking aside, having 1 additional host per target is also a waste of space and resources. It takes far too much time to allocate systems for development/debug/validation. Also, it is not very efficient for a developer to have to query the lab staff to see what is available, and not efficient for lab staff to query developers to see if they are still using a system when there is a shortage.  Except for the multi-threaded connection issue, my design is fairly optimal and easy to automate for allocation, and allows for scaling. I even have a full web front end for managing and allocating systems in the pool, complete with time expiration alerts and auto-logoff (for the developers that forget they are on a system and go on vacation).

[Altera_Forum]

--- Quote Start ---  These modules are entirely custom and designed to sit in the second cpu socket of an Intel Xeon DP server. We already have built and deployed a few hundred now. Redesign is not an option. Future platforms will have a better program flow through the processor directly and will be programmed by the bios at power on (adds ~20-30 seconds to the boot time). When we get to production stage of that design, we will no longer need jtag for everything, although we will likely need it for internal development/debug.   --- Quote End ---   Ok, thanks for the clarification.    --- Quote Start ---   My theory for 1 server, multiple targets was that it reduces the overhead per rack cabinet. Our current production system is based on a 4U server, and the next generation will allow for 1U or 2U server (same motherboard for 1U/2U). Each server already uses 2 IP addresses (1 OS, 1 Remote Management). Having a third for each system (jtag server) is not easy without moving internally to a new subnet.    --- Quote End ---   One server for multiple FPGA targets via multiple USB-Blaster connections seems reasonable. The one caveat is that USB-Blaster's ship with the same USB serial string, so you typically need to reprogram their USB strings via FTDIs FT_PROG tool. That way your JTAG server will present the same USB-Blaster identification each time the system boots, or when different numbers of USB-Blasters are connected.    --- Quote Start ---   Except for the multi-threaded connection issue, my design is fairly optimal and easy to automate for allocation, and allows for scaling. I even have a full web front end for managing and allocating systems in the pool, complete with time expiration alerts and auto-logoff (for the developers that forget they are on a system and go on vacation).  --- Quote End ---   I agree that multiple USB-Blaster connections should be supported via a single jtagd server.  I have several USB-Blasters, and could likely reproduce your problem, but I doubt that will help you :)  This forum is maintained by Altera users, not by Altera (although several Altera employees do answer questions). Have you filed a Service Request with Altera directly? Another useful person to contact is your local Altera FAE (Field Applications Engineer), eg., at Arrow Electronics or whoever you buy your Altera devices from. Let them know which operating system you are using and which version of Quartus.  Cheers, Dave

[Altera_Forum]

Yes, I have a support request in (and also a back-channel email filtering through). I was hoping maybe someone else ran into this and had their own creative workaround. I may post one of my own if Altera doesn't fix this directly. 

 As to the unique serial number issue, we renamed each module to match the target name. This way, jtagconfig reports all of the remote systems as "##) Blaster: <target name> ...". The only caveat with this is if I need to move that system to a direct Windows host, I have to either rename the usb-blaster or swap it for a stock device (Windows driver doesn't like the device to be renamed locally - works fine for remote). I wouldn't have even gone down this path if this part didn't work. Imagine running jtagconfig (or the Quartus gui) and seeing 20+ systems, all named USB Blaster. Worse than playing a shell game on a street corner.

[Altera_Forum]

--- Quote Start ---  Yes, I have a support request in (and also a back-channel email filtering through). I was hoping maybe someone else ran into this and had their own creative workaround. I may post one of my own if Altera doesn't fix this directly.   --- Quote End ---   Hopefully others will read this thread and post their work-arounds, if any.    --- Quote Start ---   As to the unique serial number issue, we renamed each module to match the target name. This way, jtagconfig reports all of the remote systems as "##) Blaster: <target name> ...". The only caveat with this is if I need to move that system to a direct Windows host, I have to either rename the usb-blaster or swap it for a stock device (Windows driver doesn't like the device to be renamed locally - works fine for remote).   --- Quote End ---   I've renamed the serial strings on several boards and used them fine under Windows. Its not as convenient as Linux, but it did work.    --- Quote Start ---   I wouldn't have even gone down this path if this part didn't work. Imagine running jtagconfig (or the Quartus gui) and seeing 20+ systems, all named USB Blaster. Worse than playing a shell game on a street corner.  --- Quote End ---   Indeed! Too many manufacturers do not appreciate that their USB development boards need to have serial strings you can change!  Good luck finding a solution.  If you need another FAE contact, email me off list (use my forum name), and I'll get you in touch with my Arrow contact.  Cheers, Dave

[Altera_Forum]

--- Quote Start ---   If you need another FAE contact, email me off list (use my forum name), and I'll get you in touch with my Arrow contact.   --- Quote End ---    Thanks. This I already have, direct (not through a distributor). So far their email response has been "Nope, that won't work." But as I said, it is filtering through back channels.

[Altera_Forum]

--- Quote Start ---  This I already have, direct (not through a distributor). So far their email response has been "Nope, that won't work." But as I said, it is filtering through back channels.  --- Quote End ---   Well, if that comes up with nothing, feel free to email me, and I'll email him. Different contacts may come up with different responses :)  Cheers, Dave