Random results on FPGA

[Altera_Forum]

Hello, 

 I'm using altera devices for about 10 years with quartus or quartus + synplify pro. I'm currently struggling for about a month !!! on this issue: Using the same netlist from synplify pro, the same timing constraints, and doing the place and route with quartus, I can get total different results on FPGA (stratix III): fully working, or failing on first HW accesses. The IP in the FPGA uses a lot of clock gating and I think that Quartus is not doing a good timing analysis (timequest reports no timing violations and the design is fully constrained), leading to such random results.  Did you ever face such problem at your place ?  Metastability report in timequest reports no problem.  Resets are ok.  Cooling the FPGA does not change anything.  I never faced such situation before.   I need help !  Alex

[Altera_Forum]

--- Quote Start ---  Hello,  I'm using altera devices for about 10 years with quartus or quartus + synplify pro. I'm currently struggling for about a month !!! on this issue: Using the same netlist from synplify pro, the same timing constraints, and doing the place and route with quartus, I can get total different results on FPGA (stratix III): fully working, or failing on first HW accesses. The IP in the FPGA uses a lot of clock gating and I think that Quartus is not doing a good timing analysis (timequest reports no timing violations and the design is fully constrained), leading to such random results.  Did you ever face such problem at your place ?  Metastability report in timequest reports no problem.  Resets are ok.  Cooling the FPGA does not change anything.  I never faced such situation before.   I need help !  Alex  --- Quote End ---    Is it random per build or per download or per reset? If it is per build then what do you change to rebuild? If it is per download(of same one build) then it points to power up reset issues. If it is per reset(one download) then it is due to your reset. Try changing its duration. do you have multicycle deconstraint? if so double check them.  Also check your clock signal and voltage levels.

[Altera_Forum]

It is random per place and route. 

When it fails, it always fails at the same point after several resets or re-sof. I change nothing between build. no multi cycles. Few false path but for debug ports. Voltage is ok (1.1V). Clocks are ok.

[Altera_Forum]

--- Quote Start ---  It is random per place and route. When it fails, it always fails at the same point after several resets or re-sof. I change nothing between build. no multi cycles. Few false path but for debug ports. Voltage is ok (1.1V). Clocks are ok.  --- Quote End ---    If you change nothing between builds then I assume the fitting results should be more or less identical. Therefore (my guess) it is likely your problem is to do with reset functionality per se which appears at power up or afterwards. I believe the place and route variation is just coincidence of same issue.   I mean some registers are not under reset. You need to check that your control signals are all under reset. You can also uncheck powerup don't care in the fitter settings so that all registers power up low(except preset ones). Finally you may add signaltap to capture the events at failure. You may find a work around if you play with the duration and release of reset until you find a stable window.  also check for any floating inputs which are not driven at your development stage

[Altera_Forum]

This is not a reset issue, it is properly driven, synchronized and with a long duration. 

power up don't care is already unchecked. There is no floating inputs.

[Altera_Forum]

I think the report is too vague to seriously guess about the problem cause. 

 I'm surprised about the "lot of clock gating" point. By nature of the clock networks, FPGAs are not designed to implement clock gating effectively, with the exception of those cases where clock control blocks or PLL blocks hold in reset can be utilized. Timing analysis should still work, but we don't know much about the design, see my first point.

[Altera_Forum]

Well, indeed, this is bad to use clock gating in FGPA. 

The problem is that they use the clock gating feature to reduce the clock speed (so the duty cycle is far from 50%). Before I was removing the gating succh that all mappings were stable and consistent, but without the clock divison feature. Thanks for your feedback anyway.

[Altera_Forum]

Mmmm I just found something interesting, 

the same sof does actually give different results depending on another parameter (an external uart connection), I will investigate that.  One of the uart (another one) is not connected and so has floating pins. I Will fix that.

[Altera_Forum]

How about if you heat the chip up in an oven or use freezer spray on it? Does that make problems appear and/or go away? 

Is the clock gating/clock divider done by generating logic clocks (like using the MSB of a counter)?

[Altera_Forum]

spray does not chane anything. 

 I just found that programming another FPGA that is on the board (dual stratix 3), that makes it to work or not. I'm looking into it.

[Altera_Forum]

There was a functional issue, that was triggered depending of the other FPGA programing, and so that was confusing me.

[Altera_Forum]

It is working fine using 91 clocks and using those settings: 

 set_global_assignment -name FAMILY "Stratix III" set_global_assignment -name DEVICE EP3SL340F1760C3 set_global_assignment -name TOP_LEVEL_ENTITY wlc_ip1302_gbb5_fpga_a set_global_assignment -name ORIGINAL_QUARTUS_VERSION 12.1 set_global_assignment -name PROJECT_CREATION_TIME_DATE "10:12:01 DECEMBER 11, 2012" set_global_assignment -name LAST_QUARTUS_VERSION "12.1 SP1" set_global_assignment -name PROJECT_OUTPUT_DIRECTORY output_files set_global_assignment -name MIN_CORE_JUNCTION_TEMP 0 set_global_assignment -name MAX_CORE_JUNCTION_TEMP 85 set_global_assignment -name DEVICE_FILTER_PACKAGE FBGA set_global_assignment -name DEVICE_FILTER_PIN_COUNT 1760 set_global_assignment -name DEVICE_FILTER_SPEED_GRADE 3 set_global_assignment -name ERROR_CHECK_FREQUENCY_DIVISOR 256 set_global_assignment -name ENABLE_INIT_DONE_OUTPUT ON set_global_assignment -name STRATIX_JTAG_USER_CODE 20130424 set_global_assignment -name USE_CONFIGURATION_DEVICE OFF set_global_assignment -name CRC_ERROR_OPEN_DRAIN OFF set_global_assignment -name RESERVE_ALL_UNUSED_PINS_WEAK_PULLUP "AS INPUT TRI-STATED WITH WEAK PULL-UP" set_global_assignment -name STRATIX_DEVICE_IO_STANDARD "3.3-V LVTTL" set_global_assignment -name OUTPUT_IO_TIMING_NEAR_END_VMEAS "HALF VCCIO" -rise set_global_assignment -name OUTPUT_IO_TIMING_NEAR_END_VMEAS "HALF VCCIO" -fall set_global_assignment -name OUTPUT_IO_TIMING_FAR_END_VMEAS "HALF SIGNAL SWING" -rise set_global_assignment -name OUTPUT_IO_TIMING_FAR_END_VMEAS "HALF SIGNAL SWING" -fall   set_global_assignment -name FITTER_EFFORT "STANDARD FIT" set_global_assignment -name FLOW_DISABLE_ASSEMBLER OFF set_global_assignment -name FLOW_ENABLE_IO_ASSIGNMENT_ANALYSIS OFF set_global_assignment -name STRATIXII_OPTIMIZATION_TECHNIQUE SPEED set_global_assignment -name TIMEQUEST_MULTICORNER_ANALYSIS ON set_global_assignment -name SMART_RECOMPILE ON set_global_assignment -name ENABLE_BENEFICIAL_SKEW_OPTIMIZATION ON  set_global_assignment -name VERILOG_INPUT_VERSION VERILOG_2001 set_global_assignment -name VERILOG_SHOW_LMF_MAPPING_MESSAGES OFF  set_global_assignment -name NUM_PARALLEL_PROCESSORS 2 set_global_assignment -name SYNTH_TIMING_DRIVEN_SYNTHESIS OFF set_global_assignment -name OPTIMIZE_POWER_DURING_SYNTHESIS OFF set_global_assignment -name OPTIMIZE_HOLD_TIMING "ALL PATHS" set_global_assignment -name OPTIMIZE_MULTI_CORNER_TIMING ON set_global_assignment -name OPTIMIZE_POWER_DURING_FITTING OFF set_global_assignment -name SYNTH_GATED_CLOCK_CONVERSION ON set_global_assignment -name PLACEMENT_EFFORT_MULTIPLIER 1 set_global_assignment -name ROUTER_EFFORT_MULTIPLIER 1 set_instance_assignment -name SLEW_RATE 3 -to *  set_global_assignment -name STATE_MACHINE_PROCESSING "ONE-HOT" set_global_assignment -name PHYSICAL_SYNTHESIS_COMBO_LOGIC OFF set_global_assignment -name PHYSICAL_SYNTHESIS_REGISTER_DUPLICATION OFF set_global_assignment -name PHYSICAL_SYNTHESIS_REGISTER_RETIMING OFF set_global_assignment -name ROUTER_LCELL_INSERTION_AND_LOGIC_DUPLICATION ON set_global_assignment -name ROUTER_TIMING_OPTIMIZATION_LEVEL MAXIMUM set_global_assignment -name PRE_MAPPING_RESYNTHESIS OFF set_global_assignment -name AUTO_PACKED_REGISTERS_STRATIXII NORMAL set_global_assignment -name ADV_NETLIST_OPT_SYNTH_WYSIWYG_REMAP OFF set_global_assignment -name MUX_RESTRUCTURE OFF set_global_assignment -name ROUTER_CLOCKING_TOPOLOGY_ANALYSIS ON set_global_assignment -name SYNCHRONIZATION_REGISTER_CHAIN_LENGTH 3 set_global_assignment -name AUTO_MERGE_PLLS OFF set_global_assignment -name POWER_USE_DEVICE_CHARACTERISTICS MAXIMUM set_global_assignment -name POWER_PRESET_COOLING_SOLUTION "NO HEAT SINK WITH STILL AIR" set_global_assignment -name POWER_BOARD_THERMAL_MODEL "NONE (CONSERVATIVE)" set_global_assignment -name FITTER_AGGRESSIVE_ROUTABILITY_OPTIMIZATION AUTOMATICALLY  set_global_assignment -name SEED 2  set_global_assignment -name PHYSICAL_SYNTHESIS_EFFORT FAST set_global_assignment -name TIMEQUEST_DO_REPORT_TIMING ON set_global_assignment -name TIMEQUEST_DO_CCPP_REMOVAL ON  set_global_assignment -name FINAL_PLACEMENT_OPTIMIZATION AUTOMATICALLY set_global_assignment -name ROUTER_REGISTER_DUPLICATION OFF set_global_assignment -name ENABLE_DRC_SETTINGS OFF  set_global_assignment -name AUTO_GLOBAL_CLOCK ON  set_global_assignment -name ALLOW_POWER_UP_DONT_CARE OFF set_global_assignment -name EXTRACT_VERILOG_STATE_MACHINES OFF set_global_assignment -name EXTRACT_VHDL_STATE_MACHINES OFF set_global_assignment -name INFER_RAMS_FROM_RAW_LOGIC OFF set_global_assignment -name DSP_BLOCK_BALANCING OFF set_global_assignment -name REMOVE_DUPLICATE_REGISTERS OFF set_global_assignment -name AUTO_CARRY_CHAINS OFF set_global_assignment -name AUTO_ROM_RECOGNITION OFF set_global_assignment -name AUTO_RAM_RECOGNITION OFF set_global_assignment -name AUTO_DSP_RECOGNITION ON set_global_assignment -name AUTO_SHIFT_REGISTER_RECOGNITION OFF set_global_assignment -name ALLOW_SHIFT_REGISTER_MERGING_ACROSS_HIERARCHIES OFF set_global_assignment -name AUTO_CLOCK_ENABLE_RECOGNITION ON set_global_assignment -name STRICT_RAM_RECOGNITION ON set_global_assignment -name ALLOW_SYNCH_CTRL_USAGE OFF set_global_assignment -name AUTO_RAM_BLOCK_BALANCING OFF set_global_assignment -name BLOCK_DESIGN_NAMING QUARTUSII set_global_assignment -name SYNTHESIS_EFFORT AUTO set_global_assignment -name SHIFT_REGISTER_RECOGNITION_ACLR_SIGNAL OFF set_global_assignment -name DISABLE_REGISTER_MERGING_ACROSS_HIERARCHIES OFF set_global_assignment -name BLOCK_RAM_TO_MLAB_CELL_CONVERSION OFF set_global_assignment -name OPTIMIZE_IOC_REGISTER_PLACEMENT_FOR_TIMING OFF set_global_assignment -name OPTIMIZE_FOR_METASTABILITY ON  set_global_assignment -name IGNORE_LCELL_BUFFERS ON set_global_assignment -name SYNTH_PROTECT_SDC_CONSTRAINT ON  set_global_assignment -name SYNCHRONIZER_IDENTIFICATION OFF

[Altera_Forum]

It looks like I can created the un-stability by propagating a variable clock (programmable pll)in the design (which is not selected by software). 

Could it be that a variation in frequency can lead to unstability in the FPGA (I get a warning about the output of the pll not being dedicated)?

[Altera_Forum]

It's something else. 

I think it is a global clock issue. If the processor gets a global clock signal, it runs fine. If not, even if timing report is clean, it fails.

[Altera_Forum]

well, it looks like the hold time analysis is a bit weak in my project, 

I added a overconstraint of 0.2 ns and that seems much better.  In the sdc:  foreach_in_collection all_clocks [get_clocks *] {  set_min_delay -from $all_clocks -to $all_clocks 0.2  }

[Altera_Forum]

--- Quote Start ---  well, it looks like the hold time analysis is a bit weak in my project, I added a overconstraint of 0.2 ns and that seems much better.  In the sdc:  foreach_in_collection all_clocks [get_clocks *] {  set_min_delay -from $all_clocks -to $all_clocks 0.2  }  --- Quote End ---    I found out that the problem was due to reset that was de-asserted internally on a wrong clock domain.

[Altera_Forum]

No, there is still another issue (

[Altera_Forum]

In the end, what is your ti, ing failure now? Setup or hold violation? Can try over constraint the hold design usingbclock uncertainty?

[Altera_Forum]

that"s the problem, there is no timing violations ^^ 

I"m now testing with more clock uncertainties, will see if that helps.

[Altera_Forum]

How is the timing margin?

[Altera_Forum]

Looks like a clock uncertainty margin of 0.5 ns helps. 

Will see over several runs if that's true ^^