How to properly setup external triggering on D415 and D435?

[HBadi]

Hi,

I'm trying to externally trigger a single D435. I was able to setup most of the recommendations of in this document:

https://realsense.intel.com/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper_rev1.1.pdf https://realsense.intel.com/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper_rev1.1.pdf

but my external trigger doesn't seem to be having any effect whatsoever in the capture. My trigger frequency accuracy is very accurate and setup to trigger at 30 Hz. In my code, I call set_option to set the sensor as slave:

auto sensors = device.query_sensors();

sensors[0].set_option(RS2_OPTION_INTER_CAM_SYNC_MODE, 2); // 0: default, 1: master, 2: slave

I have also updated the firmware (5.10.3) to the latest available version. (The set_option function call for the sync mode parameter was failing with the default firmware version.)

I have also tried to set the register directly using the terminal binary of the SDK, but still no luck. I also tried a D415 and the same happens, no luck with the external trigger.

Is this feature implemented at all in the latest firmware of these sensors? One very suspicious behavior is that when I set the cam sync mode to slave, the sensor runs in free-run mode even when not triggered.

I would greatly appreciate any help on this matter.

Thanks

[MartyG · Accepted solution]

A camera that is set to slave can operate independently even when connected to a master with sync cabling. It listens for a sync pulse and if it does not detect one within a certain time period, it triggers itself as though it were an unsynced camera.

In a RealSense webinar last week about multiple camera sync hosted by the CTO of RealSense, it was said that using an external sync source requires very fine frequency resolution. Instead, it is recommended to use one camera as the master. MIPI may work if V-sync is closely matched with the RealSense camera, though the sync frequency window is very small. It is therefore much easier if only RealSense cameras are used in hardware syncing.

Here's a quote from the session:

"If you are able to get a sync pulse from the MIPI interface camera, and its frame rate is sufficiently close to the expected frame rate for the RealSense camera, then you should be able to .. you can read the white paper for the requirements on how closely you have to match the frame rate of the sensor. It's a limitation of the sensor ... primarily a limitation of the rolling shutter, and we may look at doing something else for the global shutter solution ... right now there is a limitation of those sensors that they can only have a very small window in which they will allow you to trigger externally"..

The multiple-camera white paper says that the frequency of the signal generator needs to match the sensor frequency exactly, and that 30 FPS may have a real frame rate of 30.015 FPS. An oscilloscope may be needed to measure the real frame rate and configure the signal generator to the same frequency, the paper suggests.

At an Intel demo in January with four hardware-synched D435s, they used a commercial-grade flash that caused a spike in the data that acted as a means of syncing.

https://realsense.intel.com/intel-realsense-volumetric-capture/ Volumetric Capture @ Sundance using Intel RealSense Depth Cameras

[HBadi]

Marty,

thanks for your insights. I'm able to trigger the device now with a signal generator. The triggering works for a variety of frequency ranges, I have tried from 29.5 to 30.5 without any problem, so the requirement of being as close as possible to 30.015 doesn't seem to apply here. Now, I'm going to move to trigger the camera from the actual signal I want to use for this project. I'll report my findings.

[MartyG]

Awesome news, thanks for the report!

[MZimm4]

Can you give details about how you set this up?

I am trying to externaly trigger a D435 and it often happes that multiple externals pulses are ignored before the next frame is triggered.

I set intercam_sync_mode to 2 (slave) and enable output_trigger to false.

[HBadi]

I'm not sure how many external triggers are being ignored before it actually starts streaming data. I guess that if you don't start triggering soon enough, the device will detect this and automatically switch to free-run mode.

Is your problem that you need the device to start triggering immediately with your first trigger or that you are not able to trigger at all? A way to verify if it is triggering is to change the frequency of the trigger and see if the delta time between consecutive images changes accordingly, thats the way we have done it.

[HBadi]

MartyG The device seems to be triggering correctly even with our custom trigger device, so problem solved. We are not sure why it was not working before.

[MartyG]

Awesome news, thanks so much for letting us know!

[MZimm4]

Hernan.Badino

The trigger is active before I initiate the infrared/depth stream.

I tried to use a slightly out off spec frame rate like 14.8. When I use realsense-viewer to monitor the current FPS the frames do not arrive reliably and the device frequently sends "Right MIPI error"

Only when I get close to the specified and set frame rate, say 15fps, I start to get a continuous image stream.

One more thing: When I externally trigger the d435, what event does an incoming trigger actually indicate? Is it the beginning of exposure or slightly before? I cannot find any information on the exact time relation of trigger pulse and exposure in the intel doc.

Specifically this time relation will very likely differ for external triggering or when I use a d435's as master and take the output trigger to get the exposure timing.

Maybe you can help me out ?

[MartyG]

A hardware sync trigger tells the cameras to align their frame timestamps.

The CTO (Chief Technical Officer) of RealSense has said that "When you set a camera to slave it will still acquire frames at the right frame rate even if there is no sync input signal. It tries to sync but if there is no signal that comes in at the right time, it gives up and triggers itself".

The CTO added, "You can make a wireless version [of the sync cabling] yourself if you have a longer range as it is simply a sync pulse that needs to be shared. However, you will need to properly collect frames together using the frame counters (which will be offset for each camera but in sync)".

[MZimm4]

Thank you for the info!

I can see how this feature is used to sync mulitple cameras.

The thing that I am wondering about is more how to use sync realsense with other peripherals such as IMUs or other sensors (e.g. how to avoid using the realsenses clock but use the clock of one agent that triggers all sensors)

To do this one would need to know how the trigger signal is aligned to exposure events (on master and slave agents respectively), not to some metadata frame timestamp.

[MartyG]

Intel are considering releasing a new version of the D435 currently codenamed D435i, which would be identical to D435 except for the addition of an IMU. If it is released (not all RealSense prototypes created get a release), it is hoped that it will be available by the end of the year.

If you need to do the trigger externally though ... in January 2018 at an event at the Sundance festival, Intel signaled four hardware-cable linked D435s to start and stop capture using a commercial-grade flash. The spike in the data caused by the flash acted as the trigger.

[MZimm4]

Intel are considering releasing a new version of the D435 currently codenamed D435i, which would be identical to D435 except for the addition of an IMU. If it is released (not all RealSense prototypes created get a release), it is hoped that it will be available by the end of the year.

So for the time being we will try to use an external IMU

If you need to do the trigger externally though ... in January 2018 at an event at the Sundance festival, Intel signaled four hardware-cable linked D435s to start and stop capture using a commercial-grade flash. The spike in the data caused by the flash acted as the trigger.

Yes I understand how to generate an external trigger signal.

So the question is not if it is possible but rather: How many nano- or microseconds pass between the falling flank of an external trigger pulse and the beginning of the exposure time?

Until now I could not find this info neither in the whitepaper nor in any of the posts here or on git

[MartyG]

In the recent Intel webinar on multiple cameras, sync was referred to in terms of micro-second precision of alignment rather than nano-second. To get an answer to your question, it would be best to ask it on the RealSense GitHub, where the RealSense developers and engineers reside. you can do so by going to the link below and clicking the New Issue button.

https://github.com/IntelRealSense/librealsense/issues Issues · IntelRealSense/librealsense · GitHub

[HBadi]

Why are you triggering at 14.8. I was able to trigger at 30 fps.

[MZimm4]

Why are you triggering at 14.8. I was able to trigger at 30 fps.

Because I set the Realsense to 15FPS. The sensors supports 4FPS, 6FPS, 15FPS and 30FPS

Same observation if I set it to 30FPS and set the external trigger to the same rate

[idata]

Hello M0oz,

Thank you for your interest in the Intel RealSense Technology. We wanted to step in and inform you in regards to your query on "how the trigger signal is aligned to exposure events" , that the output sync pulse can be recognized as the VSYNC pulse of the depth sensor. It outputs 7 lines after the end of exposure. The latest development firmware, v 5.10.3 with support for multi-camera sync is now available for download here (as of August 20, 2018)https://downloadcenter.intel.com/download/28076/Latest-Firmware-for-Intel-RealSense-D400-Product-Family%3Fv%3Dt https://downloadcenter.intel.com/download/28076/Latest-Firmware-for-Intel-RealSense-D400-Product-Family?v=tI hope this information proves useful, please let me know if there is any further information you require. Best regards, Casandra