chessboard / marker detection via d415 infrared camera

[drmatt]

Hello, we are using a realsense d415 sensor and have an application that requires robust detection and precise localization of fiducial markers while we collect depth data. As recommended in the https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/BKMs_Tuning_RealSense_D4xx_Cam.pdf whitepaper on tuning, we'd like to use the left infrared camera for this because it's perfectly synchronized and pixel aligned with the depth maps. Of course, the IR images contain strong speckle patterns from the emitter, which makes detection of chessboards or other high-contrast markers virtually impossible with off-the-shelf methods.

The whitepaper, as well as various other discussions like /thread/130191 this recent one, mention that there is a special color correction table designed to suppress the IR projector pattern. This seems perfect for our application, but we may be using it incorrectly or may have unrealistic expectations of its performance. When we set the table programmatically or via the "Remove Ir Pattern" preset in the realsense viewer, and look at the rgb8 stream from the left IR camera, the main difference for black-and-white markers is that the image is quite a bit darker even with decent ambient lighting, and the markers are still undetectable. Below is an example from the left IR camera with Default preset settings (left) and Remove Ir Pattern preset (right).

For comparison, here is an image from the infrared camera with the emitter turned off (left) and from the rgb camera (right) in the same conditions:

So, our questions: is this the expected behavior? How is the color table intended to be used? Is there a way to achieve reliable detection of high-contrast features such as these in the infrared with the emitter turned on?

Thanks very much for any insight.

[MartyG · Accepted solution]

My own tests confirmed that an image becomes darker when the IR Emitter is turned off.

If you want to avoid using an IR pattern altogether so you can better detect markers then you could turn-off auto-exposure and set Gain manually to increase the brightness of the image.

You can also reduce the visibility of the IR pattern projected by the camera by reducing the Laser Power setting instead of turning the IR emitter off completely.

[drmatt]

MartyG thank you for your response.

For us, turning off the IR emitter actually produced clearer / brighter images of the markers, probably because of the auto-exposure, as we hoped to show in the attached example images. In any case, we need the emitter turned on in order to obtain the best possible depth maps for our application.

We tried your suggestion of turning down the laser power, but it seems that the power setting has almost no effect on the overall brightness or clarity of the color-corrected image - again, probably because of the auto-exposure. Unfortunately, setting the gain and exposure manually isn't likely to work well over a range of environment conditions, so we'd prefer to keep the auto settings.

[MartyG]

My tests indicate that light is a factor in the darkness of the image. When using the Remove IR Pattern preset, the image was dark in bright-light conditions, but brighter in darker areas. For example, the image was dark with the IR pattern removed when I pointed the camera at the top of my computer desk. But it was brighter when the camera was positioned down below the top of the desk, where there was less ambient light.

Above desk

Below desk

[drmatt]

MartyG strangely that's not what we observe, though our lighting conditions may be different. Above the desk and below the desk are equally dark - almost black - unless there's something bright in the image, like a monitor, in which case the image becomes a bit lighter and clearer. But generally, we can't get usable IR images of the high-contrast markers with the emitter on.

The images you posted actually look pretty nice - was the emitter turned on, and are you showing the rgb8 left-infrared images? If so, maybe our sensor is defective somehow?

[MartyG]

In my location setup, the desk is in the corner of the room and the main 70 W ceiling lightbulb source is to the side and in front of the desk (i.e not shining directly down on the desk)

My settings were:

- D415 camera

- SDK version 2.16.0

- Firmware 5.9.14

- Resolution: 1280 x 720

- FPS:: 30

- Visual Preset 'Remove IR Pattern'

- IR emitter off

- Auto-exposure on

- Depth off

- IR 1 channel = RGB8

- IR 2 channel off

I repeated the test just now with the above conditions. Both above and under the desk, the image was bright when the IR emitter was off, and went very dark as soon as the IR emitter was enabled. The image was much brighter under the desk than above it when the IR emitter is off.

The auto-exposure can be observed changing the brightness in real-time as the camera is moved. Also, rotating the camera from pointing forward to the underneath of the desk to pointing 90 degrees up to the ceiling causes the image to dim to the same brightness as when it is above desk, validating that the amount of light going into the IR imager is affecting how bright the IR image is rendered by the auto-exposure.

[drmatt]

Thanks, good idea - our settings:

- D415 camera

- Indoor lighting

- SDK version 2.16.3

- Firmware version 5.10.06

- Resolution 1280x720 @30 fps

- Depth and IR 2 off

- IR 1 set to RGB8

- Auto-exposure on

- Emitter on

- Remove IR Pattern preset

I think we're seeing similar behaviors here. So the questions remain - are we using the Remove IR Pattern preset correctly? Is there any way to detect high-contrast targets while keeping auto-exposure and emitter both on? Perhaps someone from Intel's Realsense team can shed some light.

[MartyG]

The main difference between my test settings and the ones that you listed above is that I had the IR emitter turned off, whilst you had it turned on.

A RealSense user once used an LED wall right behind the sensors, as they said that it provided uniform white light with almost no shadows.

[idata]

Hi Matthew,

I apologize for the delay in getting back to you. I tested these settings in the lab and got similar results to yours as well. With Emitter and Auto-Exposure enabled, and the IR pattern removed, you will get darker images of the high-contrast markers. The best way to get usable images of your high-contrast markers with the IR pattern removed and Emitter and Auto-Exposure enabled, is to adjust Laser Power. I found that turning laser power all the way down to value between 0 and ~25 produced brighter, more clear images of the high-contrast target.After you adjust this setting in the RealSense Viewer, you can export your presets as a JSON file. Please let me know if this information was helpful to you.Regards,Sahira

[drmatt]

Sahira, thank you for the recommendation, that did seem to help a bit. Below are three images, all with the Remove IR Pattern preset. The top image is the left IR with the emitter at default laser power of 150 mw; the middle image is with the emitter completely off; and the bottom image is with the laser power set to 30 mw. Note: it seems that the power can only be set in steps of 30 mw in the viewer, so we can't go lower.

Although the images are noisy, and there are still faint traces of the emitter pattern visible, it now seems more feasible to detect features with these settings. Also, the indoor lighting in our environment isn't very bright, so the noise should improve with brighter light. There may be somewhat of a balancing act between the amount of ambient light and the laser power setting; more ambient light will produce cleaner images but may "drown out" the projected pattern, with a possible negative effect on disparity computation.

[MartyG]

The Laser Power setting in the Viewer tends to jump in multiples of 30 if you left-click on it once. If you hold down the left mouse button on the slider and move the mouse gently though, you can move in increments of around 1.5 (0, 1,5, 3, 4.5 etc).

Yes, the 400 Series cameras get better in stronger light.

[drmatt]

MartyG the UI allows the slider to move to any value via left-drag, but when the mouse button is released, the value "snaps" to the nearest multiple of 30, at least in our version.

In any case, I think we've found a workable solution for now. I marked your original answer as correct, because you included a suggestion to turn down the laser power as a possible fix. At the time it had no effect for us, but that may have been due to low ambient light. Thanks for your help!

[MartyG]

That's odd, the slider does not snap back on my setting (Windows 10, SDK 2.16.0).

You're very welcome. Please feel free to return to this forum with further questions any time you need to.