Object detection in IR image containing projector pattern

[JWint5]

Hi,

I'm working with an IR depth camera which does not have an extra RGB camera. So the only information I get are the depth information and the image of the IR camera which contains the dot pattern of the projector.

My overall goal is to detect objects using fiducial markers. The detection would be much simpler if i could either

- remove the dot pattern before starting to look for the markers

- have markers that are somehow "immune" to the dot pattern

Since the depth information is permanently needed turning off the projector is not an option.

I have erased some "pattern related" frequencies manually using the G'MIC plugin in GIMP. With "pattern related" frequencies I mean a pattern of straight lines running through the frequency domain at an angle. While that produced some 'somewhat promising' results I am not quite sure how to cancel those lines automated instead of using the paintbrush in gimp.

Any Ideas on how the marker detection could be accomplished?

(I understand that my question is not related to an Intel product but I figured that there might be people around that have experiences with 'IR noised images')

[MartyG]

If you are not using a RealSense camera, then a marker solution applicable to depth cameras in general may be to use the OpenCV software platform's ArUco marker system.

https://docs.opencv.org/3.1.0/d5/dae/tutorial_aruco_detection.html https://docs.opencv.org/3.1.0/d5/dae/tutorial_aruco_detection.html

Regarding visibility ... an area of an image is more likely to not appear on the image if it is black or grey. This is a general physics principle applicable to depth cameras, as the dark color absorbs the light of the laser. This area of the object would typically appear on the image as a fully black region due to the lack of detail available for that part of the image. For example, if a black cable is analyzed then it may appear as a cable-shaped black line.

It would be much easier if you were using a RealSense 400 Series camera for your project, as you can alter its Laser Power setting with scripting to increase or decrease the visibility of the projected dot pattern.

[JWint5]

Thank you for quick response

"It would be much easier if you were using a RealSense 400 Series camera..."

That's true, a different Camera would make things easier. But unfortunately that is not an option for me :/

"...may be to use the OpenCV software platform's ArUco marker system."

Somehow I was convinced that the dotted pattern would destroy the clear edges and corners of the marker. But I will give it a try, and maybe the detection is very robust. Thanks

In my setup the marker will be quiet far away from the camera though (4m). I am afraid that the Aruco Marker would have to be quite big to be detectable within the dot pattern.

That's why I figured it might be nice to erase the pattern in frequency domain. Any experience with that?

I googled "remove IR pattern" and ended up at a gitHub issue of the IntelRealSense library.

https://github.com/IntelRealSense/librealsense/issues/1827 Remove IR pattern preset does not work in D435 with latest firmware · Issue # 1827 · IntelRealSense/librealsense · GitHub

It seemed like there was at some point a button called "remove IR pattern" in the IntelRealSense Viewer. That's why I thought may the community has experience with removing such a pattern from an image.

[MartyG]

Yes, the D415 model of RealSense camera supports a 'visual preset' mode called "Left Imager Color w/o IR Pattern". It looks as though the settings in the configuration file are specific to Librealsense though. The code performs 'color correction'. Googling for 'opencv color correction' may give some hints about how to achieve something similar with another type of depth camera using OpenCV.

In the discussion link that you posted, when someone asked whether it was possible to remove the IR pattern with hardware or software, the CTO of RealSense replied: "We would love to have a cheap projector that can switch between a pattern and a uniform illumination, but that does not currently exist. So the one we have is a fixed pattern. You can only turn it on and off and change the power".

[JWint5]

Alright, thanks for the investigation.

The code performs 'color correction'.

That seems to be were the magic happens. Can you point me to source file where this happens?

I cloned the librealsens repo but my search was not successful. Maybe i can learn how they did it and adapt it to my project.

Googling 'opencv color correction' leaves me with rather basic operations like contrast and brightness adjustments.

[MartyG]

Intel's camera tuning guide provides some info and values for color correction.

"One very nice patented feature of the D415, is that the Depth D4 VPU has the ability to do a rudimentary removal of the IR projection pattern, at no additional CPU or power cost. This only works for the D415 at 1280x720 resolution. It is based on changing the color correction matrix in the Advanced Mode API to the following 12 values. Note that the laser has not been turned off so the depth is not impacted at all. However, the green colors are reduced.

ColorCorrection={0.520508, 1.99023, 1.50684, -2, -2, -0.0820312, 1.12305, 1.01367, 1.69824, -2, 0.575195, -0.411133

/* Sets new values for STColorCorrection, returns 0 if success */

void rs2_set_color_correction(rs2_device* dev, const STColorCorrection* group, rs2_error** error);

[JWint5]

Alright, interesting to see. Thank you for digging out the information.

Seems like I will have to find the the correction matrix that erases the pattern of my IR pattern.