D435 FaceTracking with OpenCV not working fine (Unity)

[ESchi3]

Hello guys,

I'm using a D435 to track faces in Unity. With a regular Logitech C920 the tracking works pretty well, but with the RGB from my D435 it's working poorly. Do you guys have any advice on how to improve the quality? I'm using the stream like this:

Config.EnableStream(Stream.Color, 1280, 720, Format.Rgba8, 30);

Also, I want to detect faces with the Infrared image. I'm considering it as a greyscale, so I don't need to convert it like I do with the RGBA images. I only do an equalization of the histogram before running the face detection. But, it's not working pretty well, neither, only recognizing the face sometimes. Maybe it's related to some config of the camera? Below is how I configured the IR stream:

Config.EnableStream(Stream.Infrared, 1280, 720, Format.Y8, 30);

Do you guys have any advice on how to properly config the camera for a good detection?

Thanks.

[MartyG]

When you mentioned that you do a conversion on the RGB stream, do you mean that you convert it from color to greyscale like you aim to do with the IR? If so, you may be able to avoid the conversion by streaming in greyscale directly with the YUYV type of stream instead of RGBA8.

The YUYV stream is demonstrated below in the RealSense Viewer software at 30 FPS.

Likewise, you can get greyscale IR if you use the UYVY channel.

You can find much more advice about adjusting your D435 camera for best results using Intel's excellent illustrated camera tuning guide.

https://realsense.intel.com/wp-content/uploads/sites/63/BKM-For-Tuning-D435-and-D415-Cameras-Webinar_Rev3.pdf?language=en_US

[ESchi3]

Hi MartyG,

Thanks for your reply.

The OpenCV algorithms work on greyscale images. So, for the colored images I need to convert them. For the IR image I don't need to do that as it's already in a greyscale.

I looked at the wrapper and looks like UYVY is not supported by Unity.

Thanks for the link. I will take a look on it.

[MartyG]

I had a look at the wrapper's list of texture formats, and it does seem that UYVY has no definition beside it in the list, unfortunately.

[ESchi3]

Hey guys,

I found a solution for the issue.

I was following the Intel examples with Unity. After some investigation, I noticed there were two "problems" that were forbidden me to make face detection work.

1) When converting the image from Frame to Texture2D, they were using a constructor with linear set to true (https://docs.unity3d.com/ScriptReference/Texture2D-ctor.html). This was the reason the image I got inside Unity was different from what I had in RealSense Viewer.

2) Now, the reason the face tracking wasn't working: the image from the camera comes flipped! Don't know why, but the algorithm just doesn't work properly in this case. To fix that, when converting from Texture2D to Mat (OpenCV format), I flipped the image. Them it started working perfectly, even for the infrared images.

Maybe it helps someone in the future.

[KBrad6]

Hey @ESchi3​ do you happen to have the code you used to get this working? I'm looking for 3d face tracking with realsense in unity.

[MartyG]

Hi KBrad6,

I just wanted to add that the commercial software Nuitrack SDK supports Unity face tracking with the RealSense 400 Series cameras. It costs 40 dollars a year but has a free trial version.

​https://download.3divi.com/Nuitrack/doc/UnityFaceTracking_page.html