Noob Question: RealSense 2.0 sdk: does it have face recognition etc?

[LAlan]

I'm pretty new to the Intel RealSense ecosystem. I played a little bit in the past with SR300 camera, and the Unity SDK. Both were pretty fantastic!

The old SDK could do: gesture recognition, facial recognition, facial land marks, automatic green-screen

Now that the old SDK will no longer be supported, does RealSense 2.0 SDK support these features?

I have a project that needs some of the new features on the upcoming D435 camera. 10 meter range! So great!

But we are all still waiting for pre-order notification.

Anyway, I want to use the D435 as soon as its available, but I am unsure if the SDK currently supports these features:

- facial recognition for one person

- facial recognition for more people (up to 10?)

- facial land marks

- simple gesture recognition

If it is not supported for Unity, thats okay, I think I can write a basic C# wrapper around a C++ DLLs.

But I cannot write facial recognition algorithms or gesture detection myself... I need an SDK help for those things.

If you have any advice, please help. Thank you!

[MartyG · Accepted solution]

Whilst facial recognition and tracking is not yet an integrated part of SDK 2.0, it is possible that you may be able to do so by using OpenCV with it, as OpenCV has modules for it. OpenCV wasn't working properly in the current 2.80 build of the SDK but the dev team have been fixing that and recently updated the OpenCV documentation for SDK 2.0.

https://github.com/IntelRealSense/librealsense/blob/development/doc/stepbystep/getting_started_with_openCV.md librealsense/getting_started_with_openCV.md at development · IntelRealSense/librealsense · GitHub

I would anticipate that the OpenCV fix will be implemented in the next build that is released, as it seems to require that a file called opencv.hpp is added to the SDK's file structure. In the meantime, you can read about OpenCV facial landmark detection here:

https://www.learnopencv.com/facial-landmark-detection/ Facial Landmark Detection | Learn OpenCV

And facial recognition here:

https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html

[LAlan]

Thank you so much for the reply. I really appreciate it.

I have never heard of OpenCV before. I will give it a close look! One final question:

If I use Open CV to get the pixel location of person's face, can I use that pixel location with RealSense depth camera to find how far they are from camera?

If so, I think that is a good enough solution for my project! Thank you!

[MartyG]

You can use OpenCV to get the coordinates of pixels on the face. Here is an example:

http://answers.opencv.org/question/129328/how-to-grab-pixel-coordinates-of-detected-faceeyes/ How to grab pixel coordinates of detected face/eyes? - OpenCV Q&A Forum

Regarding finding the distance between the pixel and the camera, the link below may be of use to you for doing this in OpenCV:

https://www.codesd.com/item/opencv-how-to-calculate-the-distance-between-the-camera-and-the-object-using-the-image.html https://www.codesd.com/item/opencv-how-to-calculate-the-distance-between-the-camera-and-the-object-using-the-image.html

The above page references having to find the focal length of the camera. This used to be complicated to calculate. The new Depth Quality Tool in the latest 2.80 version of SDK 2.0 may be able to find the details for you automatically though.

https://github.com/IntelRealSense/librealsense/blob/master/tools/depth-quality/readme.md librealsense/readme.md at master · IntelRealSense/librealsense · GitHub

BTW, you can find some documentation links for SDK 2.0 here:

[LAlan]

Thanks you again, very much! It sounds like with some studying and hard work, I can get OpenCV to do what I need for the project!

Now I just have to wait for the new camera from Intel! I hope they go on sale soon! Maybe I should buy ten, haha.

[MartyG]

I'm sure Intel would be very pleased if you buy ten! In the meantime though, you could probably start prototyping your project with an SR300 if you still have one and then swap to a D435 later for the longer range, since as far as I know the OpenCV functions in SDK 2.0 are not exclusive to the D-cameras.

You will have to wait for OpenCV compatibility to be officially fixed in the SDK 2.0, but in the meantime you could research the subject. Regarding Unity compatibility with SDK 2.0, I have been experimenting with this, though it's proving tricky so far. I will release a how-to guide as soon as I can get it stable.