Scene Perception/RGBD SLAM/Motion Tracking with Intel® RealSense™ D435

[MBell7]

As new owners of the D435 Camera we succeeded to make the librealsense of the SDK 2.0 to work.

However we did not find any high-level API like the previous SDK 2016 R2 for features like "scene perception" that could compute the position and orientation of the camera according to its environment.

We are using the Unity engine with C# and would like to perform scene perception ising the D435.

①Is there a way to perform such features with the SDK 2.0?

②If no, do you have any recommendation that would use third party tools in order to easily get these features with the D435 (especially with the Unity engine)?

③Are these features still at development phase and planned to be implemented in the provided SDK 2.0 (or any SDK that uses the D435) at some point?

Thank you.

[MartyG]

Support for the camera's tracking module (the T150 Tracking Module) has just been added in the latest SDK 2.0 version 2.10.1. You can read about this module in detail in the 4000 Series camera data sheet from page 41 onwards.

https://software.intel.com/sites/default/files/Intel_RealSense_Depth_Cam_D400_Series_Datasheet.pdf https://software.intel.com/sites/default/files/Intel_RealSense_Depth_Cam_D400_Series_Datasheet.pdf

You may also be interested in OpenCV background removal and object detection sample programs that were added in the latest SDK version.

https://github.com/IntelRealSense/librealsense/tree/master/examples librealsense/examples at master · IntelRealSense/librealsense · GitHub

Regarding other RealSense tracking developments ... a 'V200' headset using a new T260 Tracking Module is in testing in engineering sample form. The headset is paired with the Intel RealSense SLAM Vision Library. SDK 2.0 already has initial support for the V200.

https://realsense.intel.com/solutions/# virtual-reality Solutions - Intel® RealSense™

[MBell7]

Thank you for your quick response MartyG.

I looked at the specifications but we do not have the T150 module.

However, I still do not fully understand: Is it easily possible with my D435 alone to perform scene perception like in the previous SDK 2016 R2 and the R200?

I do not understand why a new product would not be able to do what could be done two years ago with a product that has been discontinued so I must have missed something but I still do not see what I am sorry.

Was the R200 equipped with a tracking module that the D435 does not have?I am referring to this type of feature: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_sp_scene_perception.html Intel® RealSense™ SDK 2016 R2 Documentation

[MartyG]

I apologize for the confusion about the Tracking Module. It does seem to be a separate module from the camera. The documentation does not make this immediately clear, but I can see it now after checking further tables.

Intel have taken an open-source approach with SDK 2.0, based around third-party software such as OpenCV and community contributions on GitHub. So features that may previously have been built into the SDK by Intel may now be obtained by integrating modules provided by non-Intel sources.

The best chance of doing scene perception with the D435 is likely to be by using the RealSense SDK 2.0 in combination with OpenCV.

[MBell7]

I see, even if I am a bit disappointed by this fact, your last answer is very clear thank you very much for your time MartyG!

[MartyG]

Last December, an Intel article about building a SLAM robot was published. It was tested with earlier RealSense cameras, but the article author thought that upgrading the project for the 400 Series cameras when they were available would be straightforward.

https://software.intel.com/en-us/articles/build-an-autonomous-mobile-robot-with-the-intel-realsense-camera-ros-and-sawr?wapkw=slam+vision+library Build an Autonomous Mobile Robot with the Intel® RealSense™ Camera, ROS*, and SAWR | Intel® Software

[MBell7]

I see thank you MartyG,

By the way, you talked about getting motion tracking through OpenCV, do you know any specific feature that I could use from OpenCV to get something to work?

[MartyG]

For human tracking, I recommend googling for 'opencv person tracking',. For tracking objects, google 'opencv object tracking'.

the latest version of the RealSEnse SDK 2.0 has also added OpenCV object detection examples.

https://github.com/IntelRealSense/librealsense/tree/master/examples librealsense/examples at master · IntelRealSense/librealsense · GitHub

[MBell7]

Sorry I did not made myself clear, I am not interested (well, not directly) in 2D tracking but in the "motion tracking" like the tango project. That is to say odometry, the ability of the camera to know where it stands in space (at least according to its previous positions).

[MartyG]

There is a SLAM system for OpenCV called ORB_SLAM2

https://github.com/raulmur/ORB_SLAM2 GitHub - raulmur/ORB_SLAM2: Real-Time SLAM for Monocular, Stereo and RGB-D Cameras, with Loop Detection and Relocalizati…

[MBell7]

Thank you very much for this answer, it seems promising!