Realsense D435 and SLAM?

[PIsoa]

Hi,

We plant to use D435 in a robot for indoor navigation. I have now went trough all documentation and my undersanding is that you need external IMU unit in oder to do that, is that correct? There are also modules called tracking modules T150 and T260, which should work, but T150 is discontinued and T260 is not available, to be released Q3/2018. Current SLAM library, see link below, is done for ZR300, not for D435, will ir work with D435?

https://software.intel.com/sites/products/realsense/slam/developer_guide.html Intel® RealSense™ for Linux - SLAM Library: Developer's Guide

So what are our choises?

[MartyG]

As the ZR300 SLAM library is for the ZR300 camer, I would recommend ORB_SLAM2 for use with the D435

https://arxiv.org/abs/1610.06475 [1610.06475] ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras

The older ZR300 cameras (which have an IMU with gyro and accelerometer built in) are also still available in limited quantities from some Intel Authorized Distributor companies, like this one:

https://www.avnet.com/shop/emea/products/intel/955176-3074457345633931395/ Avnet: Quality Electronic Components & Services

[PIsoa]

Hi,

Thank you for fast reply, the reason why we are interested D435 is it's range which is 10m, that is enough for our needs, ZR300 range is up to 2.8m, which is too little.

[MartyG]

There was a paper recently on extending the depth range of the R200 model. The ZR300 has the same IR components as the R200, so the principles might be transferable to ZR300.

If D435 is your ideal choice then I believe that ORB2 will be a good way to achieve SLAM til the new V200 headwear, SLAM Vision Library and T260 Tracking Module are available.

[NHani]

MartyG I am also planning to use D435 for SLAM, but we do need an IMU for it to work, since the IMU will be external to D435, would there be any way to synchronize sensor streams using the ORB2 library? Intel's SLAM library for ZR300 does that I believe.

[MartyG]

I do not know enough about ORB-SLAM2 to offer a suggestion unfortunately and could not locate an answer through research either.

In regard to how else you might achieve it ... it is a difficult subject to advise on at present, as Intel are still working on the support for hardware sync with external devices such as compasses, with the most recently reported release date estimate being Q2 2018 (i.e sometime between now and June). So without knowing the specifics yet of what the sync feature will be able to do, it's hard to speculate.

Since a tracking module needs to be connected internally to a 50 pin connector on the D435's D4 Vision Processor circuit board, you can't attach an IMU to ia D435 without opening its casing and invalidating your sales warranty.

If you do not want to do that and you do not want a caseless Module Kit version of the camera such as the D430 for easy access to the tracking module connector, then your other option would be to connect an external IMU to one of the D435's four GPIO pins, which are accessible on the outside of the casing without harming your warranty, I believe.

A GPIO pin lets a trigger signal be sent between the camera and another piece of hardware so that timestamps can be synched. Intel's upcoming firmware update to add hardware sync support will probably cover GPIO sync, though I have no information about it so that's just speculation on my part. Sorry I couldn't be of more help in this case!

[NHani]

Thanks MartyG for your response. I looked at HW sync discussed in whitepaper for using multiple D400 series cameras. It discusses an option to provide an external sync signal in addition to treating one of the cameras in the rig to act as master. Does it not mean we can do that now, provide an external sync. signal? The whitepaper in fact goes into detail on how to ensure that the external signal frequency is matched accurately with the sampling rate of the camera.

Apart from the hw sync., signals can be synchronized using frame counters and timestamps in the application software, that is why I wanted to know if any of the SLAM libraries offer this option. I just do not want to put effort into implementing it on my own when there might be libraries out there that do that.

The module versions of the cameras (D410, D415 and D430) do have a connector for tracking module but the tracking module compatible with it is T150 and surprisingly I could not find a retailer selling this module. There is no information available on this module either. Then there is this new module T260 with a new SLAM lib, this gets me even more confused. So you are right, may be the only option for now is to sit and wait

The only real shot I can see right now is to use pin mentioned in whitepaper to do a hardware sync with an external IMU. Can you confirm in light of this whitepaper if the feature is already available? Linked below (see page 6 'E. External Trigger')

https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_Multiple_Camera_WhitePaper.pdf https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_Multiple_Camera_WhitePaper.pdf

Thanks

[MartyG]

As far as I know, the hardware sync is still being worked on. On the RealSense download section for papers, this paper has the subtitle "Feature coming Q2, 2018". This paper was originally only viewable privately by people / companies with a Non Disclosure Agreement with Intel but was recently made public for everyone on the downloads page. I have seen it described that the sync function will be "upstreamed with a firmware patch", suggesting that the functionality exists in Intel's internal testing but is not in the public firmware yet.

I'll provide the link to the paper for other people reading this in case they haven't seen it.

https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_Multiple_Camera_WhitePaper.pdf https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_Mul…

Frame counters certainly are a better way of controlling the timing of a stream than repeatedly turning the projector on and off.

The 400 Series is not the first RealSense camera to have GPIO sync pins for hardware syncing. The ZR300 model has two of them that provide the ability to receive timestamps from external sensors. These are accessible in the ZR300's 'RealSense SDK For Linux' SDK software with this instruction:

auto timestamp_callback = [](rs::timestamp_data entry){};

Because the ZR300 has a built-in IMU, it primarily relies on timestamps from the IMU for syncing during SLAM operations. You can read the documentation for the RealSense SDK For Linux's SLAM Library here:

https://software.intel.com/sites/products/realsense/slam/developer_guide.html Intel® RealSense™ for Linux - SLAM Library: Developer's Guide

I haven't seen any new details lately about the V200 headwear, the T260 Tracking Module or the RealSense SLAM Vision Library that is paired with them, so I can't give any insight on whether those upcoming products will meet your needs. All that is said in public information so far about the new SLAM library is "the SLAM vision library determines pose information by fusing data from a pair of fisheye cameras and an accelerometer in the T260 tracking module. It determines position data real-time for any VR/AR platform and provides Bluetooth protocol to support 6DoF on both HMDs and controllers".

https://realsense.intel.com/solutions/# virtual-reality Solutions - Intel RealSense

I believe the T150 Tracking Module is usually obtained through Intel Authorized Distributor retailer companies (e.g Mouser, Arrow, etc). The last time I checked its availability last month though with multiple distributors, they were all out of stock of it. This may be a reflection on the supply difficulties Intel have had with the D435 camera model, and that they may have stock to provide to the distributors once stocks of the D435 become more readily available.

Until the sync capabilitiy of the 400 Series cameras is released, the ZR300 and its built-in IMU probably represent the best option for getting SLAM up and running quickly on a camera with hardware sync capabilities. The ZR300 is in short supply now that it is a retired product, but some Intel Authorized Distributors do have limited stock of it.

Here's some example suppliers:

RS Components

Avnet

https://www.avnet.com/shop/emea/products/intel/955176-3074457345633931395/ Avnet: Quality Electronic Components & Services

[NHani]

Many thanks MartyG for your detailed responses. Very helpful. Good to know that there is ZR300 that offers everything so I might resort to it as a last option. But if I want to buy the hardware that is latest and compatible with future Realsense software and at the moment can use the existing libraries out there what are my options?

Let's say if I get a D415 module with D4 vision processing card and T150 tracking module. Would I be able to use the SLAM library version that realsense currently offers? I am thinking that for now I can use this and in future when T260 and new SLAM library is offered I may be able to integrate the T260 with D4 VPU card and use the latest library. Is it something feasible in your opinion?

Thanks

[MartyG]

If you want to use the latest 400 Series camera tech, then ORB-SLAM2 is likely to be the best option available until the new RealSense SLAM Vision Library is available. There was a discussion a couple of days ago with someone who wanted to use ORB-SLAM2 with the D415.

As the T150 Tracking Module attaches to the D4 Vision Processor circuit board, which both the D415 and D435 models have, you should be able to do that with a D415 (bearing in mind that you'd be invalidating the D415's warranty if the casing is opened). If you cannot obtain a T150 module, you may be able to attach a different brand of IMU component if you can have a custom 50-pin connection cable built for it and then calibrate the camera afterwards, since the 50-pin connector on the D4 board can accept custom sensors.

I have no definite knowledge on whether the T260 module will work with the .D435, but if the D4's tracking module connector can accept sensors other than the default choices then it is probably within the realms of possibility that it would work. But until the new SLAM Vision Library is available, you would still probably have to use ORB-SLAM2 as your SLAM method, since it's improbable that the SLAM library used with the RealSense SDK For Linux would work with it.

[NHani]

Great, I think I will go with the new hardware and use ORB-SLAM2 for now. About invalidating the D415's warranty, I think I have not been clear, I meant using D415 depth module (the one assembled on a stiffener without enclosure) and not the D415 camera module. Both are similar by the way except that the depth module version does not come with D4 in-built, I need the D4 VPU card separately purchased to attach to it. Here is the link

https://www.framos.com/en/depth-module-d415-20742 Depth Module D415 | 3D Cameras | Cameras | FRAMOS

Thanks

[MartyG]

Yeah, the naming conventions for D415 and D435 are a bit different. D415 has both Module Kit and camera versions, whilst there's only one D435 (camera), as the D435's module is called the D430. And the D435 (camera) has an RGB sensor, but the D430 (module) does not. The D415 Module Kit, meanwhile, has the RGB sensor integrated into it, unlike the D430, where it is attached as a separate component to make a D435 cased camera..