Items with no label
3335 Discussions

Extracts from the webinar "Hands-on with Multiple Intel RealSense D415 & D435 camera systems"

MartyG
Honored Contributor III
3,769 Views

Hi everyone,

 

I attended the RealSense webinar session "Hands-on with Multiple Intel RealSense D415 & D435 camera systems" and took some notes, which I will share below.

 

Due to article size constraints, the text Q&A answers can be found at the link below, whilst the article below transcribes voice questions made during the webinar.

 

https://forums.intel.com/s/question/0D50P00004B5zslSAB/transcript-of-text-faq-questions-from-the-intel-multiple-realsense-cameras-webinar?language=en_US

 

EXTRACTS OF SPOKEN ANSWERS TO QUESTIONS

 

* In general, over the same distance the D415 has 2 and a half times better resolution than the D435. The D435 though has the benefit of a lower minimum distance (MinZ), allowing you to get closer to objects.

 

* In response to a question about what was the best machine learning algorithm to use with 3D images, it was said that two of the main ones that come to mind are 'convolutional neural networks' and 'randomized decision forests'. Both provide good performance, especially in understanding the 3D structure of the scene.

 

* RealSense 400 Series cameras work really well outdoors. They also work underwater, though the camera needs to be placed in some kind of waterproof casing - "In general, if a human can see it then our camera can see it". It also works in snow in the brightest sunlight and on the beach.

 

* For body scanning, it is recommended that the D415 is used, with a pair of cameras arranged in a parallel configuration, maybe spaced a meter apart.

 

* Accuracy can be increased by using multiple cameras instead of just one.

 

* Regarding the widest camera baseline prototype available: Intel once built a 2 m wide one. The most recent prototypes at the time of writing this were 26 cm. That one is not available for sampling, though Intel do have a 13 cm baseline one that they are able to share under Non Disclosure Agreement (NDA). Please contact support.intel.com about that.

 

* The maximum number of cameras that you can run is somewhat limited to the bandwidth of the USB driver. It is recommended that you look at using dedicated USB host controllers. you can get a USB card for PCIe that has four dedicated chips on there. Additional video RAM is not required unless you are planning to record to memory buffers, and all of the depth analysis and capturing is performed on the device.

 

* Mocap can be transferred to rigged characters in other tools such as Blender using machine learning. Please see the webinar on deep learning for VR / AR.

 

https://realsense.intel.com/webinars/

 

* It is possible to use four outward-facing RealSense cameras with Intel's Aero RTF drone board. It has been done, though the users had to operate at a lower frame-rate and resolution in order to make sure that they were getting robust streaming.

 

* RealSense cameras can be used for art installations. For example, the lobby at Intel headquarters has 20 RealSense cameras that are used to control illumination on the walls and ceiling, with lights that follow you.

 

https://www.youtube.com/watch?v=rxUER9LXo8s

 

* The input of the skeleton tracking neural network is a virtual 2D view. It is an orthographic projection of the point cloud data. the idea is that the network exclusively considers the depth values as a means of estimating distance from the camera.

 

* If you need to use RealSense cameras in conjunction with non-RealSense cameras, an option would be to do post-processing on the data - for example, taking an RGB stream from a RealSense camera and doing post-processing work on it.

 

Multiple RealSense cameras do not interfere with each other, though they may cause interference in a non-RealSense camera placed in proximity to it.

 

Inputs from other non-RealSense cameras can be combined. The most important factor is that you know the extrinsic calibration between each of those cameras. Once you have that then you can potentially use the depth camera to project other modalities of data into that 3D space. The only challenge is that you need to figure out how to do the extrinsic calibration between the cameras.

 

An example use would be using depth data to map to a higher resolution.

 

* Regarding how often the cameras have to be calibrated to keep point clouds aligned during thermal vibration: in general you should not have to, although you may perhaps need to do so in harsher conditions. The stereo algorithm is designed to be robust.

 

* Only hardware depth sync is available on D435, not color sync. If the D435i is released, it will not have color sync either as it is identical to D435 except for the IMU.

 

* Cabling is not absolutely required for syncing. It is best used when micro-second precision in the syncing is required. The parts for creating a cable, available from DigiKey, should only cost a few dollars, and making the cable is relatively simple.

 

* The more cameras that are added to a multi-camera setup, the higher the specification of PC (for example, an Intel i7 processor) that is recommended. The ideal situation is for cameras to be connected directly to a USB port instead of a USB hub so that each camera's USB port has its own USB controller.

 

* Intel are continually enhancing the performance of the cameras through firmware and SDK updates. The D435, for example, now performs better in outdoor sunlight due to the enabling of 'fractional exposure'.

 

* By using two cameras overlapping in parallel, you can get much more comprehensive depth data because there is more redundancy.

 

* There is no interference to RealSense cameras with used with non-Intel cameras, though the non-Intel cameras may experience interference.

 

* A new 'Modified Huffman' lossless compression will enable the streaming of higher resolution and FPS on USB2, and more channels in multi-camera configurations. The compression depends a little on the kind of scene that you are looking at.

 

During decompression, there is no hardware or power hit on the camera. At the PC end, on an i5 or i7 PC, it takes about 5 ms per frame to decompress a 1280x720 stream at 30 FPS.

 

* RealSense cameras can run indefinitely as long as the camera's temperature is kept within tolerances. Problems with streaming ending prematurely are usually related to the PC and problems with maintaining USB.

 

* Intel are not planning on releasing their own SLAM solution, and strongly encourage other companies to provide a solution if they wish. "We will be happy to post whatever you have on our website, and you can even sell it there in the future".

 

* The camera lenses can not be changed without breaking the camera. On large volume orders though, Intel are willing to work with companies on custom designs.

 

* When choosing a large heat-sink for the camera to aid long-term running, it is suggested that the maximum power specification of USB is looked at, as this will give the worst-case scenario for what a heat sink needs to be able to cope with.

 

* Using an external sync source requires very fine frequency resolution. Instead, it is recommended to use one camera as the master.

 

* MIPI may work if V-sync is closely matched with the RealSense camera, though the sync frequency window is very small. It is therefore much easier if only RealSense cameras are used in hardware syncing.

 

"If you are able to get a sync pulse from the MIPI interface camera, and its frame rate is sufficiently close to the expected frame rate for the RealSense camera, then you should be able to .. you can read the white paper for the requirements on how closely you have to match the frame rate of the sensor. It's a limitation of the sensor ... primarily a limitation of the rolling shutter, and we may look at doing something else for the global shutter solution ... right now there is a limitation of those sensors that they can only have a very small window in which they will allow you to trigger externally".

 

* When doing hardware syncing of RealSense cameras, it is recommended that D415s are synced with D415s, and D435s with D435s. The two models can be mixed, but it increases the chance of instabilities.

 

* Regarding how to do multiple camera 3D point cloud alignment, calibration and software tools: Vicalib can be used. "It uses a board that you show to each of the cameras in turn., and it establishes overlapping regions to then minimize the pose of each of those together".

 

Link to Vicalib software:

 

https://github.com/arpg/vicalib GitHub - arpg/vicalib: Visual-Inertial Calibration Tool

 

* Regarding aligning multiple point clouds together: "Vicalib can do this, but there is a simpler approach, which will work in 90% of cases. This is is to take the point cloud from every one of the cameras and then do an Affine Transform. Basically, just rotate and move the point clouds, in 3D space, and then once you've done that, you append the point clouds together and just have one large point cloud".

 

2 Replies
Wgao6
Novice
1,176 Views

Hi

if we have the transform matrix data (R/T), is there any way that we write R/T to the Realsense camera and let the D435 perform a ridged transform of the point clouds using the extrinsics into the same space (instead of doing that outside the camera)?

0 Kudos
MartyG
Honored Contributor III
1,176 Views

In RealSense SDK 2.0, you can write extrinsics to the camera using the register_extrinsics_to instruction. Below is an example script in the Python language for doing so.

https://github.com/IntelRealSense/librealsense/pull/1118 Python API additions by zivsha · Pull Request # 1118 · IntelRealSense/librealsense · GitHub

0 Kudos
Reply