Multicam Pointclouds in Unity

[MDops]

Hi all,

I've been working with the D415 trying to stream multiple cameras into a single unity scene on one computer. The goal is to have multiple RGB Point clouds in a scene which can be independently transformed. Currently I am working with 2, but plan to work with up to 6.

It seems like I should be able to create a pipeline, list or array of devices based on their Serial NO, and stream independently to separate mesh-based point clouds - but I am not sure how I'd go about doing this.

I am rather new to C# , and am more of a designer than a programmer. If anyone has some suggestions or advice on what to try next, I would appreciate it!

What I know:

-Multicam Example app works great, and streams from all cameras. Both plugged into separate USB Controllers.

-Point clouds can be generated by each camera (one at a time) when using the default prefabs / scene. Hardware is undamaged.

-Both cameras are recognized by Windows and are updated to FW 5.9.14.

What I've tried:

-Installing the new Pre-Released 2.14.1 SDK, but the final step to Build the UnityPackage fails each time.

-Using modified C# scripts that declare a new 'realsensedeviceoriginal' class which is NON-STATIC. This has allowed me to stream depth from multiple cameras in Unity, but still no luck with point clouds. I didn't write this code myself, but tried modifiying PointCloudGenerator.cs to work in a similar way with no success.

I'm running W10 64, with a TR 1920x 12-core, 16GB Ram and a 1080TI, which should be plenty powerful for a pair of cameras at low resolutions. I also have SR300 drivers installed from using the Looking Glass 'HoloPlayer One' device, I'm unsure if this would introduce any errors or conflict.

Thanks so much in advance!

Mike

[MartyG]

I have both SR300 and D415 drivers on my PC. They do not conflict.

For Windows users, there is an easy workaround for problems with building the DLL files. You can just drag and drop the pre built ones included with the SDK. This method has worked consistently.

[idata]

Hi mcdopsa,

Thank you for your interest in the Intel RealSense D415 camera. The updated instructions for building the Unity wrapper can be found here: https://github.com/IntelRealSense/librealsense/tree/master/wrappers/unity.The instructions say that building the Intel.RealSense project "will build both the native library and the .NET wrapper and will copy both DLLs to the Unity project Plugins folder."Regards,Alexandra

[MartyG]

I'd also like to mention that an unofficial extension for the Unity wrapper in RealSense 2.0 has been published by a developer that adds a range of features, including multi-camera support and RGBD point clouds.

[MDops]

Thanks for your help MartyG, this is just what I was after.

Very strange to see the official Intel version is written to explicitly throw an error when using more than one camera. But why? Even though it asks for a device Serial Number, and has pipelines setup for multiple streams, the code specifically tries to make this impossible.

It'd be great to hear from Intel if there is future intention to allow the SDK's script to accept more than one camera, or if there is some other reason to disable multicam from the start. Multicam is an amazing feature of this camera, and would be great to be able to use it with the official SDK.

[MartyG]

A user on the RealSense GitHub forum who wanted to do multicam stream recording tried it out. They said:

**********

I installed the Unofficial extension. It's based on an old wrapper in which the recording was not rightly included. However the package seems really interesting to calibrate multicameras. It solves partly my next issues on my project. But I still wonder how to record multicamera streams at the same time.

**********

The RealSense development team do add to the Unity wrapper when they get the chance. They did a substantive update to it in the 2.14.0 SDK last month, including new samples (such as binding a point cloud prefab to the camera streams and adding processing block capabilities for them).

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/unity/Assets/RealSenseSDK2.0/Scenes/Samples librealsense/wrappers/unity/Assets/RealSenseSDK2.0/Scenes/Samples at master · IntelRealSense/librealsense · GitHub

I do not work for Intel, so I will have to leave it to them to comment on any Unity multicam support plans.

[idata]

Hi mcdopsa,

Multicam support for Unity is planned but we cannot give a timeline for the release of this feature.Regards,Alexandra

[MartyG]

Thanks for your update for mcdopsa, Alexandra.

In the meantime, a workaround edit for recording multiple streams in Unity was published over the weekend.

[ASaur1]

Are there any news on this? Is the Unity official extension for Unity supporting multicam?

Thanks,

Aniol

[MDops]

I believe at this time the official SDK supports Multicam in Unity.