3D object scan with the SR300 in Unity

[CMcQu]

Hi there,

I'm attempting to take in the point cloud values of an object, with the SR300 in Unity, and create a mesh that will reconstruct the object it is scanning. I have thought of using OpenCV and placing a tag of some sort beneath the object to enable tracking, but I'm not sure if that is the right approach. I would love to be able to place the object on some surface and move the camera around the object while capturing the data. The project needs to be code based therefore the 3Dscan program can't be used.

I have followed this community for a long time as a lurker only but need to ask this question in case someone may be able to help. I have gone through the extremely helpful tutorials by MartyG (The man is a legend and a saint) and have a decent grasp on the concepts.

Does anybody know of any existing attempts at this with Unity ? Or am I wasting my time ?

Thank you for taking a look at this for me.

Kind regards,

Conor.

[MartyG]

Thank you very much for your kind words, Conor!

Textured point-clouds in Unity were implemented in RealSense SDK 2.0's previous version before the current one, 2.10.2. You can download the latest version of the SDK (2.10.3) from here:

https://github.com/IntelRealSense/librealsense/releases/ Releases · IntelRealSense/librealsense · GitHub

Quoting from the SDK release notes:

Under the RealSenseDeivce object in Unity's Hierarchy view, you can find PointCloud object that provides a 3D point cloud of the depth data in the form of Unity Particles (Using the Particle System). The PointCloud object uses the PointCloudGenerator.cs script which allows some user control over the output:

Gradient - is a color gradient used to color the particles

Points Size - Controls the size of the Particles

Skip Particles - A factor >= 1 meaning how many points to skip when creating the particles.

*********

If you would prefer to do it in the old SDKs (2016 R2 and 2016 R3), it would be feasible for you to use 3DScan, as the source code for it is provided in the Sample Browser application that comes with the R2 and R3 SDKs so you can create your own custom version. The Browser is inside the 'Intel RealSense SDK Gold' folder that installation of R2 or R3 places on your desktop.

[CMcQu]

Hi Marty !

Thank you very much for the quick response.

I'll definitely be using that Unity script you mentioned there, the particle system seems like it will useful for other parts of the project as well.

Do you think that it would be feasible for me to create a bootleg c# script from that 3D scan program you mentioned ? I've managed to get the point cloud rendering to work but now I need to start implementing the scanning feature.

Thank you again,

Conor.

[MartyG]

The 3DScan sample program is fortunately in C# language, so converting it into another C# scanning program should be straightforward. If you want to integrate it into a Unity app, that may be trickier. Fortunately Unity uses C# too, so that is helpful.

The '2016 R3' SDK has a Unity version of the C# RawStreams sample. So if you could compare the C# RawStreams sample's code to the code of the Unity RawStreams sample, that may give you some useful indicators about how to make a Unity version of 3DScan.

[CMcQu]

Thank you very much, I'm going to jump in with what you're suggesting and hopefully it'll be successful !

[MartyG]

You're very welcome.

Bear in mind that I'm not sure if Unity RawStreams is a Unity implementation of the original C# code, or a program with different code but the same name because it performs the same purpose. But it's as good a place as any to start researching conversion.

Since the R3 SDK has considerable architectural differences to R2, I would compare C# RawStreams in R3 to Unity RawStreams in R3 so you are comparing like for like.

Here's the documentation page for C# RawStreams in R3.

https://software.intel.com/en-us/node/724967 Sample: RawStreams [C++, C# ] | Intel® Software

And its R3 Unity equivalent.

https://software.intel.com/en-us/node/724968 Sample: RawStreams [Unity] | Intel® Software

[CMcQu]

Hi Marty,

I'm having trouble finding the R3 SDK. This website asks me to register when I attempt to download the 3dscan standalone component. Is there a method to download the entire sdk in one package ?

[MartyG]

The R3 SDK has to be installed in modules, with the Essentials module first and then the other modules for the features that you need after that.

Here's some links to launch the downloads immediately in your browser.

ESSENTIALS

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9558/intel_rs_sdk_mw_core_offline_11.0.27.1384.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9558/intel_rs_sdk_mw_core_offline_11.0.27.1384.exe

3D SCAN

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9546/intel_rs_sdk_mw_scan3d_hybrid_11.0.27.8892.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9546/intel_rs_sdk_mw_scan3d_hybrid_11.0.27.8892.exe

CURSOR MODE

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9556/intel_rs_sdk_mw_cursor_hybrid_11.0.27.1404.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9556/intel_rs_sdk_mw_cursor_hybrid_11.0.27.1404.exe

FACE TRACKING AND RECOGNITION

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9560/intel_rs_sdk_mw_face_hybrid_11.0.28.2787.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9560/intel_rs_sdk_mw_face_hybrid_11.0.28.2787.exe

HAND TRACKING

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9562/intel_rs_sdk_mw_hand_hybrid_11.0.27.1404.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9562/intel_rs_sdk_mw_hand_hybrid_11.0.27.1404.exe

USER BACKGROUND SEGMENTATION

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9554/intel_rs_sdk_mw_seg3d_hybrid_11.0.27.2322.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9554/intel_rs_sdk_mw_seg3d_hybrid_11.0.27.2322.exe

[CMcQu]

Perfect, thank you very much !

[CMcQu]

Hi MartyG,

Can I ask how the RealSense 3dScan algorithm works ? As in, how does it match the point cloud taken from one saved frame to the next ?

I'm covering the ICP algorithm in my thesis and wonder if that's what's going on here too. Does the 3DScan program detect features of an object, and does the camera know it's own world coordinates ?

Thank you again for all of your help and I promise that this will be my last question

[MartyG]

There's no limit on the number of times you can ask for help.

The old SDK was not fully open source and some of the algorithms were kept private by Intel as their intellectual property. The old R3 SDK's documentation does have a series of guide pages for its 3D scan feature that you can browse through with selector arrows or the side-menu though.

https://software.intel.com/en-us/node/724971 3D Scan [F200, SR300] | Intel® Software

[CMcQu]

Haha, no worries I won't try to pry. Thank you again by the way and I actually do have another question.

Is there a method to get the camera's world coordinates ? If I want to scan then the transformation matrix multiplications will be handy if I know the position of the camera once I move it for scanning. If not then I understand that I may need to look into feature recognition or some sort of marker tag.

[MartyG]

If you are using the old 2016 R3 SDK then the instruction to project world coordinates to the depth coordinates is ProjectCameratoDepth.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v2016r3/documentation/html/index.html?projectcameratodepth_pxcprojection.html Intel® RealSense™ SDK 2016 R3 Documentation

The opposite process (projecting depth to the world coordinates) is ProjectDepthToCamera.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v2016r3/documentation/html/index.html?projectdepthtocamera_pxcprojection.html Intel® RealSense™ SDK 2016 R3 Documentation

[CMcQu]

Perfect, thank you so much.