Get depth data from D415 within a Unity script

[ALdlm]

I'm currently working with the ARBackground unity project from realsense's official repo, and wanted to get the distance from the camera to the object in the middle. So I used the following C++ code from the samples and compiled it into plugin so I can call the function from Unity:

float getDepth()

{

// Create a Pipeline, which serves as a top-level API for streaming and processing frames

pipeline p;

// Configure and start the pipeline

p.start();

float center_distance = 0.0f;

while (true)

{

// Block program until frames arrive

frameset frames = p.wait_for_frames();

// Try to get a frame of a depth image

depth_frame depth = frames.get_depth_frame();

// The frameset might not contain a depth frame, if so continue until it does

if (!depth) continue;

// Get he depth frame's dimensions

float width = depth.get_width();

float height = depth.get_height();

// Query the distance from the camera to the object in the center of the image

float dist_to_center = depth.get_distance(width / 2, height / 2);

center_distance = dist_to_center;

break;

}

return center_distance;

}

However, when I call this function from inside a Unity script, the original streaming to ARBackground stops and freezes:

[DllImport("DepthPlug")]

private static extern float getDepth();

public void MoveZ ()

{

Vector3 newPos = transform.position;

newPos.z = getDepth() * 0.98f;

newPos.y = 0.0f;

transform.position = newPos;

}

What is the proper way to call the realsense SDK from a Unity script?

[idata]

Hi amonldlm,

Thank you for your interest in the Intel RealSense D415 camera. Did you follow the steps described here? https://github.com/IntelRealSense/librealsense/tree/master/wrappers/unityRegards,Alexandra

[ALdlm]

Hi Alexandra,

Yeah I did, I can receive the depth data correctly. The problem is that the other C# scripts in ARBackground in charge of streaming stops working immediately after calling the plugin.

[idata]

Hi amonldlm,

Please refer to the provided script at .\ibrealsense-master\wrappers\unity\Assets\RealSenseSDK2.0\Scenes\StartHere.unity to see how to call the ARBackground plugin.Regards,Alexandra

[RStac]

Hi anonldlm,

I think the problem with this set up is that in the ARBackground sample the camera is already streaming and the C++ DLL then also tries to setup an additional pipeline stream on top which then confuses the camera stream with two pipelines

I have been looking to do the same thing and eventually found a much simpler way of doing this by changing the RsDevice.cs script within the Unity wrapper by creating a new float variable (CentrePixelDepthDistance) and then modifying the WaitForFrames() function:

private void WaitForFrames()

{

while (!stopEvent.WaitOne(0))

{

using (var frames = m_pipeline.WaitForFrames())

{

using (var depthDist = frames.First(x => x.Profile.Stream == Stream.Depth) as DepthFrame)

{

CentrePixelDepthDistance = depthDist.GetDistance(depthDist.Width / 2, depthDist.Height / 2);

//Debug.Log("The camera is pointing at an object " + CentrePixelDepthDistance + " meters away\t");

}

OnNewFrameSet(frames);

}

}

}

You can then access your depth distance float variable from any other script using:

RealSenseDevice.GetComponent().CentrePixelDepthDistance

Hope this helps!

[MZaid]

Hey,

Thanks for this. Works perfectly fine. Can you tell me if you are able to access your Unity scripts from within this class.. I'm having trouble accessing any custom Unity script that I write from RsDevice.cs.

Thanks.

[RStac]

Hi MZaid,

I wasn't able to access any of my external scripts from inside RsDevice class due to how it is implemented.

It's been a little while since I looked at this but for anything I needed to access externally from within the RsDevice class an public variable in the class so my external scripts could access them. This eventually included grabbing and storing a depth frame every few seconds so I could access it more easily for taking measurements and doing comparisons etc as I started running into limitations and affecting the framerate when attempting to test too many depth points in the frame in real time.

[MZaid]

Makes sense...

What I tried was I copied the whole RsDevice script and pasted it into my own RsDeviceCustom and did the same with RsStreamTextureRenderer. Everything seems to be working fine except for 1 error I get that is in my RsStreamTextureRenderer on textureBinding.Invoke(texture); ArgumentException: failed to convert parameters.

Not sure why I get this.