How to find 3 or 5 center pixels from depth image

[FMusa]

Hi everyone,

I am trying to find few center pixels from the depth image. I am trying to implement segmentation, I need to find distance of the segmented object.

This is how I was thinking, this will work. while the colored image is used for segmentation, the full frame depth image will be used to find the distance.

In my code below. I found the diagonal of the depth image, I am trying to find 3 or 5 pixels. I do not how to find the value from the array. I only have the index printed on my console.as print 40,35 but no value.

I hope i did not confuse you. Everything is new to me.

I have tried this;

void SampleArrived(object sender, SampleArrivedEventArgs args)

{

if (args.sample.Depth == null)

{

return;

}

else

{

var depthImageArray = args.sample.Depth;

Debug.Log("my depth value is" + depthImageArray);

var diagonal = Math.Sqrt((depthWidth*depthWidth)+ (depthHeight*depthHeight))/2;

texPlugin.UpdateTextureNative(args.sample.Depth, depthTex2DPtr);

for (int y =0; y < diagonal; ++y)

{

for (int x = 0; x < diagonal; ++x)

{

Debug.Log(string.Format("print {0}, {1}", x, y ));

var path = (@"C:\Users\Admin\Downloads\builtin_shaders-5.2.2f1\DefaultResourcesExtra\Unlit\sample.txt");

var content = (string.Format("print {0}, {1}", x, y));

StreamWriter File = new StreamWriter(path)

File.WriteLine(content);

File.Close();

}

}

Debug.Log("diagonal pixels" + StreamType.ToIndex[40,35];

}

[MartyG]

The link below provides code for getting the depth of a pixel at a specific XY coordinate on the image (e.g 100, 100).

https://github.com/IntelRealSense/librealsense/issues/1413 deprojection pixels to 3d point · Issue # 1413 · IntelRealSense/librealsense · GitHub

[FMusa]

Hi MartyG,

The syntax does not look familiar to Intel RealSense R3000 SDK. I really do not understand how she is getting her raw depth value

[MartyG]

I apologize, I did not see that you had tagged your entry as SR300 and 2016 R3 SDK. I do not read the tags at the bottom of posts.

SR300 is compatible with the current RealSense SDK 2.0. Would it be possible for your project to use it instead of R3?

If not: it has been a while since I used R2 and R3, but I believe the appropriate instruction for mapping 3D depth coordinates to 2D image coordinates is ProjectCameraToDepth.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v2016r3/documentation/html/index.html?projectcameratodepth_pxcprojection.html Intel® RealSense™ SDK 2016 R3 Documentation

[FMusa]

That is no problem I did not tag the question properly myself.

The documentation is not clear on how the syntax should be. Like with a sample code showing how it is done.

I will play around with it and see what it will be returning.

[MartyG]

I located a previous example where ProjectCameraToDepth was discussed on this forum.

On the link below, which has example scripts, example 6 shows what you were asking for - mapping depth to a few pixels.

https://csharp.hotexamples.com/examples/intel.rssdk/PXCMPoint3DF32/-/php-pxcmpoint3df32-class-examples.html https://csharp.hotexamples.com/examples/intel.rssdk/PXCMPoint3DF32/-/php-pxcmpoint3df32-class-examples.html

[FMusa]

Hi MartyG,

Please how can I use R2 since R3 is compatible with R2?

[MartyG]

Yes, they are closely compatible. The main difference is that in '2016 R2' you download the entire SDK as a 1.8 gb download, whilst R3 was split into a series of optional modules. You can launch a download of R2 directly in your browser by using the link below.

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9078/intel_rs_sdk_offline_package_10.0.26.0396.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9078/intel_rs_sdk_offline_package_10.0.26.0396.exe

[FMusa]

I downloaded R2 and installed but it say I have installed the latest version already but in visual studio I can not access PXCMProjection namespace.

[MartyG]

To use Visual Studio with R2 or R3, you should build a 'development environment'. To do this, please choose a link below depending on whether you plan to use C++ or C# in Visual Studio.

C++

With C++ you can set the dev environment up with a Proert Sheet OR do the settings directly.

Using a Proerty Sheet

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_devguide_property_sheets.html Intel® RealSense™ SDK 2016 R2 Documentation

Setting the project settings directly

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_devguide_project_settings.html Intel® RealSense™ SDK 2016 R2 Documentation

C#

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_devguide_csharp_application.html Intel® RealSense™ SDK 2016 R2 Documentation

[FMusa]

Hi MartG,

I tried to setup R2 but I realized i do not have UnityCSharp.unitypackage like this in the directory " $(RSSDK_DIR)/framework/Unity/UnityCSharp.unitypackage" so I went ahead with the working sample codes others got working.

Please have a look at my codes. I am trying to get the distance of the depth image. But anytime I run unity it crashes

void OnSampleArrived(object sender, SampleArrivedEventArgs args)

{

Image image = args.sample.Depth;

if (image == null)

{

Debug.Log("Null depthImage");

}

else

{

if (image != null)

{

ImageInfo cinfo = new ImageInfo() { width = colorWidth, height = colorHeight, format = PixelFormat.PIXEL_FORMAT_BGR }

ImageData outBuffer;

image.AcquireAccess(ImageAccess.ACCESS_READ, PixelFormat.PIXEL_FORMAT_DEPTH_F32, out outBuffer);

var imageHeight = image.Info.height;

var imageWidth = image.Info.width;

float[] pixel = outBuffer.ToFloatArray(0, imageWidth * imageHeight);

image.ReleaseAccess(outBuffer);

Projection projection = device.CreateProjection();

PointF32[] uvmap = new PointF32[cinfo.height * cinfo.width];

projection.QueryUVMap(image, uvmap);

var mappedPixels = new float[cinfo.height * cinfo.width];

for (int i = 0; i < uvmap.Length; i++)

{

int u = (int)(uvmap[i].x * imageWidth);

int v = (int)(uvmap[i].y * imageHeight);

if (u >= 0 && v >= 0 && u + v * imageWidth < pixel.Length)

{

mappedPixels[i] = pixel[u + v * imageWidth];

Debug.Log("distance is:" + mappedPixels);

}

}

for (int y = 0; y < depthHeight; ++y)

{

for (int x = 0; x < depthWidth; ++x)

{

Debug.Log(string.Format("print {0}, {1}", x, y));

int pixelIndex = (y * depthWidth) + x;

Debug.Log(pixelIndex);

}

}

// Clean up

projection.Dispose();

image.ReleaseAccess(outBuffer);

}

[MartyG]

I've never had to use the Unity_CSharp file with R2. What I have always done is create a new Unity project and then when the new scene is open at the start of the project, run the UnityToolkit file in the RSSDK framework folder with a double-leftclick on it whilst Unity is open. This imports all the RealSense files and tools into the project.

Although I do develop extensively in Unity, when I was using R2 I always used the Unity Toolkit's tools (where you drag and drop pre-defined scripts into objects and configure menus in the Inspector panel) to create RealSense controls rather than writing my own RealSense scripts from new (I mostly edited the Toolkit scripts to my needs). So I would be of no help in debugging your script, unfortunately.

If you are interested in using the RealSense tools that the Toolkit installs in Unity, I wrote an extensive range of very detailed step by step guides in the past. An idnex of them can be found at the link below.

https://software.intel.com/en-us/forums/realsense/topic/676139 Index of Marty G's RealSense Unity How-To Guides

[FMusa · Accepted solution]

Hi MartG,

This was the code I finally used to access the depth image, get the center depth pixel and the value. It works well.

Image dimage = args.sample.Depth;

if (dimage != null)

{

ImageData outBuffer;

Status acquireAccessStatus = dimage.AcquireAccess(ImageAccess.ACCESS_READ, PixelFormat.PIXEL_FORMAT_DEPTH_F32, out outBuffer);

if (acquireAccessStatus != Status.STATUS_NO_ERROR)

{

Debug.Log(string.Format("Failed to acquire access to the image. Return code:{0}", acquireAccessStatus));

}

var dwidth = dimage.Info.width;

var dheight = dimage.Info.height;

var centerIndex = ((640 * 190) + 320);

var dpixels = outBuffer.ToFloatArray(0, 640 * dheight);

var result = dpixels[centerIndex];

test = result / (float)2047;

dimage.ReleaseAccess(outBuffer);

}

[MartyG]

Thanks so much for sharing your code with the community!