d435 depth camera

[EAlta]

How can I get the depth (in meters for example) of each pixel of the depth image, given by the realsense d435 depth camera, obtained by the RealSense Viewer?

[MartyG]

The discussion linked to below may be of use to you. It contains code references for getting pixel depth with just SDK 2.0, or by using SDK 2.0 in combination with the OpenCV vision software.

[EAlta]

Im using matlab for the postprocessing, so Im obtaining the sensor´s data from the Real Sense Viewer or python and I need to make use of the distance in meters in matlab.

[MartyG]

There is a discussion on using raw RealSense depth data in MATLAB. Does this help?

Edit: this one too:

[EAlta]

Thanks for your answer MartyG.

Let me explain the issue I have in a better way.

I obtained a depth image, RGB image and Point cloud (.pyd) from the intel RealSense Viewer. Then I made a segmentation process in matlab, so I deleted some points of the original point cloud, but Im still having a .pyd file. Now I need to convert this .pyd file (the processed point cloud) into a depth image, either using matlab or the Intel´s SDK. How can I do it?

I really apreciate your help.

[MartyG]

Getting a .pyd file to load into MATLAB without crashing does not seem to be easy. The most useful references I found were:

https://stackoverflow.com/questions/7811387/embedding-python-in-matlab Embedding Python in MATLAB - Stack Overflow

https://mail.scipy.org/pipermail/numpy-discussion/2009-November/046676.html [Numpy-discussion] bus error in embedded numpy

It sounds as though it would be much easier to process the point cloud in the RealSense Viewer as much as you can, export it as a ply and import it into MATLAB. There are a couple of ply tools available for MATLAB.

https://uk.mathworks.com/matlabcentral/fileexchange/54041-plytomat plyToMat - File Exchange - MATLAB Central

https://uk.mathworks.com/matlabcentral/fileexchange/55968-ply-file-utilities?s_tid=FX_rc2_behav PLY File Utilities - File Exchange - MATLAB Central

[EAlta]

I made a mistake in my las message, when i wrote .pyd, y really meant .ply : ( I obtained a depth image, RGB image and Point cloud (.ply) from the intel RealSense Viewer. Then I made a segmentation process in matlab, so I deleted some points of the original point cloud, but Im still having a .ply file. )

I have alrready made some modifications to the original point cloud in matlab and I saved it as another point cloud (.ply). Now I need to convert this .ply file (the processed point cloud) into a depth image, either using matlab or the Intel´s SDK.

Original data obtained from Intel´s SDK Viewer:

a) Original depth image (I alrready have it in gray scale). b) Original point cloud

Processed pointcloud, in matlab (it is also a .ply file). It is the same as the original, but I deleted some points:

c) I need to convert this one into a depth image.

[MartyG]

This link sounds like what you are trying to achieve:

https://uk.mathworks.com/matlabcentral/answers/228917-3d-point-cloud-to-2d-depth-image 3D Point Cloud to 2D depth image - MATLAB Answers - MATLAB Central

https://uk.mathworks.com/matlabcentral/fileexchange/55031-pointcloud2image-x-y-z-numr-numc pointcloud2image( x,y,z,numr,numc ) - File Exchange - MATLAB Central

[EAlta]

Dou you think is possible to import the .ply file to python and use the SDK´s function "rs2_project_point_to_pixel" to make the convertion?

[MartyG]

This week, Intel highlighted a point cloud system for Python called Pyntcloud that allows the loading in of ply files and the performing of a large number of different types of operation on it. It contains some example programs..

https://github.com/daavoo/pyntcloud GitHub - daavoo/pyntcloud: pyntcloud is a Python library for working with 3D point clouds.

One of these example programs allows the visualization of the point cloud as a color map.

[EAlta]

But once having the .ply on python, I would have to use the function "https://github.com/IntelRealSense/librealsense/blob/5e73f7bb906a3cbec8ae43e888f182cc56c18692/include/librealsense2/rsutil.h# L15 rs2_project_point_to_pixel" that needs the intrinsic parameters.

Is there a way to obtain this intrinsic parameters from the Real Sense Viewer?

Are the intrinsic parameters always the same for the camera or they change accoding to the scene (like maximum distance measured in the scene)?

Thank you for your time.

[MartyG]

Apologies for the delay in responding, I was considering how best to answer your questions.

Yes, intrinsics can change, especially when calibrating the camera. The sample program 'Sensor Control' provides a pre-made means to interface with the camera's details.

https://github.com/IntelRealSense/librealsense/tree/814541662529bf6a0d6178db95e1f948a85dbfa7/examples/sensor-control librealsense/examples/sensor-control at 814541662529bf6a0d6178db95e1f948a85dbfa7 · IntelRealSense/librealsense · GitHub

The API How To page also provides scripting for getting field of view and video stream intrinsics.

https://github.com/IntelRealSense/librealsense/wiki/API-How-To# get-video-stream-intrinsics API How To · IntelRealSense/librealsense Wiki · GitHub

[EAlta]

Ok MartyG, I´m going to read the information and as soon as I have some results, I willl let you know.

Regards.

[EAlta]

MartyG intel_admin intel_corp

You mentioned that intrinsics can change, especially when calibrating the camera, but can intrinsics change while acquiring data from the sensor?

Lets supose Im acquiring data from the sensor and I get 100 frames, and each frame has a pointcloud and depth image,

Is it possible that the camera has taken different intrinsic values while capturing the video, and therefore having different intrinsic values for each frame?

I am going to process each frame's pointcloud independently, so I want to know if it is necessary to obtain the intrinsic values for each frame while taking the video (this would be complicated) ?

I just want to be able to convert a processed pointcloud to a depth image.

Regards.

[MartyG]

Unless the camera is absolutely motionless in position, there are very likely to be variations in the intrinsics. I have never heard of anyone having to manually update the intrinsics on every frame though. The RealSense SDK should automatically take care of those calculations as factors such as distance change.

[EAlta]

Hi again MartyG

Do you know how to get the intrinsic values with python?

[MartyG]

I tracked down the part of the SDK's Python wrapper that handles the creation and modification of intrnsics and extrinsics. It may be a starting point in working out how to access those functions in your own work.

https://github.com/IntelRealSense/librealsense/commit/658d672b97b64257afdbcf18cb60bcea38fe5f87# diff-063dde808a39e40b353ac48a477cbb5bR155 Merge pull request # 1138 from zivsha/register_extrin_python · IntelRealSense/librealsense@658d672 · GitHub

The green parts of the listing show the current version of the code, and the red parts usually indicate parts that have been removed from the script by the most recent changes. Once you find a block of green code relating to intrinsics and extrinsics, you can find it within the main python.cpp wrapper script by looking for the same script line numbers (e.g 156).

https://github.com/IntelRealSense/librealsense/blob/658d672b97b64257afdbcf18cb60bcea38fe5f87/wrappers/python/python.cpp librealsense/python.cpp at 658d672b97b64257afdbcf18cb60bcea38fe5f87 · IntelRealSense/librealsense · GitHub

It might be easier though just to use the original Python script that the SDK's intrinsics / extrinsics support is based on.

https://github.com/IntelRealSense/librealsense/pull/1138 [Python] Allow extrinsics and intrinsics creation and modification by zivsha · Pull Request # 1138 · IntelRealSense/libre…

[jb455]

Intrinsics don't fundamentally change, they're a measurement of the properties of each individual camera. However, it's hard to measure them directly, so they are estimated by a calibration process. This estimation may change slightly when calibration is done, depending on the quality of the calibration, but the actual values should be constant for the life of the camera, not dependent on scene or anything.

The current Intel calibration tool doesn't calibrate intrinsics though, just extrinsics between the depth cameras and possibly depth/colour (which can change if the camera is dropped or exposed to high temperatures/humidity etc), so unless you do your own intrinsic calibration process and write it to the camera (there is a tool for writing intrinsics/extrinsics to a camera included in the calibration API tools) they will remain constant.

[EAlta]

Thankyou jb455

I'm trying to get the intrinsics from my D435 depth camera with python and this is what i get:

>>> import pyrealsense2 as rs2

>>> intrin = rs2.intrinsics()

>>> intrin

width: 0, height: 0, ppx: 0, ppy: 0, fx: 0, fy: 0, model: None, coeffs: [0, 0, 0, 0, 0]

Do you have an idea of what might be happening?

[jb455]

It looks like you're just creating a new Intrinsics object filled with default values (0).

I don't know what the syntax is in Python, but in C# , you can get the intrinsics of a stream like so:

PipelineProfile profile = pipeline.Start();

var stream = (VideoStreamProfile)profile.GetStream(Stream.Depth);

var intrinsics = stream.GetIntrinsics();

I assume it'll be similar in Python.

[EAlta]

Thank you jb455 MartyG. I obtained the intrinsic values with the following code for python:

import pyrealsense2 as rs

pipeline = rs.pipeline()

config = rs.config()

config.enable_stream(rs.stream.depth, 424, 240, rs.format.z16, 30)

# config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

# Start streaming

cfg = pipeline.start(config)

profile = cfg.get_stream(rs.stream.depth)

intr = profile.as_video_stream_profile().get_intrinsics() # Downcast to video_stream_profile and fetch intrinsics

print intr

pipeline.stop()

The intrinsic values I obtained were:

width: 424, height: 240, ppx: 214.063, ppy: 120.214, fx: 213.177, fy: 213.177, model: Brown Conrady, coeffs: [0, 0, 0, 0, 0]

Now I have a pointcloud (.ply file, stored on my computer) that was obtained by my d435, and I want to convert it to a 2d depth image. Do you know how can i achieve this?

I think i have to use the ""https://github.com/IntelRealSense/librealsense/blob/5e73f7bb906a3cbec8ae43e888f182cc56c18692/include/librealsense2/rsutil.h# L15 rs2_project_point_to_pixel(...)" function, but i don´t know how to introduce the previous intrinsic parameters i obtained, and how to introduce the .ply file i have.