D435 Depth unexpected interpolation.

[ygoda]

I am developing an application on the D435 camera. I will convert from depth to Point Cloud with the program shown below. Then, the object and the depth of the floor seen behind it are interpolated, and Point Clouds that are arranged in a plane that does not actually exist are generated.On the other hand, when using Realsense Viewer, such interpolation can not be seen.How can I get depth information not interpolated? Thank you for your cooperation.

// librealsense-2.16.0

// Visual Studio Professional 2015

const int D435RGB_WIDTH = 640;

const int D435RGB_HEIGHT = 480;

const int DEPTH_WIDTH = 640;

const int DEPTH_HEIGHT = 480;

auto pipe = new rs2::pipeline();

rs2::pipeline_profile profile;

rs2::config cfg;cfg.enable_stream(RS2_STREAM_COLOR, D435RGB_WIDTH, D435RGB_HEIGHT, RS2_FORMAT_BGR8, 30); // RGBcfg.enable_stream(RS2_STREAM_DEPTH, DEPTH_WIDTH, DEPTH_HEIGHT, RS2_FORMAT_Z16, 30); // Depthcfg.enable_stream(RS2_STREAM_INFRARED, 1, DEPTH_WIDTH, DEPTH_HEIGHT, RS2_FORMAT_Y8, 30);cfg.enable_stream(RS2_STREAM_INFRARED, 2, DEPTH_WIDTH, DEPTH_HEIGHT, RS2_FORMAT_Y8, 30);

profile = pipe->start(cfg);

rs2::device selected_device = profile.get_device();

std::cout << get_device_name(selected_device) << std::endl;

auto depth_sensor = selected_device.first();

auto preset_value = RS2_RS400_VISUAL_PRESET_DEFAULT;

depth_sensor.set_option(RS2_OPTION_VISUAL_PRESET, preset_value);

if (depth_sensor.supports(RS2_OPTION_LASER_POWER)) {

auto range = depth_sensor.get_option_range(RS2_OPTION_LASER_POWER); depth_sensor.set_option(RS2_OPTION_LASER_POWER, range.max); // Set max power}if (depth_sensor.supports(RS2_OPTION_ENABLE_AUTO_EXPOSURE)) { depth_sensor.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 0);}if (depth_sensor.supports(RS2_OPTION_EXPOSURE)) { auto exposure_value = 4500.f; depth_sensor.set_option(RS2_OPTION_EXPOSURE, exposure_value);}

auto frames = pipe->wait_for_frames();

auto depth = frames.get_depth_frame();

m_cloud->clear();

points = pc.calculate(depth);auto vertices = points.get_vertices();for (int i = 0; i < points.size(); i++) { pcl::PointXYZRGB b;

b.x = vertices[i].x * 1000;

b.y = vertices[i].y * 1000; b.z = vertices[i].z * 1000; b.r = 255; b.g = 255; b.b = 255;

m_cloud->push_back(b);

}

[MartyG]

I ran extensive tests with the RealSense Viewer but was not able to obtain anything like the results from your own application.

If you want this functionality in the RealSense Viewer, you could try creating your own custom Viewer, as the SDK is open-source.

https://github.com/IntelRealSense/librealsense/tree/master/tools/realsense-viewer librealsense/tools/realsense-viewer at master · IntelRealSense/librealsense · GitHub

[idata]

Hello yoichi_goda,

Thank you for your interest in the Intel RealSense D435 camera.For your project, you might want to check the https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/BKMs_Tuning_RealSense_D4xx_Cam.pdf Best-Known-Methods for Tuning Intel® RealSense™ D400 Depth Cameras for Best Performance paper and also the https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-Depth-PostProcess.pdf Depth Post-Processing for Intel® RealSense™ D400 Depth Cameras.Please let us know if this information helps you and if we can assist you with anything else! Thank you,Eliza

[ygoda]

Hello MartyG, Eliza.

Thank you very much for your reply.

I tried reading the source code of Realsense Viewer, but it was difficult for me. Also, I read the paper of Depth Post-Processing, Best-Known-Methods for Tuning, but I thought that it would not help to solve this problem. However, these documents will be of great help to my knowledge. Thank you very much.

After posting this question, in addition I experimented with the Realsense Viewer as follows. I did output 3D information in PLY format and displayed it with CloudCompare. Then I got the same result as output by my application. Also, when I changed the Render switch icon on the upper right of Realsense Viewer's 3D display from QUADS to POINTS, I got the same result.

So my next aim is to eliminate the Point Cloud interpolation between the object edge and the floor which actually does not exist.

Thank you,

Yoichi

[MartyG]

I see in your script that you turn off auto-exposure in your own application and set a manual exposure value. When you took an image in the RealSense Viewer, did you also disable auto-exposure in the Viewer and set your own exposure value?

If your Viewer image was using auto-exposure, then your own application might get an image closer to it by not disabling auto-exposure.

[ygoda]

Thank you, MartyG, Eliza.

I have another question about advanced_mode_control.

About Rsm group, parameter diffThresh, (below)https://github.com/IntelRealSense/librealsense/blob/master/common/realsense-ui-advanced-mode.h https://github.com/IntelRealSense/librealsense/blob/master/common/realsense-ui-advanced-mode.h

https://github.com/IntelRealSense/librealsense/blob/master/common/realsense-ui-advanced-mode.h librealsense/realsense-ui-advanced-mode.h at master · IntelRealSense/librealsense · GitHub

Line275 slider_float(error_message, "Disparity Difference Threshold", amc.rsm.vals, &STRsm::diffThresh, to_set);

I want to know about the meaning of diffThresh, and how to set the proper value.

Thank you.

Yoichi Goda

[MartyG]

Intel have stated that they do not plan to provide documentation for every setting in Advanced Mode. They say that there is a complex interplay between all parameters, so they instead use machine learning to globally optimize for different usages. If you want to learn what the undocumented functions do, you would need to change their values with trial and error experimentation and see how it affects the camera.

The Advanced Mode documentation that does exist is primarily focuses on detecting cameras, setting Advanced mode, retrieving Advanced Mode values, and writing values to a customized camera configuration file.

https://github.com/IntelRealSense/librealsense/blob/master/doc/rs400/rs400_advanced_mode.md librealsense/rs400_advanced_mode.md at master · IntelRealSense/librealsense · GitHub .

[ygoda]

Hello, MartyG.

Thank you very much for advice.

I see there are no detailed documents for individual parameter for advanced mode.

Thank you,

Yoichi Goda

[ROhle1]

I have to admit that I don't fully understand what you trying to do. But from what I get from the conversations I see a couple of possibilities.

1). In the Viewer, there are custom settings... try the high accuracy setting. I think this does a good job of getting rid of edge artifacts that tend to connect fore objects with background objects... it could be one of the others, look around until you find it... it is there somewhere:).

or

2) find the plane of the floor and then filter the data according to the distance from this plane.

3) Compute the normal for the point cloud and then filter the data to eliminate or include what you want from the scene.

Take a look at Open3D... while the Open3D project is not exactly tailored to your goal, there are enough tools in there that you might find something useful.

Regards,

Rich

[ygoda]

Thank you, rjo__.

About 1),

Thank you for advice about setting.

I tried using Presets, high accuracy setting, but it didn't work well because the exist point of side surface disappear.

About 2) and 3),

how can I judge that a plane is real or not? If Open3D is good for the judging, I'll try to use it.

Thank you very much, rjo__!

Yoichi Goda

[idata]

Hi Yoichi,

Here is some documentation you might find helpful:Best known methods for tuning RealSense depth cameras: https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/BKMs_Tuning_RealSense_D4xx_Cam.pdfDepth post processing for RealSense cameras: https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/Intel-RealSense-Depth-PostProcess.pdfRegards,Sahira

[ygoda]

Thank you for useful information, Sahira.

Although I'm not good at reading/writing documents in English, I'll do my best.

Thank you,

Yoichi Goda

[MartyG]

Hi Yoichi,

Google Translate can translate PDF documents like the one Sahira linked to.

https://translate.google.com/ https://translate.google.com/

STEP ONE

Save the PDF file to your computer.

STEP TWO

Go to the Google Translate website and click on the 'Translate a document' link

STEP THREE

On the next page, click the 'Choose file' button and select the PDF file you saved. Set the original language (English) and the language you want to translate to (on the right side of the screen by the blue Translate button) and click Translate.

STEP FOUR

The computer will now begin translating. After a time, a translation of the PDF will appear in the language that you selected. An example below is an English documentation PDF I converted into Japanese.