Value of unknown pixels in depth-map

[BHerl1]

Hello forum.

Processing values in depth-streams from a D435-camers, I observe both 0 and 65535 (== max 16 bit int). I was under the impression that 0 indicates an "unknown" pixel (i.e. its depth is sufficiently uncertain, based on selected heuristics). But what does a 65535-value indicate? (I see this both in C++ and in the Python-wrapper.)

[MartyG]

The 0 to 65535 range is apparently related to the brightness of the pixel, where '0' is black and 65535 is white. In terms of general image processing principles, if 65535 appears in your results then it can sometimes mean that this part of the image is light-saturated. It is okay to have some of these non-useful pixel values in your image though, and may be unavoidable.

[BHerl1]

Hi Marty.

Thanks for your quick reply, however is this really true for pixels in the depth-map? Do you have links to documentation where this is described?

Regards,

- Bjarne

[MartyG]

The 0 to 65535 range can be used in a variety of settings related to color brightness, such as hue and intensity. Googling for 'pixel value 65535' can get some good search results on the subject.

Here is my particular source I quoted though. If you do a Find operation on the page for 65535, it will highlight all the instances where it is mentioned in relation to pixel brightness.

http://wiki.freepascal.org/BGRABitmap_Pixel_types http://wiki.freepascal.org/BGRABitmap_Pixel_types

Edit: apparently the '0 to 65535' description in terms of brightness bests fits greyscale images. It gets a bit more complex when dealing with color images.

[BHerl1]

Thanks Marty, but I believe this document refers to BGRA-bitmaps. I'm talking about the depth-matrix which should contain distance (in mm) from the camera.

[MartyG]

I was just using that particular document to illustrate how the 0-65535 range is applicable to a variety of different lightness / color settings.

In general, a white or black pixel may represent areas that the camera was not able to perceive detail on and create useful data representing that area. This may be because the lighting cast on the object may be too dark, too strong or too reflective / light-absorbing. Non-useful values may also be returned if the area being scanned has a shape or edges that make it harder for the camera to read.

If these values are occurring in small numbers then my advice would be to just ignore them, as they are not likely to greatly impact the quality of the depth image.

[BHerl1]

Marty,

do you mean that 0 and 65535 are both used to indicate pixels of unknown depth? Is there a document or source/header mentioning this, or perhaps someone from Intel would confirm?

Regards,

- Bjarne

[MartyG]

My experience is that RealSense usually describes unknown detail in terms of '0'.

An Intel support agent can probably confirm the meaning of 65535 in the context of RealSense depth scans better than I can.

[BHerl1]

Your experience is the same as mine, Marty. That is, '0' is normally denoting a pixel of unknown depth.

So this boils down to my original question: What do the 65355-values in the depth-map mean? Anyone?

Regards,

- Bjarne

[idata]

Hello BjarneG,

Depth values of 65535 correspond to the maximum depth that the camera can read. Does this make sense based on your observations? If not, please clarify under which circumstances you are seeing 0 vs 65535.Regards,JesusIntel Customer Support

[BHerl1]

Hi Jesus.

Although I agree that 65355 is the max possible depth a D400-camera can read (65355 == 2^16-1 === max value for a 16-bit unsigned int), it doesn't make sense to get these values in my setting. I see 65355 scattered around in the depth-matrix, although less frequent and not in similar patterns as 0-values. I'm indoor with a mix of natural light from windows and T8-tubes. Depth should be between 300 and 8000 cm.

My workaround is an extra pass over the matrix to filter out 65535, otherwise basic statistics is polluted and simple operations like contrast-stretch doesn't make much sense. For me, this works at the moment but such wasted cycles is unacceptable in our finished solution. (Besides, it is a little confusing before you become aware that both 0 and 65535 may occur. )

I use this little snippet

---------------

# include

# include

using namespace std;

using namespace cv;

int main()

{

rs2::pipeline pipe;

rs2::config cfg;

cfg.enable_stream(RS2_STREAM_COLOR, 640, 480, RS2_FORMAT_BGR8, 30);

cfg.enable_stream(RS2_STREAM_DEPTH, 640, 480, RS2_FORMAT_Z16, 30);

pipe.start(cfg);

rs2::frameset frames;

while (true)

{

// Wait for a coherent pair of frames: depth and color

frames = pipe.wait_for_frames();

rs2::depth_frame depth_frame = frames.get_depth_frame();

rs2::video_frame color_frame = frames.get_color_frame();

unsigned int d_w = depth_frame.get_width();

unsigned int d_h = depth_frame.get_height();

unsigned int c_w = color_frame.get_width();

unsigned int c_h = color_frame.get_height();

void *depth_data = (void*)depth_frame.get_data();

void *color_data = (void*)color_frame.get_data();

// Creating OpenCV Matrices

Mat depth_raw(Size(d_w, d_h), CV_16U, depth_data, Mat::AUTO_STEP);

Mat color(Size(c_w, c_h), CV_8UC3, color_data, Mat::AUTO_STEP);

Mat filtered_0(Size(d_w, d_h), CV_8U, cv::Scalar(0));

Mat filtered_65535(Size(d_w, d_h), CV_8U, cv::Scalar(0));

filtered_0.setTo(255, depth_raw == 0);

filtered_65535.setTo(255, depth_raw == 65535);

// Display

imshow("Image", color);

imshow("Filtered 0", filtered_0);

imshow("Filtered 65535", filtered_65535);

int k = waitKey(10);

if (k == 27)

break;

}

return 0;

}

---------------

What I would really like is a post-processing option for the camera to "fill in" unknown values based on some (possible user-selected) heuristics. I believe this is most effectively done at the disparity-map level, hence it belongs in the ASIC / camera, but methods in the SDK might also make sense.

Regards,

- Bjarne

[idata]

Hello BJarneG,

I received feedback from the engineering team. A depth value of 0 means the value is unknown and we know it. A depth value of 65535 means we don't know the value is unknown, but rather think it is valid, but it is not valid. This is just what the ASIC is reporting. At the moment we don't offer invalid pixel filtering but I will raise this as a customer feature request to the development team.Regards,Jesus G.Intel Customer Support

[BHerl1]

Thanks for following up on this Jesus.

Not 100% sure I grasp the full meaning of "we don't know the value is unknown, but rather think it is valid, but it is not valid" But it is nice to know that both can be dealt with in the same way.

Looking forward to the filter to "fill in" these values.

Regards,

- Bjarne

[idata]

Hello BjarneG,

Sorry for the confusing expression. The ASIC erroneously assigns a value of 65535 to those pixels it thinks are valid. However, the pixels are not valid. That's it.Regards,Jesus

[BHerl1]

Thanks for clarifying Jesus,

I observe that "hole filling" seems to be implemented in the spatial post-processing filter. The https://github.com/IntelRealSense/librealsense/tree/master/examples/measure measure example shows how to this in code, and there is a dropdown in realsense-viewer.

Regards,

- Bjarne

[idata]

BjarneG,

Does the "hole-filling" option meet your needs?Jesus

[BHerl1]

I have not had time to try it in my context yet but its effect looks reasonable in the viewer.

As long as it "fills" all "unknown" pixels somehow, the exact heuristics can be improved over time, improving the client-solution also.

Regards,

- Bjarne

[YLi40]

When detecting small targets or detecting edges of objects, the depth value of the depth map is 0. I want to know the underlying cause of the lack of edge depth information and how I should modify it using algorithms. Is it due to a hardware problem or a software problem, is it because the binocular matching for details fails or the active infrared resolution is too low to be projected onto a small object.