How to find equivalence of a pixel in the color image to the depth image?

[FApon]

Good afternoon,

I currently have the code below to perform the mapping of the color image to the depth image.

I know that it creates a map of vertices and it goes over them depending on the size of both images, which works perfectly, however in this moment I need to find the exact coodenate of a specific pixel.

For example my pixel of interest is (150,20) in the color image, of this pixel I need to know the depth and in order to find out the exact depth I need to know what is the equivalence of that pixel in the depth image.

I can not figure out how to do this transformation, I even tried to build a pxcImage with a red dot on it to project it over the depth image but it did not work correctly.

Thank you very much.

PS: I'm working with OpenCV, R200, intel SDK and C ++ in Visual Studio

PXCImage* R200Driver::ColorToDepthByQueryUVMap()

{

if (!drawDepth || !projection) return 0;

PXCImage::ImageInfo drawDepthInfo = drawDepth->QueryInfo();

PXCImage::ImageData drawDepthDat;

if (PXC_STATUS_NO_ERROR > drawDepth->AcquireAccess(PXCImage::ACCESS_WRITE, drawDepthInfo.format, &drawDepthDat))

return 0;

/* Retrieve the color pixels */

PXCImage::ImageData cdata;

pxcStatus sts = frameColor->AcquireAccess(PXCImage::ACCESS_READ, PXCImage::PIXEL_FORMAT_RGB32, &cdata);

if (sts >= PXC_STATUS_NO_ERROR) {

sts = projection->QueryUVMap(frameDepth, &uvMap[0]);

if (sts >= PXC_STATUS_NO_ERROR) {

PXCImage::ImageInfo cinfo = frameColor->QueryInfo();

PXCImage::ImageInfo dinfo = frameDepth->QueryInfo();

for (pxcI32 y = 0; y < dinfo.height; y++) {

pxcI32* pDrawDepthDat = (pxcI32*)(drawDepthDat.planes[0] + y * drawDepthDat.pitches[0]);

PXCPointF32 *uvTest = &uvMap[y * dinfo.width];

for (pxcI32 x = 0; x < dinfo.width; x++) {

pDrawDepthDat[x] = 0;

if (uvTest[x].x >= 0 && uvTest[x].x < 1 && uvTest[x].y >= 0 && uvTest[x].y < 1) {

pxcI32 tmpColor = *(pxcI32*)(cdata.planes[0] + (int)(uvTest[x].y * cinfo.height) * cdata.pitches[0] + 4 * (int)(uvTest[x].x * cinfo.width));

pDrawDepthDat[x] = tmpColor;

}

}

}

}

frameColor->ReleaseAccess(&cdata);

}

drawDepth->ReleaseAccess(&drawDepthDat);

return drawDepth;

}

[MartyG]

A useful instruction for you may be MapDepthToColor, which maps an array of depth coordinates to the color coordinates.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?mapdepthtocolor_pxcprojection.html Intel® RealSense™ SDK 2016 R2 Documentation

[FApon]

It could be useful, however I should be able to map a point of the color image on the depth, and this point should be easily identifiable in order to identify that coordinate in the depth image.

It is a very long process taking into account that I only need the equivalent coordinate, ie:

That the pixel (57,23) Color = (67,34) Depth

EDIT: Taking a look to the option you said "MapColorToDepth" it turns out that this function takes a pixel of a specific coordinate in the color image and maps it on a pixel in a coordinate of the depth image, which means that if I want to map a pixel to the equivalent one I need to know anyway what it's coordinate is, so we return to the beginning of the problem.

[MartyG]

As a further avenue to consider, I'll link you to a page with a script on it for mapping depth to color using the UV map. Click on the 'c++' tab to get the C++ version of the script.

https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_essential_coordinates_mapping.html Intel® RealSense™ SDK 2016 R2 Documentation

I'll also link RealSense stream programming expert jb455 to this discussion to get his input. JB, what do you think is the best way to map depth to a single color point?

[jb455]

Do you only want to map a single pixel each time, or will you need to map lots of pixels?

If it's a single pixel the easiest way will be to use MapColourToDepth to get the corresponding pixel in the depth image, then retrieving the depth of that pixel from the depthData object. This isn't very time/CPU efficient when mapping "many" pixels though, so be aware of that.

If you want to do lots of pixels, it'd be best to use either the /thread/110837 inverse UV map or ProjectColourToCamera.

[FApon]

Good morning, thanks for your answer. I already managed to do the mapping of the pixels long time ago with the code above, the real problem is that I need to know the equivalence of coordinates:

I have a pixel in position (x, y) in the color image and I need to know which is equivalent to (x', y') in the depth image.

As I have no way to put a visible marker on the color image due to the problems of converting from CvMat to PXCImage, it is no use for me to map the pixels graphically, I need to find the "mathematical" or "geographic" equivalence.

[jb455]

Ok, in the code I linked (inverse uv map), lines 13&14 are calculating the coordinates in the depth image (u,v). So in your case, you'd need invuvmap[x + y * color.info.width] instead of invuvmap[i].

BTW, this is equivalent to what MapColorToDepth does: you give it (i,j) colour image coordinates and it returns corresponding (u,v) depth image coordinates (if a mapping exists: because the cameras look at different places, the depth camera can't see everything the colour camera can see so if there is no corresponding pixel it'll return (-1,-1)). I don't understand what you mean in the edit of post # 2 - from what you say you need, this should be perfect (again, assuming a small number of pixels).