Coordinate mapping from left or right IR image to depth

[FApon]

Hi guys, the community has already helped me to guide me in this project.

I am working with the R200 camera, Visual studio 2015, C ++, Windows 10 and OpenCV 3.1.

I currently do the image preprocessing separately in the left and right infrared cameras to identify objects. I need the coordinates (x, y, z) of the object's geometric center, so I have to do a coordinate mapping, but the SDK only allow to do it between depth and RGB:

// Create the PXCProjection instance.

PXCProjection *projection=device->CreateProjection();

// color and depth image size.

PXCImage::ImageInfo dinfo=depth->QueryInfo();

PXCImage::ImageInfo cinfo=color->QueryInfo();

// Calculate the UV map.

PXCPointF32 *uvmap=new PXCPointF32[dinfo.width*dinfo.height];

projection->QueryUVMap(depth, uvmap);

// Translate depth points uv[] to color ij[]

for (int i=0;i

ij[i].x=uvmap[(int)uv[i].y*dinfo.width+(int)uv[i].x].x*cinfo.width;

ij[i].y=uvmap[(int)uv[i].y*dinfo.width+(int)uv[i].x].y*cinfo.height;

}

// Clean up

delete[] uvmap;

projection->Release();

Does anyone know how to perform the mapping between left or right IR and depth? Or in its default IR left or right RGB for me to do the mapping in depth?

Thank you very much.

[idata]

Hello fhfonsecaa,

Thanks for reaching out!Please correct me if I'm wrong. You are able to map depth using the RGB image using QueryUVMap ( https://software.intel.com/sites/landingpage/realsense/camera-sdk/v2016r3/documentation/html/queryuvmap_pxcprojection.html) but now you would like to do it using the IR (right or left) image, right? If not, then I'm confused and I would like you to explain me your goal again.According to https://software.intel.com/sites/landingpage/realsense/camera-sdk/v2016r3/documentation/html/doc_essential_coordinates_mapping.html, which is where you got the lines of code above, this is the only way in which the SDK maps depth. Unfortunately the SDK has this limitation, therefore it seems that the only way you will be able to do this mapping will be between RGB and depth.You might want to check if librealsense ( https://github.com/IntelRealSense/librealsense) has this option available, perhaps it has an option that fits your needs better.I hope this information helps you,Pedro M.

[FApon]

Hello Pedro,

Thanks a lot for your answer, I am so sorry it took me so long to reply.

You are right, I need to the mapping from the IR left or right camera to the depth image, but the SDK does not have that option.

I have also tried with the librealsense without luck, but I think I found a solution.

Kind regards.

[jb455]

Using the Raw Streams sample, it looks like the left IR image is aligned with the depth image (if not exactly then very close) - can you use that fact? You may be out of luck with the right image though unless you can use the calibration parameters somehow to get a map from left to right?

[FApon]

Hello jb455

Your suggestion was a nice attempt, but actually the depth image is located exactly between the two infrared cameras, so is not possible to take advantage of that fact. Nevertheless, you gave me a great idea to solve my problem.

Thank you.

[FApon · Accepted solution]

The previous answers gave me an idea,

at the beginning I needed to find out how to map a specific point detected on one of the infrared cameras to the depth image. So, my final solution was to subtract the sector of interest (detected in the IR Left camera, which is the nearest to the color camera) from the RGB image and from the result doing the mapping to depth.

The gap exists, but is not too much.