Items with no label
3335 Discussions

CreateDepthImageMappedToColor Alternative

lquad
New Contributor I
2,480 Views

Hello,

I can see from this discussion and from my code that the memory leakege problem of CreateDepthImageMappedToColor persist.

I need an alternative way to map the depth image to the color image of the camera. Do you know an alternative solution? Have you found some code (possibly easy to understand)

I hope to find help here,

Thanks for your time and assistance.

0 Kudos
1 Solution
lquad
New Contributor I
382 Views

Ok Guys, I followed your suggestions and now something works!. I post my code here, sorry if it is not optimized but it seems to work (obviusly this is only the code related to this topic, for everyone who will use this code remember also to release the access to the rgb image (I did it but few lines of code later).

I'm very grateful for your help

rgb_image = sample->color;

rgb_image->AcquireAccess(PXCImage::ACCESS_READ, Intel::RealSense::Image::PIXEL_FORMAT_RGB24, &data_camera_rgb);

int rgb_width = sample->color->QueryInfo().width;

int rgb_height = sample->color->QueryInfo().height;

//depth_image = sample->depth;

//creating new depth image mapped to Color image

if (projection != NULL) {

depth_image = sample->depth;

PXCImage::ImageData ddata; //creo un oggetto image data

depth_image->AcquireAccess(PXCImage::ACCESS_READ, Intel::RealSense::Image::PIXEL_FORMAT_DEPTH, &ddata); //accedo all'immagine, in mdoalita depth f32

int dwidth = depth_image->QueryInfo().width; //larghezza dell'immagine di depth

int dheight = depth_image->QueryInfo().height; // altezza dell'immagine di depth

pointer = (unsigned short*)ddata.planes[0];

for (int i = 0; i < dwidth *dheight; i++) {

dPixels[i] = (unsigned short) pointer[i];

}

depth_image->ReleaseAccess(&ddata); //release access

projection->QueryInvUVMap(depth_image, &invuvmap[0]); //right syntax: pxcStatus QueryInvUVMap(PXCImage *depth, PXCPointF32 *inv_uvmap);

for (int i = 0; i < rgb_width *rgb_height; i++) {

int u = (int)(invuvmap[i].x * dwidth);

int v = (int)(invuvmap[i].y * dheight);

if ((u >= 0) && (v >= 0) && (u + v * dwidth < dwidth * dheight)) {

mappedPixels[i] = dPixels[u + v * dwidth];

}

else {

mappedPixels[i] = 0;

}

}

}

View solution in original post

12 Replies
jb455
Valued Contributor II
382 Views

MartyG brought this to my attention the other day: https://mtaulty.com/2015/04/16/m_15794/ https://mtaulty.com/2015/04/16/m_15794/

Lines 131-183 create an image similar to CreateDepthImageMappedToColor, though it may be a little slower than the native method.

0 Kudos
idata
Employee
382 Views

Hello andakkino,

 

 

I was wondering if you had the chance to check the suggestion provided by MartyG.

 

 

If you have any other question, don't hesitate to contact us.

 

 

Regards,

 

Andres V.
0 Kudos
lquad
New Contributor I
382 Views

Hello,

Unfortunately I have not solved my problem yet but I'm investigation on my own code (not strictly related to the SDK, in particular I begin to think that I'm creating a wrong texture with depth data collected from the camera and so I don't obtain the final desired result).

I will let you know if I'll need more help from you.

Obviously if I'll find a working solution I will let you know (so I will help other members of the community with their projects).

In the meantime, thank you for the support.

0 Kudos
idata
Employee
382 Views

Hello andakkino,

 

 

In case you need further assistance, don't hesitate to contact us.

 

 

Thank you for considering sharing your experience with the community, we really appreciate it.

 

 

Regards,

 

Andres
0 Kudos
lquad
New Contributor I
382 Views

I'm here again,

I still have my problem, I try to explain it at my best and I'm sorry for my bad English.

Due to the differences in a camera's physical location, lens size, and field of view, color and depth coordinates do not map 1:1 to each other.

So the depth image is "translated" with respect to the color image.

the CreateDepthImageMappedToColor solves my problem because it generates a new depth image with the same resolution of the color image and with the modified coordinates. Unfortunately, due to the memory leakage problem I can't use this function.

I need at least, to correct the coordinate mismatch between the depth and color image. Is there any function in the SDK that can do this?

Jb455, thanks for the link. I studied that code, but I have some doubts:

why he creates a UInt32* colorPtr pointing to colorData.planes[0] [line 138] but then he sets every element of the vector to 0x22FFFFFF; [line 146]

then at line 170 he does the |= 0xCC000000.

I don't understand clearly what the author of the code wants to do. I was thinking that his purpose was to obtain a new deph image with coordinates adapted from the color image. But at this point I think I'm wrong.

ps: I'm programming using C++;

0 Kudos
jb455
Valued Contributor II
382 Views

What do you want to do with it? Create a hybrid depth/colour image to view or get the depth values for each (valid) pixel in the colour image? Also, what language are you using? I outlined a method to do the latter using C# in this thread: . The code I linked to before creates an image where only pixels with valid depth values are coloured.

0 Kudos
MartyG
Honored Contributor III
382 Views

Also, this person used Projection to calibrate their camera.

https://putuyuwono.wordpress.com/2016/03/17/realsense-sdk-calibrate-color-and-depth-stream/ [RealSense SDK] Calibrate Color and Depth Stream | Putu Yuwono Kusmawan

0 Kudos
lquad
New Contributor I
382 Views

Jb455:

I'm using C++;

What I want is to have two images: One color image and one depth image. But I need that color and depth coordinates are mapped 1:1 to each other. So for instance if I were able to create a mesh using the "depth image" I could texturing it using the "color image".

If I use the CreateDepthImageMappedToColor function then I'm able to do what I described above.

For this reason I'm in search of an alternative way.

In other words:

-I have a color image

-I have a depth image

-In the best scenario I would like to have a new depth image where the coordinates mismatch is correct, So the new depth image is spatially aligned with the color image.

MartyG:

I saw that, unfortunately that person uses CreateDepthImageMappedToColor function that is affected by the memory leakage problem.

Thanks again for your help, I really appreciate that.

0 Kudos
MartyG
Honored Contributor III
382 Views

Intel staffer David Lu posted some C++ calibration code a couple of years ago. I apologize if it's the function with the memory leak (JB455 is the resident color / depth programming expert on this forum - thanks for all you do in this field to help people, JB!)

The script was found here: https://software.intel.com/en-us/forums/realsense/topic/597711 R200 Calibration Data

PXCCalibration *calib = m_pProjection->QueryInstance();

PXCCalibration::StreamCalibration calibData={};

PXCCalibration::StreamTransform calibTrans = {};

m_sts[0] = (STATUS)calib->QueryStreamProjectionParameters(PXCCapture::StreamType::STREAM_TYPE_COLOR, &calibData, &calibTrans);

m_sts[0] = (STATUS)calib->QueryStreamProjectionParametersEx(PXCCapture::StreamType::STREAM_TYPE_COLOR, PXCCapture::Device::StreamOption::STREAM_OPTION_ANY,&calibData, &calibTrans);

jb455
Valued Contributor II
382 Views

Ok, yeah, the 'Correct Answer' /thread/110837 here will give you what you want (an array of depth values mapped to the colour image) once you translate it. So the nth item in the MappedPixels array will contain the depth value for the nth pixel in the colour image (scanning left to right, top to bottom).

lquad
New Contributor I
383 Views

Ok Guys, I followed your suggestions and now something works!. I post my code here, sorry if it is not optimized but it seems to work (obviusly this is only the code related to this topic, for everyone who will use this code remember also to release the access to the rgb image (I did it but few lines of code later).

I'm very grateful for your help

rgb_image = sample->color;

rgb_image->AcquireAccess(PXCImage::ACCESS_READ, Intel::RealSense::Image::PIXEL_FORMAT_RGB24, &data_camera_rgb);

int rgb_width = sample->color->QueryInfo().width;

int rgb_height = sample->color->QueryInfo().height;

//depth_image = sample->depth;

//creating new depth image mapped to Color image

if (projection != NULL) {

depth_image = sample->depth;

PXCImage::ImageData ddata; //creo un oggetto image data

depth_image->AcquireAccess(PXCImage::ACCESS_READ, Intel::RealSense::Image::PIXEL_FORMAT_DEPTH, &ddata); //accedo all'immagine, in mdoalita depth f32

int dwidth = depth_image->QueryInfo().width; //larghezza dell'immagine di depth

int dheight = depth_image->QueryInfo().height; // altezza dell'immagine di depth

pointer = (unsigned short*)ddata.planes[0];

for (int i = 0; i < dwidth *dheight; i++) {

dPixels[i] = (unsigned short) pointer[i];

}

depth_image->ReleaseAccess(&ddata); //release access

projection->QueryInvUVMap(depth_image, &invuvmap[0]); //right syntax: pxcStatus QueryInvUVMap(PXCImage *depth, PXCPointF32 *inv_uvmap);

for (int i = 0; i < rgb_width *rgb_height; i++) {

int u = (int)(invuvmap[i].x * dwidth);

int v = (int)(invuvmap[i].y * dheight);

if ((u >= 0) && (v >= 0) && (u + v * dwidth < dwidth * dheight)) {

mappedPixels[i] = dPixels[u + v * dwidth];

}

else {

mappedPixels[i] = 0;

}

}

}

MartyG
Honored Contributor III
382 Views

It's great that you found a solution! Thanks so much for sharing this valuable information for the community, andakkino. Please come to the forum again if you need further help.

0 Kudos
Reply