How to iterate over pointcloud for color data

[Andrew_Carluccio]

I am attempting to access the color values of pixels from the RGB sensor with the goal of storing a few frames as "snapshots" in a bitmap format. Because there is no native .getColor() method available in the SDK like there is for depth, my reading has lead me to create a PCL pointcloud from aligned frames from which to extract RGB values point by point.

I do not understand how to iterate over the pointcloud so that a bitmap in (x,y) could be "colored in" with matching values from cloud->points[i] . How should I approach this next step?

Also, if there is an easier way to access pixel (x,y) from the RGB sensor, please let me know!

[Andrew_Carluccio]

It is probably a good idea to show exactly how I am creating the pointcloud:

pcl::PointCloud *points_to_pcl(const rs2::points& points, const rs2::video_frame& color){

pcl::PointCloud *cloud(new pcl::PointCloud);

auto sp = points.get_profile().as();

cloud->width = sp.width();

cloud->height = sp.height();

cloud->is_dense = false;

cloud->points.resize(points.size());

auto tex_coords = points.get_texture_coordinates();

auto vertices = points.get_vertices();

for (int i = 0; i < points.size(); ++i)

{

cloud->points[i].x = vertices[i].x;

cloud->points[i].y = vertices[i].y;

cloud->points[i].z = vertices[i].z;

std::tuple current_color;

current_color = get_texcolor(color, tex_coords[i]);

cloud->points[i].r = std::get<0>(current_color);

cloud->points[i].g = std::get<1>(current_color);

cloud->points[i].b = std::get<2>(current_color);

}

return cloud;

}

This is adapted from Yonatan-Sade's solution...

https://github.com/IntelRealSense/librealsense/issues/1601# issuecomment-387512731 https://github.com/IntelRealSense/librealsense/issues/1601# issuecomment-387512731

[Andrew_Carluccio]

So I realized in looking at that code more closely that I could skip the PCL entirely, so I am now running this:

//for reference, myFrame is an EasyBMP bitmap

void rsFrameManager::makeBitmap(const rs2::points& points, const rs2::video_frame& color){

auto sp = points.get_profile().as();

auto tex_coords = points.get_texture_coordinates();

auto vertices = points.get_vertices();

for (int i = 0; i < points.size(); ++i)

{

std::tuple current_color;

current_color = get_texcolor(color, tex_coords[i]);

int w = (tex_coords[i].u)*width;

int h = (tex_coords[i].v)*height;

myFrame(w, h)->Red = std::get<0>(current_color);

myFrame(w, h)->Green = std::get<1>(current_color);

myFrame(w, h)->Blue = std::get<2>(current_color);

myFrame(w, h)->Alpha = 0;

}

}

Unfortunately, when I print the resulting bitmap, it is just a white frame, so this still does not work, but I suspect this is a step towards my solution.

[MartyG]

The RealSense GitHub has a couple of discussions about getting RGB values from a colorized point cloud.

https://github.com/IntelRealSense/librealsense/issues/1601 Obtaining RGB values fora colorised PCL point cloud. · Issue # 1601 · IntelRealSense/librealsense · GitHub

https://github.com/IntelRealSense/librealsense/issues/1429 Can't get color of point · Issue # 1429 · IntelRealSense/librealsense · GitHub

[Andrew_Carluccio]

Thank you for the reply!

I was familiar with the first link you mentioned but not the second. I think I understand the solution used in the second link but am confused as to what color_intrinsic , used to calculate colorLocation, is and how I should calculate it on my end. How does color_intrinsic work?

EDIT:

I have defined color_intrinsic as

rs2_intrinsics color_intrinsic = colorFrame.get_profile().as().get_intrinsics();

which I believe is proper. Now I am just trying to figure out how to map the verticies to pixels in a BMP and set the colors accordingly. Suggestions?

[MartyG]

If you are mapping 3D XYZ vertices to 2D pixels, I believe that the appropriate instruction is rs2_project_point_to_pixel. There is a detailed discussion about it in the link below.

[Andrew_Carluccio]

Interesting. I was in a time crunch so I used writing to a .png and then converting to .bmp as a bandaid solution to get me up and running for my deadline, but obviously I want to replace that mechanism for performance reasons soon. I will take a look at rs2_project_point_to_pixel this weekend with hopes that it can resolve my current issue on the pointcloud side, where the resulting bitmap only consists of white pixels. I will update once I've gotten to play with this instruction.