Items with no label
3335 Discussions

Difference between PXCImage view and cv::Mat view for the depth stream

lluci1
Novice
1,916 Views

Hello

I am working with the RealSense R200 camera using Visual Studio & C++.

I need to use OpenCV to do image processing so I converted the PXCImage * into cv::Mat.

But, as you can see, the two images are different (like if I am loosing information).

Does anybody can help me?

Render display screenshot Imshow display screenshot

Thanks,

Lucie

7 Replies
idata
Employee
629 Views

Hi Lucie,

 

 

Thank you for contacting us.

 

 

We've investigated about your case and found two useful links that relate to this case. The first one is a community thread where the user has a similar question and the second one is a guide to use LibRealSense and OpenCV to stream RGB and Depth Data. The latter has a complete example that explains how to use LibRealSense and OpenCV to display both an RGB stream as well as Depth stream into two separate OpenCV created windows.

 

 

https://software.intel.com/en-us/forums/realsense/topic/599415

 

https://software.intel.com/en-us/articles/using-librealsense-and-opencv-to-stream-rgb-and-depth-data

 

 

Let us know if you find this useful.

 

 

Regards,

 

-Sergio A

 

0 Kudos
lluci1
Novice
629 Views

Hi Sergio A,

Thank you for you answer ,

Unfortunately, I already looked those two links and I am still blocked. It might be a format problem or I don't know..

This is my code (a mix of what I found on internet), if someone can help me :

# include "opencv2/core.hpp"

# include "opencv2/highgui.hpp"

# include

# include "pxcsensemanager.h"

# include "pxcmetadata.h"

# include "util_cmdline.h"

# include "util_render.h"

# include

# include

using namespace std;

using namespace cv;

void RecordORPlayback(pxcCHAR *file, bool record) {

// Create a SenseManager instance

PXCSenseManager *sm = PXCSenseManager::CreateInstance();

PXCSession *session;

// Set file recording or playback

sm->QueryCaptureManager()->SetFileName(file, record);

// Select the color stream

sm->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, 320, 240, 30);

// Initialize and Record 300 frames

sm->Init();

for (int i = 0; i<100; i++) {

// This function blocks until a color sample is ready

if (sm->AcquireFrame(true)

// Retrieve the sample

PXCCapture::Sample *sample = sm->QuerySample();

// Go fetching the next sample

sm->ReleaseFrame();

}

// close down

sm->Release();

}

cv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format)

{

PXCImage::ImageData data;

pxcImage->AcquireAccess(PXCImage::ACCESS_READ, format, &data);

int width = pxcImage->QueryInfo().width;

int height = pxcImage->QueryInfo().height;

if (!format)

format = pxcImage->QueryInfo().format;

int type;

if (format == PXCImage::PIXEL_FORMAT_Y8)

type = CV_8UC1;

else if (format == PXCImage::PIXEL_FORMAT_RGB24)

type = CV_8UC3;

else if (format == PXCImage::PIXEL_FORMAT_DEPTH_F32)

type = CV_32F;

cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0], data.pitches[0]);

pxcImage->ReleaseAccess(&data);

return ocvImage;

}

int main(int argc, WCHAR* argv[]) {

//Define some parameters for the camera

cv::Size frameSize = cv::Size(320, 240);

float frameRate = 30;

cv::Mat frameColor = cv::Mat::zeros(frameSize, CV_8UC3);

cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_32F);

/* Creates an instance of the PXCSenseManager */

PXCSenseManager *pp = PXCSenseManager::CreateInstance();

if (!pp) {

wprintf_s(L"Unable to create the SenseManager\n");

return 3;

}

/* Sets file recording or playback */

PXCCaptureManager *cm = pp->QueryCaptureManager();

pxcCHAR *filename = (L"DEPTHRenderTEST");

RecordORPlayback(filename, true);

// Create stream renders

UtilRender renderc(L"Color"), renderd(L"Depth"), renderi(L"IR"), renderr(L"Right"), renderl(L"Left");

pxcStatus sts;

do {

/* Apply command line arguments */

pxcBool revert = false;

PXCVideoModule::DataDesc desc = {};

desc.deviceInfo.streams = PXCCapture::STREAM_TYPE_COLOR | PXCCapture::STREAM_TYPE_DEPTH;

revert = true;

pp->EnableStreams(&desc);

/* Initializes the pipeline */

sts = pp->Init();

if (sts

if (revert) {

/* Enable a single stream */

pp->Close();

pp->EnableStream(PXCCapture::STREAM_TYPE_DEPTH);

sts = pp->Init();

if (sts

pp->Close();

pp->EnableStream(PXCCapture::STREAM_TYPE_COLOR);

sts = pp->Init();

}

}

if (sts

wprintf_s(L"Failed to locate any video stream(s)\n");

pp->Release();

return sts;

}

}

/* Reset all properties */

PXCCapture::Device *device = pp->QueryCaptureManager()->QueryDevice();

device->ResetProperties(PXCCapture::STREAM_TYPE_ANY);

for (;;) {

// for (int i=0;i<51;i++){

/* Waits until new frame is available and locks it for application processing */

sts = pp->AcquireFrame(true);

if (sts

if (sts == PXC_STATUS_STREAM_CONFIG_CHANGED) {

wprintf_s(L"Stream configuration was changed, re-initilizing\n");

pp->Close();

}

break;

}

const PXCCapture::Sample *sample = pp->QuerySample();

if (sample) {

if (sample->depth && !renderd.RenderFrame(sample->depth)) break;

if (sample->depth) {

frameDepth = PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH_F32);

/* string result_depthCV = "DepthOpenCV.png";

imwrite(result_depthCV, frameDepth);*/

namedWindow("Depth", CV_WINDOW_NORMAL);

imshow("Depth", frameDepth);

waitKey(1);

}

if (sample->color) {

frameColor = PXCImage2CVMat(sample->color, PXCImage::PIXEL_FORMAT_RGB24);

namedWindow("Color", CV_WINDOW_NORMAL); //works without this line too

imshow("Color", frameColor);

waitKey(1);

}

/* Releases lock so pipeline can process next frame */

pp->ReleaseFrame();

}

if (_kbhit()) { // Break loop

int c = _getch() & 255;

if (c == 27 || c == 'q' || c == 'Q') break; // ESC|q|Q for Exit

}

}

} while (sts == PXC_STATUS_STREAM_CONFIG_CHANGED);

wprintf_s(L"Exiting\n");

// Clean Up

pp->Release();

system("Pause");

return 0;

}

Thanks,

Lucie

0 Kudos
idata
Employee
629 Views

Hi Lucie,

 

 

We looked online and even though we didn't find any official guides that might help you with the code, we did found a couple of links that may be helpful

 

https://software.intel.com/en-us/forums/realsense/topic/624897

 

http://stackoverflow.com/questions/37266390/how-to-convert-realsense-rgb-frame-to-cvmat-in-ubuntu-or-other-linux-env

 

 

From the looks of both pictures, they're obviously different, but, the depth patterns don't seem to be so different from each other.

 

It looks as if one of the images needs further calibration in order to look the same as the first image.

 

 

I looked at some examples and found that frameColor and frameDepth used this way solved the issue for some customers:

 

cv::Mat frameColor = cv::Mat::zeros(resolutionColor.height, resolutionColor.width, CV_8UC3);

 

cv::Mat frameDepth = cv::Mat::zeros(resolutionDepth.height, resolutionDepth.width, CV_32FC1);

 

 

Keep us updated.

 

 

-Sergio A

 

0 Kudos
idata
Employee
629 Views

Hi Lucie,

 

 

Do you still need assistance with this case? We'll be waiting for your response.

 

 

-Sergio A

 

0 Kudos
lluci1
Novice
629 Views

Hi Sergio,

Thanks for your interest in my issue.

I just solved it

Finally, I used :

cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_16U);

cv::Mat frameDepth8b = cv::Mat::zeros(frameSize, CV_8UC1);

and then,

if (sample->depth) {

frameDepth = PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH);

frameDepth.convertTo(frameDepth8b, CV_8UC1, 255.0 / 4096.0);

// imwrite

string result_depthCV = "DepthOpenCV.png";

imwrite(result_depthCV, frameDepth8b, { CV_IMWRITE_PNG_STRATEGY_DEFAULT });

// imshow

namedWindow("Depth", CV_WINDOW_AUTOSIZE);

imshow("Depth", frameDepth8b);

waitKey(1);

}

I don't really understand with I have to convert frameDepth in CV_8UC1 because normally imwrite can save an 16-bit image in PNG format. But it now at least, it works

Thanks again ,

Lucie

0 Kudos
idata
Employee
629 Views

Hi Lucie,

 

 

Thank you for confirming you've fixed the issue and sharing your results with the community. Don't hesitate to come back if help is needed, we'd be happy to assist you.

 

 

Regards,

 

-Sergio A

 

0 Kudos
秦文翔
Beginner
629 Views

Hi Lucie

You said you have solved this problem, but when I used your method, it didn't work. As you can see:

It is still different.

I hope I will receive your reply.

0 Kudos
Reply