Transformation between Depth, IR and POINTS doesn't match on SR300

[KK7]

I'm using SR300 and trying to get the correct transformation between the color camera and the depth(=IR) camera. I expected that the depth, IR and point clouds streams are all based in the same coordinate system or in the identical transformation, but I noticed that they are slightly different. Please see the following output which I got from `rs_extrinsics`.

Please see the following output which I got from `rs_extrinsics`. The transformations from the color stream to the depth/IR/point clouds are all different. Shouldn't they be the same? Which transformation should I use to do the extrinsic calibration? If anyone saw this issue before, please leave comments too.

- From RS_STREAM_COLOR to RS_STREAM_DEPTH

rotation [[ 0.99999791 0.00196758 -0.0005486 ]

[-0.001966 0.99999398 0.00285708]

[ 0.00055421 -0.00285599 0.99999577]]

translation [[-0.02549983]

[-0.00113199]

[-0.00392998]]

- From RS_STREAM_COLOR to RS_STREAM_INFRARED

rotation [[ 0.99999791 0.00196758 -0.0005486 ]

[-0.001966 0.99999398 0.00285708]

[ 0.00055421 -0.00285599 0.99999577]]

translation [[-0.02549983]

[-0.00113199]

[-0.00392998]]

- From RS_STREAM_COLOR to RS_STREAM_POINTS

rotation [[ 1. 0. 0.]

[ 0. 1. 0.]

[ 0. 0. 1.]]

translation [[-0.02549983]

[-0.00113199]

[-0.00392998]]

- From RS_STREAM_DEPTH to RS_STREAM_POINTS

rotation [[ 0.99999791 -0.001966 0.00055421]

[ 0.00196758 0.99999398 -0.00285599]

[-0.0005486 0.00285708 0.99999577]]

translation [[ 0.]

[ 0.]

[ 0.]]

- From RS_STREAM_COLOR to RS_STREAM_COLOR_ALIGNED_TO_DEPTH

rotation [[ 0.99999791 0.00196758 -0.0005486 ]

[-0.001966 0.99999398 0.00285708]

[ 0.00055421 -0.00285599 0.99999577]]

translation [[-0.02549983]

[-0.00113199]

[-0.00392998]]

- From RS_STREAM_COLOR to RS_STREAM_DEPTH_ALIGNED_TO_COLOR

rotation [[ 1. 0. 0.]

[ 0. 1. 0.]

[ 0. 0. 1.]]

translation [[ 0.]

[ 0.]

[ 0.]]

[MartyG]

It sounds to me like a camera calibration issue. The discussion linked to below may be of use to you.

[KK7]

Thank you for your comment. I suspect this is not the calibration issue because I suppose the output of rs_extrinsics is predefined in the factory based on each sensor's position and rotation (This is not the issue about the mismatch of the actual stream image, but about the mismatch of the predefined transformation matrix). What I don't get is that the depth/IR/points streams are all from the same sensor, but it shows the different extrinsics.

[idata]

Hello kazoo_kmt,

Thanks for reaching out!Perhaps the information in this document will be of help, it explains how the extrinsic camera parameters work. It might help you understand how the coordinate system work on librealsense: https://github.com/IntelRealSense/librealsense/blob/master/doc/projection.md# extrinsic-camera-parametersPedro M.

[KK7]

Thanks for replying. I've already checked the document, but even it says "The extrinsic transformation between depth and infrared is the identity transform", the returned transformation between the IR and the Depth streams are slightly different.

https://github.com/IntelRealSense/librealsense/blob/master/doc/projection.md# appendix-model-specific-details librealsense/projection.md at master · IntelRealSense/librealsense · GitHub

Am I missing something?

[idata]

Perhaps the difference is due to the physical distance between the IR and the depth sensors. As you may know they are a little bit separate one from the other which may translate into slight differences in their readings.

Pedro M.

[KK7]

Both depth and IR streams are from the same IR sensor, shouldn't have any physical distance between them.

[idata]

There are two IR sensors on the SR300 as you can see in Figure 2-1 of the SR300's datasheet ( http://www.mouser.com/pdfdocs/intel_realsense_camera_sr300.pdf), the IR camera and the IR laser projector.

If you check section 4.1 of the same document you can read that "...the IR projector and IR camera operate in tandem using coded light patterns to produce a 2D array of monochromatic pixel values. These values are processed by the imaging ASIC to generate depth and/or infrared video frames ..." So, the depth and IR frames are indeed captured by two different sensors and I believe this is the reason why of the slight offset.Pedro M.

[KK7]

As Figure 4-2 and 4-3 show, both use same IR camera. The depth uses the projected pattern from IR laser projector, but when it captures the pattern, it uses the IR camera and doesn't look there are two separate sensors. I still don't understand why the difference of the image processing creates the difference in the extrinsics...

[KK7]

Maybe this is just from a miscalibration in the factory?

[idata]

Please let me check what might be happening, I'll get back to you as soon as possible.

Pedro M.

[idata]

Hi kazoo_kmt,

We have an update for this case.The matrices for RS_STREAM_INFRARED and RS_STREAM_DEPTH are equal to each other because they come from the same IR sensor. They are very close to, but not exactly the same as, the matrix for RS_STREAM_POINTS because RS_STREAM_POINTS is not actually read from the sensor. It is a synthetic stream, as shown in this doc: https://software.intel.com/sites/products/realsense/camera/rs_8h.html, It can be said that RS_STREAM_POINTS is the ideal, therefore it has a perfect identity matrix. The matrices for RS_STREAM_DEPTH and RS_STREAM_INFRARED are based on real world data, therefore not perfect, but very close to ideal.Does that help?Pedro M.

[KK7]

That makes sense. Thank you for your answer!

[垚李1]

I can't find pxcfaceconfiguration.h file. Where can I download this file?