How to get the same RGB images from the two different RealSense attached to the same location?

[KK7]

I'd like to get the same RGB images when I switch the RealSense to another. In other words, the RealSense is attached to the particular position such as the tip of the end effector or the top of the robot station at first. Then, it is removed and the second RealSense is attached to the same location. For now, I suppose that the extrinsic parameters for both cameras are same. I'm using two Realsense (SR300).

If I don't do any image processing, the images from the first and the second cameras are slightly different. Even after undistorting both images by the intrinsic calibration, they shouldn't be same. What is the right way to transform the picture from the second RealSense so that it looks as same as the image from the first RealSense? I don't too much care if the pictures don't match at the peripheral area, but I want to match two images near the center.

I'm thinking to use the intrinsic parameters for both RealSense: getting the image from the second sensor, multiplying the inverse matrix of the second sensor's intrinsics, then multiplying the intrinsic matrix for the first sensor. Does this work correctly?

[MartyG]

The extrinsic matrix of no two RealSense cameras will be exactly the same due to manufacturing variances at the factory.

[KK7]

Are you talking about the intrinsic matrix, not the extrinsic matrix? If so, yes. I understand it and I'm asking how to convert/do some image processing to make the both images same.

[MartyG]

The extrinsic matrix. Yes, I just saw your comment from Aug 24 about this:

"I suspect this is not the calibration issue because I suppose the output of rs_extrinsics is predefined in the factory based on each sensor's position and rotation (This is not the issue about the mismatch of the actual stream image, but about the mismatch of the predefined transformation matrix). What I don't get is that the depth/IR/points streams are all from the same sensor, but it shows the different extrinsics."

[KK7]

No, the comment isn't related to this question. This question is not about the extrinsic parameters. Please suppose that you could switch the camera at the exact same position and pose.

[MartyG]

Do you mean that you are looking for a way to physically swap out one camera for another using robotics?

If so, the idea of swapping out the cameras physically made me smile, because it reminded me of my favorite fictional robotics mechanism from the Japanese animation series Zoids: Chaotic Century. A system called CAS (Changing Armor System) detaches parts, stores them and spins a carousel round to select a different set of parts and install them into place with robot arms.

https://www.youtube.com/watch?v=BctGWQL-m8k Zoids Liger Zero transformation theme Toonami - YouTube

In the real world though, you could probably do a camera swap-out system more easily by having a mounting point (like a rotating claw) that rotates, and putting the cameras on the top and the underside of the rotator, with the underside camera upside down. Then when the rotating mount is spun round, the underside camera should end up in the exact same position that the other camera was before the rotation took place.

[KK7]

No, I'm looking for a way to process the image from a different camera (= different intrinsics) attached to the same location of the previous camera.

[MartyG]

I apologize for the confusion. I feel that the kind of math multiplication equations that you want to do with the camera may be possible but they are more often found in multi-page university research papers that use RealSense than found on this forum. The knowledge therefore may not exist within this forum community - it is certainly beyond my own understanding of camera mechanics.

I will link RealSense stream expert jb455 into this discussion in case he has some insight on the topic. Perhaps others on this forum will have suggestions too. Good luck.

[jb455]

Are you just looking at the colour and/or depth images by themselves, or the mapping between them?

If the former: my application doesn't need that level of repeatability so I haven't investigated these differences at all so I can't say whether it's a common issue of if one of your devices may be defective. I haven't done any image correction stuff so can't say whether your suggested approach is valid or not.

I have seen differences in the quality of the colour<->depth mapping between different SR300s though, but unfortunately Intel haven't given us any tools to be able to correct these by recalibration etc.

[KK7]

Yes, I'm looking only the color images among the different RealSense. I shouldn't get the exact same image because each camera has the different intrinsic and distortion parameters, but want to know about the image processing technique to compensate the minor difference.