Problem aligning color to depth

[rchan37]

I have a D435 and am trying to align the color stream to the depth stream. As a sanity check, I also capture infrared images which I believe are automatically aligned with depth. When I compare the infrared and aligned color images, I can see a small offset between the two (~2 pixels in x and ~1 pixel in y). Is this normal? Why don't the images line up?

I am using RealSense SDK 2.11.1 and firmware v5.9.11. The object is about 210mm away from the camera.

Here are the infrared and aligned color images.

Here is the c# test code I am using:

public static void BasicCaptureTest()

{

using (var pipeline = new Pipeline())

using (var config = new Config())

{

config.EnableStream(Stream.Color, 1280, 720, Format.Bgr8);

config.EnableStream(Stream.Depth, 848, 480);

config.EnableStream(Stream.Infrared, 848, 480);

using (var profile = pipeline.Start(config))

{

Thread.Sleep(2000);

using (var frames = pipeline.WaitForFrames())

using (var align = new Align(Stream.Depth))

using (var alignedFrames = align.Process(frames))

{

foreach (var frame in alignedFrames)

{

if (!(frame is VideoFrame vFrame))

{

continue;

}

PixelFormat format;

string filename;

if (vFrame.BitsPerPixel == 8)

{

format = PixelFormat.Format8bppIndexed;

filename = "infrared.png";

}

else if (vFrame.BitsPerPixel == 24)

{

format = PixelFormat.Format24bppRgb;

filename = "color.png";

}

else

{

continue;

}

using (var bitmap = new Bitmap(vFrame.Width, vFrame.Height, format))

{

if (format == PixelFormat.Format8bppIndexed)

{

var palette = bitmap.Palette;

for (var i = 0; i <= byte.MaxValue; i++)

{

palette.Entries[i] = Color.FromArgb(i, i, i);

}

bitmap.Palette = palette;

}

var rect = new Rectangle(0, 0, bitmap.Width - 1, bitmap.Height - 1);

var data = bitmap.LockBits(

rect,

ImageLockMode.ReadWrite,

bitmap.PixelFormat);

CopyMemory(data.Scan0, vFrame.Data, vFrame.Stride * vFrame.Height);

bitmap.UnlockBits(data);

bitmap.Save(filename);

}

}

}

}

}

}

[DllImport("kernel32.dll", EntryPoint = "CopyMemory", SetLastError = false)]

public static extern void CopyMemory(IntPtr dest, IntPtr src, int count);

[MartyG]

I recommend using the same resolution for all streams.

[rchan37]

I tried setting the resolution of all streams to 640x480 (top 2) and 1280x720 (bottom 2). The offset still exists and is about the same in both cases.

[MartyG]

This was like one of those Magic Eye optical illusion puzzles where you stare hard at an image to try to see the hidden picture within it.

Having considered various possibilities, I wonder if it is due to how the 400 Series cameras work over distance. Apparently, at close range the older SR300 model may be more accurate than the 400 Series. This is because of scaling is handled to enable the 400 Series to have a maximum 'expressive range' of 65 meters, whilst the SR300 only needs to see for a maximum expressive range of 2 meters.

The SR300's default scale is 1/32th of a millimeter and the 400 Series' default scale is 1 millimeter, The documentation for the 400 Series' 'Projection' system of coordinate calculation states that "the depth scale can be modified by calling rs2_set_option(...) with RS2_OPTION_DEPTH_UNITS, which specifies the number of micrometers per one increment of depth. 1000 would indicate millimeter scale, 10000 would indicate centimeter scale, while 31 would roughly approximate the SR300's 1/32th of a millimeter scale".

[rchan37]

It's definitely hard to see unless you're specifically looking for it. I had to zoom in on a checkerboard corner and measure the pixels.

I tried changing the depth scale to 0.1mm which I believe is the minimum, but it didn't seem to affect the offset.

For now I'll try to work around it. If the offset is constant and repeatable I can probably account for it. Thanks for your help.

[MartyG]

You're very welcome.

You could also try using the High Accuracy "Visual Offset" setting in the RealSense Viewer.