ZR300/Eulid Cameras with ORB_SLAM2

[SMish7]

Has anyone here, tried to use ORB_SLAM2 with Euclid? If so, can you please share the yaml file, I want to check my values.

Here are my values:

# Camera calibration and distortion parameters (OpenCV)

Camera.fx: 306.17877197265625

Camera.fy: 306.9290466308594

Camera.cx: 158.52369689941406

Camera.cy: 122.74759674072266

Camera.k1: 0.0669681504368782

Camera.k2: -0.11570779234170914

Camera.p1: 0.00017276972357649356

Camera.p2: -0.0005453163757920265

Camera.k3: 0.0

Camera.width: 320

Camera.height: 240

# Camera frames per second

Camera.fps: 30.0

# IR projector baseline times fx (aprox.)

Camera.bf: 6 # baseline = 0.02 m for euclid sensor

# Color order of the images (0: BGR, 1: RGB. It is ignored if images are grayscale)

Camera.RGB: 1

# Close/Far threshold. Baseline times.

ThDepth: 40.0

# Deptmap values factor

DepthMapFactor: 5000.0

P.S. I don't touch the ORB parameters.

[idata]

Hello SubMarIn,

Thanks for contacting us.I have never used ORB_SLAM2, but I will try my best to help you.I'm going to investigate more about this, and I'll let you know when I have updates.Have a nice day.Regards,Leonardo R.

[idata]

Hello SubMarIn,

Unfortunately ORB_SLAM2 is not supported and I didn't find any information that can be helpful for this, nevertheless we are wondering how you retrieved the camera calibration and distortion parameters. Could you be so kind to tell us?We will be waiting for your response.Regards,Leonardo R.

[SMish7]

Actually it supports ORB_SLAM2. My preliminary investigation implies so. I obtained the camera parameters by using the ros camera calibrator but the factory set values can be easily accessed from ros topics of type

http://www.ros.org/doc/api/sensor_msgs/html/msg/CameraInfo.html sensor_msgs/CameraInfo

which is published on running the camera scenario of the euclid for each of the cameras.

People have reported results of ORB_SLAM2 using one of your realsense cameras, eg. https://youtu.be/yjI5RoFeX2c realsense orb-slam indoor outdoor - YouTube

So, I can conclude that it is very much possible to integrate ORB_SLAM2 with euclid realsense. Yeah little bit of tuning may be necessary.

[MSrou2]

Hi,

ORB slam with the fisheye should work on Euclid, you can see the calibration data under /euclid/config/calibration .. i don't remember the file name but it does start with calibration

Best,

Majd

[SMish7]

Hi,

Thanks for your response.

I have a few more questions.

1. Any reasons for prefering fisheye cams above the color camera? Till now I was using the color camera.

2. Do you think that I need to calibrate the camera before using them, or I can rely on the factory set config values?

3. What is the perpendicular distance between the fisheye camera centre and the IR projector?

Regards,

Subodh.

[MSrou2]

Hi,

1. the RGB camera has a rolling shutter, and the fisheye is global shutter, so if you are moving really slow, rolling shutter should be fine. Moreover, the fisheye camera gives you a much wider field of view with more features (ORB features are grayscale if i remember correctly).

2. No need, the factory calibration under the config should be more than enough

3. from the config, look at the depth, that is the distance between left IR and fisheye.

Best,

Majd

[SMish7]

Thanks a lot! since I don't have the Euclid device with me now and I have wait until Monday morning to check this, could you please tell me if these config files are available online in the GitHub? I have some bag files and I can try running orb_slam if I find these configs,

[SMish7]

Hi,

How and where can I access this /euclid/config folder? I am not able to find it after I ssh into the euclid device.

[MSrou2]

it should be on the device .. ? cd /euclid/config from terminal

and I don't think the calibration file is available online .. as it differs from device to device ..

[SMish7]

No it isnt, I get this is output when I try to cd the desired folder.

-bash: cd: /euclid/config: No such file or directory

[MSrou2]

Sorry my bad it's /intel/euclid/config

path: /intel/euclid/config/calibrationResult.json

[SMish7]

Hi,

This is the data contained in the file u have mentioned above..

"Cx": 318.57716080833916,

"Cy": 235.615443920609,

"Fx": 255.9401958615271,

"Fy": 255.92336579003057,

"Kw": 0.920654096071996,

"Rot0": 0.99997449999999999,

"Rot1": -0.001940478,

"Rot2": 0.0068709000000000001,

"Rot3": 0.001990323,

"Rot4": 0.99997170000000002,

"Rot5": -0.0072550519999999997,

"Rot6": -0.006856628,

"Rot7": 0.0072685429999999997,

"Rot8": 0.99995009999999995,

"Tc0": 0.006854383248580051,

"Tc1": 0.0022766429192643835,

"Tc2": 3.0451596814177475e-05,

"TcVar0": 9.999999974752427e-07,

"TcVar1": 9.999999974752427e-07,

"TcVar2": 9.999999974752427e-07,

"Wc0": -0.007261923014907566,

"Wc1": -0.006863882516608069,

"Wc2": -0.0019654345076159587,

"WcVar0": 9.999999974752427e-07,

"WcVar1": 9.999999974752427e-07,

"WcVar2": 9.999999974752427e-07,

"aMeasVar": 0.0049984899999999995,

"abias0": 0.3556473140718861,

"abias1": 0.2170348728970497,

"abias2": -0.31102952411162893,

"abiasvar0": 0.02,

"abiasvar1": 0.02,

"abiasvar2": 0.02,

"accelerometerTransform": [

1.0142311684647092,

0.0,

0.0,

0.0,

1.0068883384978153,

0.0,

0.0,

0.0,

1.009879595867468

],

"calibrationVersion": 8,

"depth": {

"Cx": 159.5,

"Cy": 119.252,

"Fx": 309.86930000000001,

"Fy": 309.86930000000001,

"Rot0": 0.99963380000000002,

"Rot1": -0.0013361169999999999,

"Rot2": 0.027026109999999999,

"Rot3": 0.0032428719999999999,

"Rot4": 0.99749719999999997,

"Rot5": -0.070632120000000007,

"Rot6": -0.026864099999999998,

"Rot7": 0.070693889999999995,

"Rot8": 0.99713620000000003,

"Tc0": 0.037866219999999999,

"Tc1": -0.00069206010000000002,

"Tc2": -0.0019116319999999999,

"TcVar0": 9.999999974752427e-07,

"TcVar1": 9.999999974752427e-07,

"TcVar2": 1.000000013351432e-10,

"Wc0": 0.070730598429258093,

"Wc1": 0.026970880602730781,

"Wc2": 0.0022916843805895031,

"WcVar0": 9.999999974752427e-07,

"WcVar1": 9.999999974752427e-07,

"WcVar2": 9.999999974752427e-07,

"distortionModel": 0,

"imageHeight": 240,

"imageWidth": 320

},

"device": "Alloy",

"distortionModel": 1,

"gyroscopeTransform": [

1.005005262995675,

0.0,

0.0,

0.0,

1.0180072575288375,

0.0,

0.0,

0.0,

1.0012451727595397

],

"imageHeight": 480,

"imageWidth": 640,

"wMeasVar": 0.00019993959999999999,

"wbias0": -0.0033792282090512387,

"wbias1": -0.002028714688184263,

"wbias2": -0.00690673371777983,

"wbiasvar0": 1.136220313356222e-06,

"wbiasvar1": 1.136220313356222e-06,

"wbiasvar2": 1.136220313356222e-06

}

Although I could extract the camera parameters, fx, fy, cx, cy and kw.. I still don't know what is the value of the baseline between the fisheye camera and the projector. But I measured it myself using a ruler and its approximately about 8.15 cm. Correct me if I am wrong?

Besides this, I wanted to know the fps of the fisheye camera.. in the datasheets it is mentioned that the fps is 5,30,60.. but how can I know which fps does it currently have.

Thanks for your time.

Regards,

S.

[MSrou2]

Hi,

You can see in the known issues, that the fps of the fisheye is now 30 (rostopic hz /camera/fisheye/image_raw )

Regarding the translation, under depth: {

"Tc0": 0.037866219999999999,

"Tc1": -0.00069206010000000002,

"Tc2": -0.0019116319999999999,

}

those are the translations between the fisheye and left IR camera (left = the one closer to the fisheye), so it is 0.0378662 in your case..

Best,

Majd

[SMish7]

Thanks! I forgot to check the known issues before posting the query.

[SMish7]

Hi,

Btw, quite contrary to what you suggested, the color camera works better with ORB_SLAM2 as compared to the fisheye camera.

[CWu48]

Hi SubMarln,

We are also use Realsense ZR300 to record a ros bag, and then to run on ORB SLAM2.

My ORB SLAM2 camera configure *.yaml file setting was:

%YAML:1.0

# --------------------------------------------------------------------------------------------

# Camera Parameters. Adjust them!

# --------------------------------------------------------------------------------------------

# Camera calibration and distortion parameters (OpenCV)

Camera.fx: 306.0342102050781

Camera.fy: 306.1972351074219

Camera.cx: 154.90231323242188

Camera.cy: 121.67462921142578

Camera.k1: 0.06804432719945908

Camera.k2: -0.12413176894187927

Camera.p1: -0.0004631184274330735

Camera.p2: 0.00028756982646882534

Camera.k3: 0.0

Camera.width: 320

Camera.height: 240

# Camera frames per second

Camera.fps: 30.0

# IR projector baseline times fx (aprox.) # fx*0.07

Camera.bf: 21.42

# Color order of the images (0: BGR, 1: RGB. It is ignored if images are grayscale)

Camera.RGB: 1

# Close/Far threshold. Baseline times.

ThDepth: 40.0

# Deptmap values factor

DepthMapFactor: 1000.0

# --------------------------------------------------------------------------------------------

# ORB Parameters

# --------------------------------------------------------------------------------------------

# ORB Extractor: Number of features per image

ORBextractor.nFeatures: 3000

We used realsense slam's "/camera_pose" raw data to become slam ground truth trajectory, and then use orb slam2's output "keyframe trajectory" to compute slam benchmark result "ATE"

The ATE result is about 0.13~0.15m

my ros bag information was about that:

Duration: 114sec

Total distance: 18 meter

Avg Velocity: 0.125m

I hope this information is helpful to you

[SMish7]

Hi,

Thanks for your response. I have some followup points at the moment:

1. Which camera are you using? RGB or Fisheye? (I think its the RGB camera as the camera parameters are same as mine)

2. I think the base line for this sensor is about 0.02 m because for a RGB-D type camera, the baseline is the distance between the IR projector and the IR receiver(IR camera), in this case I consider the left IR camera and the distance between the left IR camera and the IR projector is 0.02 m. 0.07 m is the distance between the two ir cameras, so it is the stereo baseline but I dont think it is the IR projector baseline.

3. What is the configuration of the computer on which you are running ORB_SLAM2, I am running ORB_SLAM2 on the euclid itself and the performance is not as good as it is in the PC.

Kindly help me with these queries .

[CWu48]

Hi SubMarln,

1. Which camera are you using? RGB or Fisheye? (I think its the RGB camera as the camera parameters are same as mine)

CH:

I use realsense ros sample to launch camera, and then I enable RGB and Fisheye camera. Because RGB camera for ORB SLAM2, the Fisheye camera for realsense slam

2. I think the base line for this sensor is about 0.02 m because for a RGB-D type camera, the baseline is the distance between the IR projector and the IR receiver(IR camera), in this case I consider the left IR camera and the distance between the left IR camera and the IR projector is 0.02 m. 0.07 m is the distance between the two ir cameras, so it is the stereo baseline but I dont think it is the IR projector baseline.

CH:

I don't consider this condition.

But I first run the Orb slam2 with intel realsense slam record a ros bag files, and then the performance is also pool. I check orb slam2's camera setting *.yaml, I think that it could be camera resolution problem. Finally, I change zr300 rgb camera's resolution from 1920*1080 to 320*240, and then ORB SLAM2 performance is better.

3. What is the configuration of the computer on which you are running ORB_SLAM2, I am running ORB_SLAM2 on the euclid itself and the performance is not as good as it is in the PC.

CH:

My PC is ASUS N55JX Notebook, and then I expand RAM size from 8GB to 16GB, OS is ubuntu16.04 64bits, connect USB port to ZR300 use ros command

BRs,

ChiaHsiu

[SMish7]

Thanks for your response!!