Items with no label
3335 Discussions

D415 in Advanced Mode

BHerl1
Beginner
6,704 Views

Hello community! My first post here so please be gentle...

I cannot find any documentation or explanation of the "Advanced Mode" settings for the D415 camera. Does this exist somewhere?

In particular, I need to remove (or at least significantly reduce) speckles of "unknown depth" in the depth-stream returned from the camera. It would be nice to achieve this by tweaking settings but I could also implement some post-procesing - any hints in this direction would be most welcome!

Regards,

- Bjarne

0 Kudos
26 Replies
MartyG
Honored Contributor III
1,703 Views

Welcome to the community!

The documentation for the 400 Series Advanced Mode is here:

https://github.com/IntelRealSense/librealsense/blob/master/doc/rs400_advanced_mode.md librealsense/rs400_advanced_mode.md at master · IntelRealSense/librealsense · GitHub

0 Kudos
BHerl1
Beginner
1,703 Views

Thankyou, and thankyou.

The documentation you link provides good basic info, as well as explaining why rs-sensor-control do not display advanced-mode params when cmake'ing it like I did.

However, in order to avoid randomly permute 100+ params, some description of the algorithm and heuristics / thresholds in the ASIC would be useful. Browsing the forum I found this https://arxiv.org/pdf/1705.05548.pdf paper , also containing interesting information, but although it mentions the D400-series it focus on R200 which do not have the advanced-mode params.

Does anyone know a paper or article describing how the D400 actually calculates depth? In particular - what heuristics are used to determine if a pixel-depth is unknown?

I'll play around over the next few days and if I discover anything definite, I'll mention it here. Feel free to do the same!

0 Kudos
MartyG
Honored Contributor III
1,703 Views
0 Kudos
BHerl1
Beginner
1,703 Views
0 Kudos
BHerl1
Beginner
1,703 Views

I`m building v2.9.0 from github with -DBUILD_RS400_EXTRAS=true and -DBUILD_EXAMPLES=true.

The docs mention a program called rs400-advanced-mode-sample but I cannot find any trace of it after building, nor do I find any source-file with a similar name. Have anyone successfully built and used rs400-advanced-mode-sample?

0 Kudos
MartyG
Honored Contributor III
1,703 Views

If you are using cmake, it suggests you use the following instruction, which has "cmake .." on the front of it. Have you done this, please?

cmake .. -DBUILD_EXAMPLES=true -DBUILD_RS400_EXTRAS=true

0 Kudos
BHerl1
Beginner
1,703 Views

Hi Marty.

I do the equivalent of what you suggest, but with cmake-gui (I want to control some other settings as well). Running cmake from cmd-line on a clean build-directory gives me the following

begreen@intel-sdk:~/git/librealsense/new-build$ cmake .. -DBUILD_EXAMPLES=true -DBUILD_RS400_EXTRAS=true

-- The C compiler identification is GNU 5.4.0

-- The CXX compiler identification is GNU 5.4.0

-- Check for working C compiler: /usr/bin/cc

-- Check for working C compiler: /usr/bin/cc -- works

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Detecting C compile features

-- Detecting C compile features - done

-- Check for working CXX compiler: /usr/bin/c++

-- Check for working CXX compiler: /usr/bin/c++ -- works

-- Detecting CXX compiler ABI info

-- Detecting CXX compiler ABI info - done

-- Detecting CXX compile features

-- Detecting CXX compile features - done

-- Info: REALSENSE_VERSION_STRING=2.9.0

-- Performing Test COMPILER_SUPPORTS_CXX11

-- Performing Test COMPILER_SUPPORTS_CXX11 - Success

-- Performing Test COMPILER_SUPPORTS_CXX0X

-- Performing Test COMPILER_SUPPORTS_CXX0X - Success

-- Try OpenMP C flag = [-fopenmp]

-- Performing Test OpenMP_FLAG_DETECTED

-- Performing Test OpenMP_FLAG_DETECTED - Success

-- Try OpenMP CXX flag = [-fopenmp]

-- Performing Test OpenMP_FLAG_DETECTED

-- Performing Test OpenMP_FLAG_DETECTED - Success

-- Found OpenMP: -fopenmp

-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")

-- Looking for pthread.h

-- Looking for pthread.h - found

-- Looking for pthread_create

-- Looking for pthread_create - found

-- Found Threads: TRUE

-- Found OpenGL: /usr/lib/x86_64-linux-gnu/libGL.so

-- Checking for module 'gtk+-3.0'

-- Found gtk+-3.0, version 3.18.9

-- Configuring done

-- Generating done

CMake Warning:

Manually-specified variables were not used by the project:

BUILD_RS400_EXTRAS

-- Build files have been written to: /home/begreen/git/librealsense/new-build

begreen@intel-sdk:~/git/librealsense/new-build$

Note the comment near end about BUILD_RS400_EXTRAS not used, and that REALSENSE_VERSION_STRING==2.9.0

After building it (make -j8), I do the following:

begreen@intel-sdk:~/git/librealsense/new-build$ find . -name *rs400* -print

begreen@intel-sdk:~/git/librealsense/new-build$

Note that no filename containing the string "rs400" exists. Have you been able to sucessfully build rs400-advanced-mode-sample ?

0 Kudos
MartyG
Honored Contributor III
1,703 Views

I don't have a 400 Series camera unfortunately, so haven't been able to do a build with it in SDK 2.0 myself yet.

I note that you are using BUILD_RS400_EXTRAS in your cmake, but the warning message says BUILD_RS400_EXTRAS (i.e missing the D on the front of BUILD). I'm not sure why it would be losing the D, but perhaps that is why it cannot find it if it is looking for BUILD instead of DBUILD?

I note though that you were not able to find any file reference containing the term 'RS400'.

I also note that the docs say that "You will need one of the following compilers (or newer) for JSON support:

GCC 4.9

Clang 3.4

Microsoft Visual C++ 2015 / Build Tools 14.0.25123.0

Are you using one of these?

0 Kudos
BHerl1
Beginner
1,703 Views

Hi Marty. Thanks for your relentless efforts to help me out here.

The "-D" is just the way you tell cmake to "define" a property, hence "BUILD_RS400_EXTRAS" (without -D) is a named property defined to be true.

I am using GCC 5.4 so this should not be an issue (although I don't really look for JSON support at this point)

begreen@intel-sdk:~/git/librealsense/new-build$ gcc --version

gcc (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609

Copyright (C) 2015 Free Software Foundation, Inc.

This is free software; see the source for copying conditions. There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

You don't need a 400-camera to build the SDK I believe. It would be interesting to know your results if you decide to try it out.

0 Kudos
MartyG
Honored Contributor III
1,703 Views

I develop primarily in the Unity game engine and so do not have any kind of conventional C# / C++ development environment such as Clang or Visual Studio set up on my PC, unfortunately.

I am not an Intel employee and so I have taken the resarch into your issue as far as I can, as I do not have access to the RealSense developer team for advice. Hopefully one of the Intel support agents can take up your case and find a solution with the greater resources they have available. I apologize that I was not able to resolve your issue myself and wish you the best of luck!

0 Kudos
BHerl1
Beginner
1,703 Views

Your efforts are greatly appreciated Marty. Thanks!

I also hope someone with closer access to the dev-team can respond to this...

0 Kudos
BHerl1
Beginner
1,703 Views

Mmm.... quick update on this. Perhaps someone from Intel would like to comment / clarify? I'm currently on the "master" branch...

The files named "rs400_advanced..." apparently have been renamed to "rs_advanced_mode....". Moreover, the rs400-advanced-mode-sample I have been trying to find seems to exist in source-form under ./build/CMakeFiles/realsense2.dir/src/ds5/ but is no compiled to an executable by the build-process.

The information https://github.com/IntelRealSense/librealsense/blob/master/doc/rs400_advanced_mode.md here should (at least) be updated with the right names and info, and the build process should be fixed. (There are also some missing includes in several sourcefiles on master at the time of writing, but this will probably be fixed quickly as it's pretty obvious to anyone trying to compile.)

0 Kudos
idata
Employee
1,703 Views

Hello BjarneG,

 

 

The Advanced Mode functionality has been built into the RealSense Viewer tool as shown in the attached screenshot. I will share your feedback regarding the status of the documentation with the RealSense team.

 

 

Disregard the posts by tester123. We are investigating them.

 

 

Regards,

 

Jesus G.

 

Intel Customer Support
0 Kudos
BHerl1
Beginner
1,703 Views

Thanks for getting back to me so fast Jesus.

I have seen the Advanced Mode-settings in the realsense-viewer. However, what I really try to understand is how the different settings impact the various steps of your algorithm. Instead of randomly permute all these settings in order to get where I want, I could modify selected parameters with some understanding of the effect on the result.

Btw, loading saved settings from realsense-viewer does not seem to work - perhaps you can investigate this also with the team?

0 Kudos
idata
Employee
1,703 Views

Hi BJarneG,

 

 

I have forwarded your request to document these controls to the RealSense team and have also filed a bug regarding the loading of a saved configuration. Thank you for bringing this to our attention.

 

 

Regards,

 

Jesus

 

Intel Customer Support

 

0 Kudos
BHerl1
Beginner
1,703 Views

Much appreciated Jesus. I recently discovered issue # 975 which seems to be requesting essentially the same. Will someone bump one of these threads when documentation has been updated, or should I look for new documentation regularly?

About crash on loading: It happens when the viewer / SDK tries to push the "Rau-color-threshold-control" param-block to the camera. If I remove that block from the JSON-file, loading works fine. Moreover, setting that block manually from Python works as long as values are proper. If however values are outside the valid range I get the same error as when loading from the viewer. Hence my guess is that something goes wrong when converting from JSON (percentage of max, by the look of them) to the actual values for that particular param-block.

0 Kudos
idata
Employee
1,703 Views

Hello BjarneG,

 

 

We cannot guarantee that we will be able come back to this thread to update on the status of the documentation so you should check documentation regularly.

 

 

When you said earlier that loading saved settings didn't work I thought you mean that the loaded settings didn't get applied correctly. We have seen this happen with some color controls and submitted it as a bug. However, we have not seen a crash occur when loading saved settings. Can you be more specific regarding your system environment and reproduction steps so we can troubleshoot it?

 

 

Regards,

 

Jesus

 

Intel Customer Support
0 Kudos
BHerl1
Beginner
1,703 Views

Ok, I'll browse docs from time to time then.

About loading: The realsense-viewer doesn't crash on loading but shows an alert with an error-msg. However, the underlying cause of the alert is an exception which makes the loading-code abort somewhere in the middle. The result is that only some of the params are loaded correctly (those loaded before the exception), while the rest are untouched. You can work around this by removing the 3 "Rau-color-threshold-control" params from the JSON being loaded.

Note that you can actually set these 3 params if values are proper (integers between 1 and 1022, it seems from the SDK-code). The following Python-snippet based on one of your examples works nicely:

import pyrealsense2 as rs

import time

DS5_product_ids = ["0AD1", "0AD2", "0AD3", "0AD4", "0AD5", "0AF6", "0AFE", "0AFF", "0B00", "0B01", "0B03", "0B07"]

def find_device_that_supports_advanced_mode() :

ctx = rs.context()

ds5_dev = rs.device()

devices = ctx.query_devices();

for dev in devices:

if dev.supports(rs.camera_info.product_id) and str(dev.get_info(rs.camera_info.product_id)) in DS5_product_ids:

if dev.supports(rs.camera_info.name):

print("Found device that supports advanced mode:", dev.get_info(rs.camera_info.name))

return dev

raise Exception("No device that supports advanced mode was found")

try:

dev = find_device_that_supports_advanced_mode()

advnc_mode = rs.rs400_advanced_mode(dev)

print("Advanced mode is", "enabled" if advnc_mode.is_enabled() else "disabled")

# Loop until we successfully enable advanced mode

while not advnc_mode.is_enabled():

print("Trying to enable advanced mode...")

advnc_mode.toggle_advanced_mode(True)

# At this point the device will disconnect and re-connect.

print("Sleeping for 5 seconds...")

time.sleep(5)

# The 'dev' object will become invalid and we need to initialize it again

dev = find_device_that_supports_advanced_mode()

advnc_mode = rs.rs400_advanced_mode(dev)

print("Advanced mode is", "enabled" if advnc_mode.is_enabled() else "disabled")

ctrls = advnc_mode.get_rau_thresholds_control()

print("RAU Thresholds Control: \n", ctrls)

ctrls.rauDiffThresholdRed = 200

ctrls.rauDiffThresholdGreen = 500

ctrls.rauDiffThresholdBlue = 1000

advnc_mode.set_rau_thresholds_control(ctrls)

print("After Setting new value, RAU Thresholds Control: \n", advnc_mode.get_rau_thresholds_control())

except Exception as e:

print(e)

pass

Now, if you set for example the param rauDiffThresholdRed on line 37 to an invalid value, e.g. 2000, an exception with error-code -6 will be raised. The value -6 is also displayed in the alert realsense-viewer shows when loading. Hence, pushing proper values to the camera works and my guess is that there is a bug somewhere in the code converting these values from JSON.

Regards,

- Bjarne

0 Kudos
idata
Employee
1,406 Views

Hello BJarneG,

 

 

Are you saying that if you use the RealSense Viewer to configure and save the values then reload there is no problem?

 

 

But, if you go into the JSON file and change the rauDiffThresholdRed value to an invalid value then the tool raises an exception?

 

 

I just want to be clear for reproduction.

 

 

Regards,

 

Jesus

 

Intel Customer Support
0 Kudos
BHerl1
Beginner
1,406 Views

Jesus,

my apologies if I express myself clumsily. English is not my native language.

I am saying that if I (at the time of opening this thread - I have not tried in a week or so) use the realsense-viewer to save then load the saved file, I get an alert with error-code == -6.

By removing the three mentioned params from the saved JSON-file, loading works fine in realsense-viewer.

The Python-snippet demonstrates that setting the three params actually works for values in the correct value-range. Setting the values outside the correct value-range raises an exception with error-code == -6, in fact the same error as realsense-viewer shows in its alert.

Based on this I conclude that something went wrong when converting these values from the JSON-file (expressed as percentages of max-value apparently).

Regards,

- Bjarne

0 Kudos
Reply