Simulating RealSense sensor in Unity

[MEtti]

I would like to simulate the operation of a RealSense depth sensor within Unity. This would facilitate the development of algorithms for RealSense by making it easy to collect simulated data within Unity. Are there sufficiently detailed technical specs available to pursue this or would this require proprietary information? If the technical information is available pointers and links are much appreciated.

Please note: I'm aware of the RealSense wrapper for Unity which permits the streaming of RealSense sensor data in the real world into Unity. This is entirely different than what I'm interested in making, i.e. a simulated RealSense sensor in Unity which would provide simulated data with similar operating and noise characteristics as the actual sensor.

[MartyG · Accepted solution]

There is a risk that you would be infringing Intel's copyright if you attempt to create what is effectively a software emulator of RealSense camera hardware.

It is already possible to test the RealSense Viewer without a RealSense camera. You do need to have some pre-recorded data to load in though. The page linked to below provides some pre-recorded data that you can drop into the RealSense Viewer program to look at it.

https://github.com/IntelRealSense/librealsense/blob/master/doc/sample-data.md librealsense/sample-data.md at master · IntelRealSense/librealsense · GitHub

[MEtti]

Thank you Marty for the information! You have indeed concisely described what I desire: "...a software emulator of RealSense camera hardware." I doubt I am at risk because I would not embark on this project if there was propriety information involved. I'm guessing the emulator would be difficult enough without reverse engineering!

For what it is worth, I have a D415.

[MartyG]

I thought carefully about what you wanted to do. I recalled that before the Unity wrapper was available, I wrote a script in Unity to try to talk to the RealSense camera library files directly and access their functions. This kind of low level access without an in-between interface like a wrapper used to be called '''hitting the metal'. Anyway, it was horribly unstable, suggesting that a simulated camera probably would not work well even if it were legit to do so.

[MartyG]

I transcribed some new information by Dorodnic, the RealSense Software Manager, in a video presentation that might be relevant to you.

"The sample program 'Software Device' can be used to create a fake RealSense device when it is run. The example will teach you how to inject synthetic frames into this fake device, and then you can use the regular point cloud, texture mapping, all the regular APIs to work with it. It can be useful to inject data from other sensors into our existing ecosystem. We're also thinking about giving people tools to map RealSense to other sorts of input".