Streaming compressed video data to disk (via FFMpeg and DirectShow)

[AWit]

We've been working with the RealSense SR300 model for several weeks now, and we hope to start working with the D435 when it arrives. We primarily use the Python interface to the SDK 2.0. However, I'll soon be collecting a sizable data set that will be saved for later processing via computer vision algorithms, and I am not sure how best to approach this. I am somewhat new to computer vision and video processing.

I see that it is possible to stream to disk in ROS bag format. That's great, but it results in very large data files, and we will be storing/transferring a lot of data. After some discussion, we've decided that lossy video compression would be acceptable. I have successfully transcoded both the RGB and depth streams to disk using FFMpeg, but I did so using custom Python code that manually passes each video frame from the RealSense pipeline to FFMpeg via a pipe. It would be much more straightforward if I could use DirectShow to capture the streams directly (It is a requirement of our project that we must use the Windows platform). This seems to work fine for the RGB stream, but fails for the depth data. The symptom is the same as you would see if viewing the camera through VLC: there is only a solid green video stream. From what I understand, DirectShow is interpreting the depth stream as a color stream because it does not recognize an appropriate pixel format. Is there a workaround for this? Or an alternate solution?

Although I did not create it, this forum post seems to describe the problem I have:

https://ffmpeg.zeranoe.com/forum/viewtopic.php?t=4865 https://ffmpeg.zeranoe.com/forum/viewtopic.php?t=4865

In this example, DirectShow seems to support only the yuyv422 pixel format for the depth stream. All of the other entries are listed as "unknown compression type". This is also what I see. For the record, when writing directly via FFMpeg, I used the gray16le pixel format.

Please let me know if I can provide any additional information that might be useful.

[MartyG]

I am not aware of a previous case where RealSense has been used with DirectShow, so I cannot comment on whether it would work, unfortunately. There are some other options though that I can highlight.

* It is possible to use Python to convert a RealSense stream to ASCII text characters, which would use very little storage space.

* Users of Intel Aero drones, which can have a RealSense R200 camera attached to them,use software called Gstreamer to record and save files via FFMPEG. Although those users do so via Ubuntu Linux, Gstreamer has a Windows version, so perhaps you could adapt that technique.

https://gstreamer.freedesktop.org/download/ https://gstreamer.freedesktop.org/download/

* On the subject of RGB-D depth stream compression in general, the link below contains sources for a range of different compression methods.