Adding custom gestures and / or poses

[idata]

Hi there,

Currently I am working on a project that uses the Intel RealSense SR300 and the RealSense SDK to create an interactive application.

However, I want to add some custom gestures and / or poses. Is this possible?

I know Intel doesn't support adding custom gestures and poses in the RealSense SDK but do you guys have any tips for me on how to make this work?

Thanks in advance!

Kind regards,

Martijn

[MartyG]

It is theoretically possible to create new RealSense gestures but there is not a tool for doing so - you usually have to program them from scratch yourself. So it is a difficult thing to do.

There was a developer who used a program called TouchDesigner to create new RealSense custom gestures though by defining finger positions but I'm not sure how easy it would be to integrate it into your RealSense application (assuming that it probably saves gestures in a different format to the gestures used by the RealSense SDK).

https://www.youtube.com/watch?v=YFD7wDMP984 RealSense custom gesture recognition with TouchDesigner - YouTube

[idata]

Thanks for your quick reply, Marty.

Do you (or anyone else) have experience with creating your own custom gestures/poses using TouchDesigner?

I definitely want to explore the possibilities but I don't know where to begin.

Any other suggestions are welcome.

Thanks.

[MartyG]

Others have asked about programming custom gestures directly in the RealSense SDK in the past, but as far as I know no tutorials exist. It used to be relatively easy in RealSense's predecessor, the 2013 'Perceptual Computing Camera' (also known as Creative Senz3D - not to be confused with the more modern SR300-compatible Creative BlasterX Senz3D). It became more difficult in RealSense because its architecture was more complex than the Perceptual Computing SDK.

Regarding TouchDesigner: this article introduces how to integrate TouchDesigner with RealSense.

https://software.intel.com/en-us/articles/using-the-intel-realsense-camera-with-touchdesigner-part-1 Using the Intel® RealSense™ Camera with TouchDesigner*: Part 1 | Intel® Software

There are also follow-up guides, including one (in Part 3) that teaches how to capture body pose.

Part 2

https://software.intel.com/en-us/articles/using-the-intel-realsense-camera-with-touchdesigner-part-2 Intel® RealSense™ Camera with TouchDesigner*, Part 2: Set Up Camera Nodes to Render for VR | Intel® Software

Part 3

https://software.intel.com/en-us/articles/using-the-intel-realsense-camera-with-touchdesigner-part-3 Intel® RealSense™ Camera with TouchDesigner* Part 3: Create Performance Art | Intel® Software

My own approach to gesture and body pose recognition was to build my own custom system in the Unity game engine that I called CamAnims. This detects body positions and finger configurations based on the rotation angles of virtual objects generated by real-time RealSense camera input.

[idata]

Hi Marty,

Thanks for your helpful tips.

However, I just found out about Project Prague from Microsoft: https://labs.cognitive.microsoft.com/en-us/project-prague https://labs.cognitive.microsoft.com/en-us/project-prague

This seems like a possible solution for creating custom gestures and or poses. Does anyone have experience with this combining the RealSense SDK?

I wanted to try this out, but unfortunately my computer doesn't recognize the SR300 camera anymore so I need to fix this first.

[MartyG]

Thanks for the mention of Project Prague - I haven't heard of that before, and do not know anyone else who has tried it yet.