Items with no label
3335 Discussions

sr300_Blender_Ni-Mate

ggera
Novice
1,716 Views

Ni-Mate by Delicode is a great app:

"NI mate is a unique, cross-platform, cross-sensor real-time motion capture suite that's been designed for anyone to use. No markers, complex and expensive camera setups, long render times - just install NI mate, attach a sensor and open up your favorite 3D suite."

Prior to a Windows10 update, they supported the sr300 and the facial animation possibilities, live in Blender, were excellent.

Here is the current situation:

"We believe the "CreateInstance()" issue is that the runtime on your computer was updated, and the RealSense SDK is, for some reason, not backwards compatible. Another problem with the runtimes / DCM is that they occasionally update the sensor firmware and break compatibility with older software. If you could find an older version of the runtime / DCM you might get the current version of NI mate working with your sensor. I spent a while searching through the Intel webpage, but it seems like they don't have links for the older versions... 

"starting with NI mate 2.13 unfortunately the current plan is to move away from the official Intel RealSense SDK and utilize the alternative librealsense driver without the face recognition algorithms. Due to the massive amounts of complications with sensor devices and runtimes we couldn't justify continuing the support for the official SDK with RealSense."

The above is in response to this situation of mine (prior to the mentioned update I had excellent results, now no function whatsoever):

"after reloading Windows10 this fall, and i think the error i get might be relatively simple?

This is version 2.12

Thanks again for your help--

(13:32:36: Input 1 quit normally.

13:32:41: Starting input 1: RealSense SR300

13:32:41: Input 1: Creating QApplication

13:32:41: Input 1: Failed at PXCSession::CreateInstance()

13:32:41: Input 1: startlive failed)"

So, the dilemma is how to achieve markerless live facial animation without C++ ? Also, since purchasing the sr300 last year the Intel SDK no longer supports Javascript which I can use.

Any ideas? help? suggestions? Thanks.

3 Replies
MartyG
Honored Contributor III
542 Views

Than you very much for the information, GE.

You are not the only person who has been seeking to create and record facial mo-cap animation in real-time with their camera. It's not an easy thing. If all you want to do is scan your face as a model and have it react to your expressions in front of the RealSense camera, there are certainly applications for that, which include:

FACESHIFT

http://faceshift.com/studio/2015.2/ What is faceshift studio? — faceshift studio 2015.2 2015.2.00 documentation

URANIOM

http://www.uraniom.co/ Uraniom - Universal 3D avatar for VR/AR apps and video games

If you want to record live mo-capped animation clips that could then be stored in a file and dropped into a project such as a game with little editing, then I haven't yet seen an example of that.

In my own project, my approach was to create a full-body avatar model, give it joints and import it into the Unity game engine. You can then drop pre-made control scripts from the Unity Toolkit packaged with the SDK into the joints to control the model with your face and hands.

The control scripts that come with the Toolkit (such as 'TrackingAction' aren't hugely stable and precise though, and have a tendency to make a controlled body part "flip out" when you try to move the part into an extreme angle. For this reason, I spent a couple of years building my own custom animation engine for RealSense in Unity that lets you forward and reverse through animation clip timelines with the camera in real-time. Most of that time was experimentation though - you could build a similar system in less than a week now that the procedure for doing so has been finalized.

Here's a video of my tech in action from last July. The avatar in this video is pretty ancient in comparison to its current advanced standard, so don't take this as being representative of the final product. Live facial animation starts at 1 minute in.

https://www.youtube.com/watch?v=HspFO87NHeY Tech trailer # 1 for 'My Father's Face', a RealSense-powered PC game - YouTube

0 Kudos
ggera
Novice
542 Views

Thanks for your quick response--I am looking for live, markerless facial motion /trying to avoid the Unity engine...Faceshift is the best solution ever but Faceshift has been discontinued since 2015 (purchased by Apple). Again, Ni-Mate was a great substitute but no longer working with the Intel sr300 (see my notes in post)

If you candirect me to a download for Faceshift i will be eternally grateful.

Thanks again

0 Kudos
MartyG
Honored Contributor III
542 Views

There doesn't seem to be a reputable source remaining for the Faceshift installation files - it looks like Apple scoured it from the internet after their purchase. That is generally the modus operandi for company purchases where you are buying the tech for your own use - to shut off supply of a product to new licencees as soon as purchase of a company is complete.

0 Kudos
Reply