Problem with landmark points

[ACarv2]

Greetings

I'm currently working on a project for the University and I'm using the F200 realsense camera with the r3 2016, soon upgrading the camera to the R300, working on visual studio 2013 and programming in C# .

My project consists in using the camera with the face tracking software to detect a yawn but we are having trouble using the software. Visual studio is giving us an error with the Intel namespace not being found.

We found a forum (https://communities.intel.com/thread/113038 https://communities.intel.com/thread/113038) where it was said the namespace error was currently being worked on but we didn't find any updates since then (7th of April).We also have problems while trying to access to the face's points since the documentation says to do

"LandmarksData ldata=face.Landmarks; //where face comes from Intel.RealSense.Face.Face face

FaceData.LandmarkPoint[] points = ldata.Points;"but there's no variable called Landmarks on the face object. After some debugging, we found we were using"PXCMFaceData.Face = faceOutput.QueryFaceByIndex(0);"instead of"Intel.realsense.Face.Face variableFace =faceOutput.QueryFaceByIndex(0);"

Is there a specific version/sdk that uses the "Intel" lib? If so how can we have access to it?

Best regards

[idata]

Hello André_Filipe,

Thank you for your interest in Intel® RealSense™ Technology.The issue related with the namespace error is currently under analysis, that's why there hasn't been an official announcement.Could you please share further information regarding your project? What is the documentation you have been using as reference?I'll be waiting for your response.Regards,Andres V.

[ACarv2]

Hello Andres,

First of all, thank you for the quick reply.

Our project is based on detecting if a person (driver of a car) is sleepy in a way that it can interfere with his/her driving. We also want to see where the person is looking at (for a certain amout of time) and determine if it's related with driving or not. If not, we will alert him/her to pay attention to the road.

We were thinking about doing the detections based on the landmarks on the face of the driver.

We are using the R3 2016 documentation (https://software.intel.com/sites/landingpage/realsense/camera-sdk/v2016r3/documentation/html/index.html?doc_face_face_landmark_data.html Intel® RealSense™ SDK 2016 R3 Documentation).

We program in JAVA and when we started the project, we looked into the R2 2016 SDK, however, we couldn't find any links to download it. So, we decided to use the R3 in C# (although we have never programmed in it before). With this problem of the "Intel" namespace, we might start to use the R2 2016 SDK, where it seems to be a different way of getting the landmark points, with the "face" being from the object "PXCMFaceData".

If you could provide a link for download the R2 2016 SDK, we would be very thankful.

Best regards,

André C.

[MartyG]

I have some knowledge of the kind of tiredness analysis system that you are developing. It is gaining traction particularly in trucking and in construction vehicle operation.

You can download the 2016 R2 from the following link, which launches the 1.8 gb download in your browser immediately.

http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9078/intel_rs_sdk_offline_package_10.0.26.0396.exe http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/9078/intel_rs_sdk_offline_package_10.0.26.0396.exe

Please be aware that web functionality is being removed from RealSense SDKs due to potential security vulnerabilities. As far as I am aware, it should still function in the R2 though for as long as that SDK is available. If you are likely to need that SDK and Java functionality for the long term, I recommend burning a copy of the above install file to a backup dvd. You could download the file to your computer without launching it by right clicking on the above link and selecting Save Target As / Save Link As. Continued use of the R2 SDK with the known web vulnerability potential will of course be at your own risk.

[idata]

Hello André_Filipe,

Please check the following documentation that explains the current state of the Face tracking and landmark detection features: http://registrationcenter-download.intel.com/akdlm/irc_nas/9560/release_notes_RealSense_SDK_2016_Face_Detection_v11.0.pdf http://registrationcenter-download.intel.com/akdlm/irc_nas/9560/release_notes_RealSense_SDK_2016_Face_Detection_v11.0.pdf.

"Alpha Features" imply that the feature is in certain maturity level where the reliability is not guaranteed. There are known issues, but the corresponding fix may not be planned.

Have a nice day.

Regards,

Andres V.

[ACarv2]

Hi,

I downloaded the R2 2016 SDK software you provided.

Now I'm having trouble with the face detection.

From what I've been testing, I gathered that it's not possible to have the face tracking module and the stream of the camera on the same sense manager instance.

Here's some part of the code I'm using:

public void SimplePipeline()

{

PXCMSenseManager sm = PXCMSenseManager.CreateInstance();

if (sm == null) throw new Exception("Failed to create sense manager");

if (sm.captureManager == null) throw new Exception("Capture manager does not exist");

bool instantMode = false;

bool configureStream = true;

disconnected = false;

/* Set Source & Profile Index */

if (this.form.GetRecordState()) sm.captureManager.SetFileName(this.form.GetFileName(), true);

else if (this.form.GetPlaybackState())

{

sm.captureManager.SetFileName(this.form.GetFileName(), false);

configureStream = false;

}

if (configureStream) sm.captureManager.FilterByDeviceInfo(form.GetCheckedDeviceInfo());

/* Set Module */

sm.EnableTracker(form.GetCheckedModule());

/* Initialization */

form.UpdateStatus("Init Started");

sm.EnableStream(PXCMCapture.StreamType.STREAM_TYPE_COLOR, 640, 480);

//Enable the face module

sm.EnableFace();

faceModule = sm.QueryFace();

faceConfig = faceModule.CreateActiveConfiguration();

if (sm.Init() >= pxcmStatus.PXCM_STATUS_NO_ERROR)

{

form.UpdateStatus("Streaming");

form.timer = new FPSTimer(form);

PXCMCaptureManager captureManager = sm.QueryCaptureManager();

if (captureManager == null) throw new Exception("Failed to query capture manager");

PXCMCapture.Device device = captureManager.QueryDevice();

if (device != null && !this.form.GetPlaybackState()) device.SetDepthConfidenceThreshold(7);

if (sm.captureManager == null) throw new Exception("Failed to query capture manager");

if (sm.captureManager.QueryDevice() == null) throw new Exception("Invalid device");

// Configure for 3D face tracking

faceConfig.SetTrackingMode(PXCMFaceConfiguration.TrackingModeType.FACE_MODE_COLOR_PLUS_DEPTH);

// Known issue: Pose isEnabled must be set to false for R200 face tracking to work correctly

faceConfig.pose.isEnabled = true;

// Track faces based on their appearance in the scene

faceConfig.strategy = PXCMFaceConfiguration.TrackingStrategyType.STRATEGY_APPEARANCE_TIME;

// Set the module to track four faces in this example

faceConfig.detection.maxTrackedFaces = 50;

// Enable alert monitoring and subscribe to alert event hander

faceConfig.EnableAllAlerts();

faceConfig.SubscribeAlert(FaceAlertHandler);

// Apply changes and initialize

faceConfig.ApplyChanges();

faceData = faceModule.CreateOutput();

while (!form.stop)

{

pxcmStatus acquireStatus = sm.AcquireFrame(true);

if (acquireStatus < pxcmStatus.PXCM_STATUS_NO_ERROR) break;

if (!DisplayDeviceConnection(!sm.IsConnected()))

{

/* Display Results */

PXCMCapture.Sample sample = sm.QuerySample();

Bitmap colorBitmap;

PXCMImage.ImageData colorData;

sample.color.AcquireAccess(PXCMImage.Access.ACCESS_READ, PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32, out colorData);

colorBitmap = colorData.ToBitmap(0, sample.color.info.width, sample.color.info.height);

if (sample == null) continue;

if (sample.color == null) continue;

DisplayPicture(sample.color);

// Acquire face data

if (faceData != null)

{

faceData.Update();

facesDetected = faceData.QueryNumberOfDetectedFaces(); // Always 0...

if (facesDetected > 0)

{

// Get the first face detected (index 0)

PXCMFaceData.Face face = faceData.QueryFaceByIndex(0);

// Retrieve face location data

PXCMFaceData.DetectionData faceDetectionData = face.QueryDetection();

}

}

}

sm.ReleaseFrame();

}

faceConfig.Dispose();

faceModule.Dispose();

}

else form.UpdateStatus("Init Failed");

sm.Close();

sm.Dispose();

}

When I try to run both the face tracking and the camera stream on the same "sm" instance, the sm.Init() gives the following state: "PXCM_STATUS_ITEM_UNAVAILABLE".

How can we have both modules working at the same time? I did try to run in different Sense Manager instances and it worked, however, it wasn't able to detect any faces that way.

How can we detect the faces as well?

Thank you,

André

[ACarv2]

Hello,

Here are some updates.

I did some more research and I finally solved the problems I described here before (having the face tracking module and the stream of the camera working at the same time and the landmark points).

Thank you for your time and help,

André

[idata]

Hello André,

I'm glad to hear that you could solve the issue.Thank you for sharing your experience with the community, we really appreciate it.Have a nice day.Regards,Andres V.