Ball controlled from hand gestures via SR300

[MMoha17]

Hi MartyG,

Basically I am trying to control a Sphero SPRK+ robot with my Realsense camera. I am having difficulty connecting the sphero to my pc as the manufacturers of Sphero have informed that Sphero SPRK+ does not support Windows PCs. Hence I was thinking about using Android or iOS devices as a middle man to control the Sphero SPRK+.

Hence in my other post ( ), I was trying to build a software version of the game. But I cannot control the ball nicely. I want to control it's direction and speed just by using gestures detected.

Moving hand forward = faster

Close fingers= stop

Spread fingers = start moving

and so on.

Thank you.

[MartyG]

What you are describing should be quite simple to achieve with a Unity TrackingAction. What you need to do is define Start and Stop "rules" for the TrackingAction, instead of using the default Hand Detected activation setting.

(a) Right-click on the 'Hand Detected' setting under the Start Event label to bring up its menu and select the 'Remove' option to remove it from the TrackingAction.

(b) Left-click on the 'Add' button beside the Start Event label and select the 'Gesture Detected' option.

(c) In the new set of options that has appeared, expand open the Gesture Detected option using the small arrow beside it and leave the setting on the default 'Spread Fingers' setting. With this setting, the TrackingAction will only activate when the camera sees the Spread Fingers gesture.

(d) Go to the 'Stop Event' option in the middle of the TrackingAction. Usually, this is set to 'Hand Lost' by default. Right-click on the 'Hand Lost' option and remove it. Then left-click on the Add button beside the Stop Event label and set it to 'Hand Closed'. Now the TrackingAction will stop tracking and cease your object's movement when the camera sees the Hand Closed gesture.

If you need to reduce the movement speed of your object, go to the 'Smoothing Factor' of the TrackingAction. Ensure that it is set to the 'Weighted' type of smoothing, and give if a higher value than the default of '0'. '5' is a good test value for a quick but stable movement, '10' for a slow movement, and a maximum of '19' for very slow movement (objects tend to stop moving once their weighting is 20 or more on the Weighted setting).

[MMoha17]

Hi MartyG,

I can now control it smoothly. Thank you. However there is a small issue. When the hand is closed, the ball should stop moving completely (freeze at it's last position). But it continues to roll even though the hands are closed.

May I know how I can make the ball freeze at the position the hands closed action is activated and continue from that position when the gesture detected action is activated?

Thank you.

[MartyG]

My guess would be that when the TrackingAction stops tracking when the hand is closed, the Rigidbody physics component is taking over and making the ball roll. This is similar to a setup I had in my project, with a TrackingAction that lifted a block up in the air and a gravity-enabled Rigidbody. Whenever tracking was lost, the gravity influence of the Rigidbody - which is usually disabled during TrackingAction use - kicked in and made the block fall down.

If you have gravity ticked on your Rigidbody, I would un-tick it. Then when tracking ceases when you close the hand, the ball shouldn't roll. This is because without gravity, the Rigidbody physics can't accelerate the ball and it should stay fixed on whatever spot you left it on when the hand was closed.

[MMoha17]

Hi MartyG,

Thank you very much. That solved the problem. But I still have 1 small issue. Let's say the ball stops completely once I have initiated the stop action and the ball is said to be at position X. Then when I move the hand to another position and resume the gameplay, the ball does not continue from position X. It starts from a different position. I have tried to experiment but cannot seem to solve the issue.

Thank you.

[MartyG]

I spent about 2 years trying continuously without luck to solve that trait of TrackingAction with experiments and editing of its code. It is built into its code to update the object's position to the current position of the hand when tracking of the hand is re-acquired. I ended up writing my own custom control system last year that takes the angle of the object and converts it into a movement animation generated by an animation clip. By doing it this way, you can control an animation in real-time by using the TrackingAction to wind backwards and forwards through an animation sequence.

So if you create, for example, an animation of an arm lifting from fully dropped to fully raised above the head, moving the real life hand upward winds forward through the clip's timeline towards its end, lifting the arm object, whilst lowering the arm rewinds the clip back towards its beginning, making the arm object drop down.

It's an extremely precise, smooth and stable system. It also solves a couple of other drawbacks of TrackingAction (objects moving when you turn the wrist even if you haven't set wrist detection, and the object flipping-out if you do too extreme a movement in a particular direction). Unfortunately I haven't got around to writing a how-to guide to the technique yet.

[MMoha17]

Hi MartyG,

Thank you for sharing your expertise in Unity and Realsense.

[MartyG]

You're welcome!

I was just answering your post about Unity volume but the post got deleted, so I'll put the reply here!

You should not need an Audio Listener in your canvas. You only need one listener in each Unity scene. There is one inside the main camera by default. So unless you un-tick the Audio Listener option in the camera's settings in the Inspector then you will get the message in the debug console that "There are 2 audio listeners in this scene".

I do not see a problem with putting your Audio Source inside the canvas if your canvas is permanently active. Otherwise, if the Canvas is deactivated, the Audio Source inside it should become deactivated too.

I think I see what the problem in your code may be. When you create a float, its value is '0' by default unless you give it another value, or connect it to something that will change the value. So what the above code is saying is that the float 'controller' will have the value '0'. And because 'Audiolistenervolume' uses the 'controller' float for its volume value, then the volume will be '0' (i.e silent) when the function public void volumeControl is activated.

[MMoha17]

Hi MartyG,

Yes I deleted it because I forgot this forum is supposed to be for Intel Realsense.

I tried adjusting the script. I was able to reduce the volume to a specific value because I hardcoded the value. I think a for loop needs to be created for the variable newVolume.

float newVolume = 0.4f;

public void volumeControl(float controller=0.0f)

{

AudioListener.volume = controller;

AudioListener.volume = newVolume;

}

[MartyG]

Yes, you need to use a For or a Foreach. I use Foreach in my own project's volume analysis system. It's not exactly what you are trying to do - in my system, it finds out what the volume is instead of trying to change the volume - but it might give you some useful ideas about how to proceed.

STEP 1

As well as your newVolume float, you need to put a definition for an Array type variable in the header of your script. Array type variables have a pair of square brackets (not the squiggly ones) after the variable name.

private int SampleArrayLength = 2;

private float Data[];

STEP 2

In your Start() section:

Data = new float[SampleArrayLength];

STEP 3

In the Update() section: (this script is going to assume that the activation code for your AudioSource is in the canvas at the same location as your volume script).

GetComponent().clip.GetData (Data, GetComponent().timeSamples);

foreach (var sample in Data) {

newVolume = math.Abs (sample);

}

STEP 4

The above foreach code will look at the volume of the audio clip currently playing in the Audio Source and constantly update newVolume to a volume value between 0 and 1.

In my project, I use the volume to move the mouth of the player character. If the audio input is silent then volume = 0 and the mouth is closed. The louder the volume, the wider the mouth opens. In the silent space between words, the mouth automatically closes as volume nears 0. In this way, the mouth lip-syncs to the audio in real-time without having to be pre-programmed.

https://www.youtube.com/watch?v=VBp_9_mVpUw "My Father's Face" Tech Trailer 4 - Lip Synching And Facial Expression From An MP3 - YouTube

[MMoha17]

Hi MartyG,

My button controller script has no start and update section. It's all public void declarations.

[MartyG]

You can define Start and update sections in the following way:

public void Start() {

public void Update() {

public void Start() should be the first function in your script (below the Public Class instruction), and public void Update() below it.

Remember to define the Start and update names as written, with capital S and U, or it will not work.

This is probably a good opportunity to give readers a basic lesson about functions. The Start() function is a "fire once" type of function. It typically contains setup instructions for your script, including the location of objects so that the script can find them in your project. Once it has completed processing the list of instructions inside it, it does not run that section again.

Update() functions are infinitely looping functions that run the list of instructions inside them over and over until the function is stopped. They are good for processing information that is constantly changing and so needs to be re-checked every second, such as the current volume of an audio source or the position of an object.

If you create a function with your own user-defined name then these are treated as fire-once functions like Start. You can run a fire-once function again though at any time though by putting a reference to its name in another part of the script, and the script then jumps to that function when it sees the name. For example, if you had a function called MyVolume(), then somewhere else in the script, you would use this line to make the script jump to MyVolume:

MyVolume();

So basically, the function name followed by ();

[MMoha17]

Hi MartyG,

Thanks for the important definition about Start and Update functions. I have declared some variables according to the steps you mentioned, however I am getting 2 errors. I have tried to rectify them. Thank you.

[MartyG]

You have two exact copies of the Update() block in your script. You can only have one each of a particular function name, so you should delete one of the Update() blocks.

In the 'Data' error, the word 'float' is missing from it. It should be:

Data = new float[SampleArrayLength];

Also, where you have put math.abs, it should be:

Mathf.Abs (sample);

That's Math with a capital M, and a small f on the end.

[MMoha17]

Hi MartyG,

Thank you. It all works now. Thank you very much.

[MMoha17]

Hi MartyG,

The tracking is working perfectly. However, I would like to know if it is possible to get the ball to continue rolling if the hand is static and slightly forward. And as the hand is moved back and kept static, the ball slows down subsequently (like a car reducing it's speed when the brake is pressed) and starts to move backwards. Is it in the Unity Realsense package or a script needs to be written separately for that as a Player Controller script? I would like to know how I can get started if a script needs to be written separately.

Another question is how do I create a hole on my ground such that the ball drops into the hole? Thanks for your answers to my Unity questions.

Thank you.

[MartyG]

You're very welcome, I'm glad to be able to help.

I understand what you are asking. You want the ball to keep rolling forwards if the hand is forward, and slow down and then reverse if the hand is drawn back. I use something similar in my own game. Here is the steps that I recommend:

(1) I remember in another post that you said you had built a version of my camera-controlled menu system. I believe that this involves a rocker that moves up and down when the hand moves and touches a trigger block. You can adapt this for the accelerate / brake and reverse system.

Make a copy of the rocker and its trigger blocks above and below it.

(2) Write a 'logic script' to place inside your ball. This script is very simple, with no functions. It would just be a pair of true or false 'bool' statements under the 'Using UnityEngine' header of the script. For example:

public bool Accelerate;

public bool reverse;

(3) Create BoxTrigger trigger scripts for the rocker's trigger blocks, like the ones in the guide. Except in these scripts, put in an OnTriggerEnter function:

In the header of the script, write a statement to tell Unity the name of the ball object that the bool script is stored in. For example:

public GameObject ball;

(4) In the Start() function of the script - which should go above the OnTriggerEnter() function - write something like:

ball = GameObject.Find("ball");

This tells Unity to look for the object in the speech marks that has the name ball.

(4) In the OnTriggerEnter() function, if it is the upper trigger block then put:

ball.GetComponent().Accelerate = true;

ball.GetComponent().Decelerate = false;

This tells Unity to use the location variable 'ball' - which is shorthand for GameObject.Find("ball"); - and inside your ball, look inside your logic script and look for logic variables called Accelerate and Decelerate.

Because you pushed your hand forward to make the rocker rise up and touch the upper block, the script tells Unity to set the logic statuses so that when the hand is forward, Accelerate is true and Decelerate is false (i.e you want to go forwards, not backwards).

In the lower trigger block, use the opposite statements to say that you want to go backwards, not forwards.

ball.GetComponent().Accelerate = false;

ball.GetComponent().Decelerate = true;

(5) Now you need to write a motion script for the ball that will make it travel forwards or backwards depending on whether the Accelerate or Decelerate status is currently active. I've had a go at writing a movement script for you.

private float WalkPace;

private Rigidbody rb;

public void Start()

{

rb = GetComponent();

}

public void FixedUpdate()

{

// The FixedUpdate function is used with scripts involving movement with Rigidbodies

if (GetComponent ().Accelerate == true

&& GetComponent ().Deccelerate == false ) {

Roll = 2.5f;

}

if (GetComponent ().Accelerate == false

&& GetComponent ().Decelerate == true ) {

Roll = -2.5f;

}

rb.MovePosition(transform.position + (transform.up / Roll));

}

}

***********************

What this script does is tell the Rigidbody in the ball to keep moving 2.5 m forward after your hand has pushed forward and made the Accelerate status true.

When you move your hand back and the Decelerate status is activated, -2.5 m will be added to the ball's movement. If you add a minus value to an object then it travels backwards, not forwards.

(6) In each of your trigger scripts, also put this OnTriggerExit function so that when you move your hand so it is not pushed forward or backward (i.e the rocker is between the two blocks), Accelerate and Decelerate will be false and ball motion will stop.

OnTriggerExit()

{

GetComponent ().Accelerate == false;

GetComponent ().Decelerate == false;

}

***************

In regard to your final question about dropping through the hole, construct a floor for your ball arena that is made from four separate pieces that are arranged so there is a hole in the middle of the floor space. Then tick the 'Gravity' option on the ball's Rigidbody. Then when it rolls over the hole, it will drop down through it.

[MMoha17]

Hi MartyG,

Thank you for this detailed explanation.