Adding hand control in Unity application using SR300

[MMoha17]

Hi,

I have built a ping pong game and would like to use the Hand gestures to control the paddles. However I am unable to do so. May I know how I can allow the SR300 to detect my hand gestures to control the paddles?

I used this project as a reference (https://software.intel.com/en-us/articles/game-development-with-unity-and-intel-realsense-3d-camera RealSense - Game Development with Unity* and Intel® RealSense™ Camera | Intel® Software ) but cannot get the paddles to follow my hand gestures. I am not sure where I am supposed to attach the Paddles.cs file.

Thank you.

[MartyG]

Whilst you can write your own scripts for controlling objects with the camera like the author of the guide chose to do, it is not actually necessary to do so. The RealSense SDK comes with a program called the Unity Toolkit, It is in the RSSDK > Framework > Unity folder of the SDK. If you open your project in Unity and then run the toolkit program, it will import numerous pre-made RealSense Unity scripts into your project.

One of these is called TrackingAction. You can drag and drop it into your paddle object and then set it up to follow your hand movements in real-time. You do not even need to use special hand gestures - the paddle will just follow your hand's position.

It should be mentioned though that you need to be using the R2 version of the RealSense SDK for the Toolkit to work. The Toolkit does not work in the latest R3 SDK that is the current release.

If you need further help with this, please feel free to ask!

[MMoha17]

Hi MartyG,

Thanks for this reply. I am now able to control the paddles using hand gestures. However, I do not know why the ball just goes through the paddles and not bounce off. Similarly, the ball goes through the table (ground) and is able to go under the ground.

Thank you.

[MartyG]

Firstly, you should ensure that your ball, paddle and floor have 'collider' fields attached to them that help to stop pass-through. If you created your objects from basic Unity objects such as cubes and spheres then they should already have a collider field installed in them by default.

If you created the objects by another means and they do not have colliders - look in the Inspector panel's details for an object to see whether they have one or not -, then you can add them manually by selecting the object and then going to the 'Component > Physics > Box Collider option (for paddle and floor, to put a square collider around them), and the 'Component > Physics > Sphere Collider' option (for the ball, to put a spherical collider around it).

If your objects have colliders and are still passing through, try inserting a 'Rigidbody' physics component into the ball and paddle - do this by going to 'Component > Physics > Rigidbody. A Rigidbody gives objects better physics and improved ability to know when a collision has occurred.

You do not need to give the floor a Rigidbody, since Rigidbodied objects can collide solidly with static objects that do not have a Rigidbody. In my own game project, I have a Rigidbodied avatar that stops falling when the box colliders around its feet touch the non-rigidbodoed floor.

[MMoha17]

Hi MartyG,

Thank you for your reply. The Ping Pong game now works perfectly using hand gestures.

[MartyG]

Great, glad I could help!

[MMoha17]

Hi MartyG,

Similarly I tried to implement a rolling ball game similar to this (https://www.youtube.com/watch?v=W_fAidYRGzs Unity 5 - Roll a Ball game - 1 of 8: Setting up the Game - Unity Official Tutorials - YouTube ). I want to control the ball using Hand gestures. The game works fine if i use the keyboard controls. The ball is able to hit the walls and not go through.

However, when I added in the Tracking Action, the ball is going through the walls. The ball has a Rigidbody feature added. I have added a picture of the problem below. May I know how I can prevent the ball from going through the wall? Please help.

Thank you.

[MartyG]

A TrackingAction overrides the settings and influence of a Rigidbody when tracking is active. For example, if you have a gravity-enabled Rigidbody and a ball on the ground, when your hand is detected by the camera the TrackingAction blocks the gravity, enabling you to lift the ball up without it falling back. When you take your hand outside of the camera's view then hand tracking is "lost" and the Rigidbody re-asserts its influence, causing the ball to drop back to the ground under the force of gravity.

The first thing I would suggest that you try is to tick a box called 'Effect Physics' at the bottom of the TrackingAction. This forces the TrackingAction to give priority to the Rigidbody for its physics settings instead of trying to impose its own on the object.

Also, check to make sure that your walls have Colliders in them.

[MMoha17]

Hi MartyG,

Thank you for the reply. The Effect Physics is already ticked for the player element and the walls have Box Colliders activated. I have shown the 2 settings in the picture below. I am still unsure as to why this happens.

Thank you.

[MartyG]

The Virtual World Box settings in your TrackingAction are curious, if the aim is to just roll a ball inside the confines of the wall, which will stop your ball from moving when the wall is hit.

The Box Center values (0, 0.5, 0) are likely correct. I can see what the creator of this example was trying to do. The Virtual World Box draws a border around your object that determines how far it can travel from its current position before it is stopped from moving further, kind of like putting a fence around it. The values equate to 1 = 1 meter in Unity distance. Using 0, 0, 0 as the Box Center values puts the object at the center of the box. 0, 0.5, 0 though moves the restriction border 0.5 meters distance from your object. What this achieves in this case is to put the bottom of the box directly under the floor, preventing the ball from moving down through the floor - it can only travel upwards and horizontally.

Actually, this presents a potential solution to your wall pass-through problem. In the Virtual World Box Dimensions values, edit the values that define the size of the box (currently 27, 33, -39) until the red box's edges fit tightly against the insides of your walls. Then this should prevent the ball from being able to pass through the walls, as it can only travel from its start position to the edge of the defined Virtual World Box before it is stopped from going further.

You should take out the keyboard controls though (or at least not use them) if you use this method. If you use them to move the ball physically from its start position then the Virtual World Box will move with it, taking its boundaries outside of the wall and allowing pass-through. Instead, allow the TrackingAction to handle the moving of the ball within the confines of the Virtual World Box.

[MMoha17]

Hi MartyG,

Thank you. I am now able to restrict the ball within the 4 walls.

However, the ball is only moving in the x direction. I need it to move in both the x and z direction. Does it have anything to do with constraints? I have tried to play around with the x,y, and z constraint settings in the Rigidbody and TrackingAction features a number of times but to no avail.

Thank you.

[MartyG]

If you have Effect Physics ticked then the object should be using the Rigidbody's defined constraints instead of the TrackingAction's. So if you have Position constraints ticked in the Rigidbody then that should lock some of the travel directions.

What I would do is to (a) un-tick all of the constraints in the TrackingAction so that they are all unlocked; and (b) place a tick in all of the Rigidbody's Rotation constraints, and have one of the Rigidbody's Position constraints ticked at a time to lock it until you get the results that you need.

In an ideal world, you would just tick the Y position constraint in the Rigidbody to stop up-down movement. However, directional axes in a Unity project do not always correspond to real-world expectations, so having Y unlocked can cause movement on the horizontal X plane. Experimentation is the key!

[MMoha17]

Hi MartyG,

I cannot seem to be able to solve the issue. Is HandTracking the only way to move the ball using gestures?

Thank you.

[MartyG]

If you cannot achieve the results you want by using the Rigidbody constraints, I recommend unticking them all and using only the TrackingAction constraints. This is because sometimes, a TrackingAction object that uses Effect Physics cannot produce the same range of movements that a TrackingAction that uses its own constraints can. And your object should still have collision physics thanks to its collider and Rigidbody.

Edit: it's important for me to mention that if you stop using the Rigidbody constraints, you MUST un-tick the Effect Physics box at the base of the TrackingAction so that the TrackingAction's constraints can be used instead.

If you really cannot use a TrackingAction, you could use SendMessageAction. This allows you to trigger a script (such as an object movement script) when a particular gesture such as Thumbs Up or Hand Closed is detected.