Skeletal

In NUI Skeletal Tracking mode, the user's hands operate as the cursors.

Cursors

Hovering - While the cursor is in-range and not engaged, the cursors will appear as open hands representing the current location the user is hovering.

Over interactive object - Hovering directly over a dynamic object.

Interacting - Selecting, dragging, etc.

Out-of-range - If at any time the user's hands exit the physical interaction zone, dots will represent the cursors. Though the user cannot interact with the system when out of range, the dots will represent the location of the users hands on screen.

Gestures

NUI Skeletal Tracking mode also makes use of Gestures.

Waving Gesture

Users can wave their hand to start interaction. They can use their hands or body to interact with 2d and 3D objects. Commands vary according to settings.

Some actions that can be performed:

Rotating or orbiting
Panning and moving forward
Zooming

Pick Actions Using Gestures

Pick actions can be triggered using Gesture Commands (in Skeletal Tracking Only).

First, enable IcoNui use for the open display by setting Preferences > GwxRuntimeOptions > EnableNuiFeature to True.

Then, set IcoNui to Skeletal Tracking mode.

Next, select the pick action of an object (e.g.: a button). The pick action will then have an option under Execution Trigger – Natural UI named VoiceCommand.

All that is needed is to select the type of gesture to associate with the pick action. Gesture types include:

RightHandSwipe - gesture is recognized when right hand moved from right to left.
LeftHandSwipe - gesture is recognized when left hand moves from left to right.

Voice Commanding

Pre-commands

In Skeletal Mode, commands can also be issued using voice commands. The user can issue voice commands in one of two ways:

The user can say the pre-command word. By default, the pre-command word is set to "Computer", but can be changed in the Natural User Interface Settings of the NUI Ribbon in GraphWorX64. Stating the pre-command word will give the user a "Listening" notification on the bottom of the on-screen interaction assistant. Once the "Listening" notification is given, the user may issue voice commands.
The user can also issue voice commands by using a physical gesture. By putting a hand close to the mouth, the user should receive a "Listening" notification on the bottom of the on-screen interaction assistant. Once the "Listening" notification is given, the user may issue voice commands.

Giving a Voice Pre-command

Pick Actions Using Voice Commands

Pick actions can be triggered using voice commands (in Skeletal Tracking only).

First, enable IcoNui use for the open display by setting Preferences > GwxRuntimeOptions > EnableNuiFeature to True.

Then, set IcoNui to Skeletal Tracking mode.

Next, select the pick action of an object (e.g.: a button). The pick action will then have an option under Execution Trigger – Natural UI named VoiceCommand.

All that is needed is to type the voice command you want associated with the pick action in that field. Notice that this voice command can also be a language alias, in the same way as with the pre-command in the IcoNui Speech recognition settings. [NOTE: Voice command text can only contain alphabetical characters and spaces.]

Voice Interactions On-Screen GUI

Speech Recognition Status:

Listening… - User can speak.

Listening Cancelled – The recognizer has not heard any spoken word in 5 seconds timeout.

Command Recognized – Spoken word has been recognized as a command.

Speech Recognition Problem Message:

Too Loud, Too Fast, Too Noisy, Too Slow, No Signal: Problem with speech pronunciation.

Audio Input Has Stopped – Input device has been disconnected. After the device is connected again, the NUI must be restarted.