Computer Graphics for Virtual  
and Augmented Reality  
Lecture 04 Interaction Methods for  
Virtual Reality in Unity  
Edirlei Soares de Lima  
Interaction Methods for Virtual Reality  
Interaction with a virtual world is a key ingredient of a VR  
Selection and Manipulation:  
Selection methods allow users to select one or more objects from a  
Manipulation methods allow users to modifying object properties  
Position, orientation, scale, shape, color, texture, behavior, etc.  
Defines how users move from place to place within the virtual  
Selection Techniques  
Selection Techniques  
Simple Virtual Hand:  
Direct mapping between physical and virtual hands  
Object can be selected by “touching” with the virtual hand  
Limitation: user can only select objects that can be reached by hand  
Virtual Hand in Unity  
Adding virtual hands models:  
Create an XR Rig in the scene: GameObject -> XR ->  
XR Origin;  
3D hand model:  
hand-models/ (Maya files) (FBX files)  
Add the virtual hands as child objects of Left- and  
Right-Hand Controllers in the XR Rig.  
Adjust the rotation and position of hands to match the  
position orientation of the real hands (must be tested in VR).  
Virtual Hand in Unity  
Grab Animation:  
Create “Idle”, “Grab” and “Activate” animations  
using Unity’s Animation System.  
Use the same hand model to reuse the same  
animations on both left and right hands.  
Create an animator and state machine to  
control the animation transitions.  
Use boolean parameters for the transitions.  
Virtual Hand in Unity  
public class HandAnimation : MonoBehaviour{  
public InputActionProperty grabAction;  
public string grabAnimationParameter;  
public InputActionProperty activateAction;  
public string activateAnimationParameter;  
public Animator handAnimator;  
private bool grabbing;  
private bool activating;  
private void Start(){  
grabbing = false;  
activating = false;  
private void Update(){  
if (handAnimator)  
float grabButton = grabAction.action.ReadValue<float>();  
float activateButton = activateAction.action.ReadValue<float>();  
Virtual Hand in Unity  
if (grabButton > 0)  
if (!grabbing)  
grabbing = true;  
handAnimator.SetBool(grabAnimationParameter, grabbing);  
else if (grabbing)  
grabbing = false;  
handAnimator.SetBool(grabAnimationParameter, grabbing);  
if (activateButton > 0)  
if (!activating)  
activating = true;  
handAnimator.SetBool(activateAnimationParameter, activating);  
Virtual Hand in Unity  
else if (activating){  
activating = false;  
handAnimator.SetBool(activateAnimationParameter, activating);  
Direct Hand Interaction in Unity  
XR Interaction Toolkit implementation:  
Remove the default XR Ray Interactor  
components from the hand controllers.  
Add a XR Direct Interactor and Collider  
component to the hand controllers.  
The collider must be set to trigger.  
Create the interactive objects to be activated  
by hand interaction (buttons and grabbable  
Add a collider and rigid body to the objects.  
Direct Hand Interaction in Unity  
The grabbable cube must have XR  
Grab Interactable Component.  
The buttons will use a custom script  
based on the XRBaseInteractable.  
Direct Hand Interaction in Unity  
public class InteractiveButton : XRBaseInteractable  
private bool buttonActivated;  
private void Start(){  
buttonActivated = false;  
GetComponent<Renderer>().material.color = Color.white;  
private void ActivateButton(ActivateEventArgs arg0){  
buttonActivated = !buttonActivated;  
if (buttonActivated){  
GetComponent<Renderer>().material.color =;  
GetComponent<Renderer>().material.color = Color.white;  
Selection Techniques  
Simulates a “laser pointer” attached to the virtual hand  
First object intersected by the ray can be selected  
Perform well for remote selection  
Ray Casting Interaction in Unity  
XR Interaction Toolkit implementation:  
Add a XR Ray Interactor component to the hand  
Add a XR Interactor Visual Line component to  
the hand controllers.  
Optional component to draw the ray cast line.  
Create the interactive objects to be activated by  
hand interaction (buttons and grabbable cube).  
Add a collider and rigid body to the objects.  
Use the XR Grab Interactable Component in the Cube.  
The buttons use the same script created before.  
Selection Techniques  
Image-plane technique (2D)  
Occlude/cover object with the selector object (e.g. hand or finger)  
Nearest object along a ray from the eye through selector object can be  
Selection Techniques  
Go-Go (Arm-extension):  
Non-linear mapping between physical and virtual hand position  
Can access local and distant objects  
Selection Techniques  
Other common selection techniques:  
3D Bubble Cursor  
A survey of 3D object selection techniques for VR:  
Assignment 1  
) Implement the interaction process to open and close a door  
in Virtual Reality.  
When the door is closed and the user grabs the door handle, the door  
must open.  
When the door is open and the user grabs the door handle, the door  
must close.  
Free 3D door models:  
Manipulation Techniques  
Manipulation Techniques  
Selection: ray-casting  
Manipulate: directly with virtual hand  
Include linear mapping to allow wider range of placement in depth.  
Manipulation Techniques  
A miniature of the world held in the user’s hand  
Miniature objects can be manipulated directly  
Moving miniature objects affects full-scale objects  
Can also be used for navigation  
Manipulation Techniques  
Scaled-World Grab:  
At selection, scale user up (or world down) so that virtual hand is  
actually touching selected object  
Assignment 2  
2) Implement the world-in-miniature manipulation technique.  
The miniature of the world can be presented in front of the user.  
The use must be able to select and manipulate the miniature objects  
Moving miniature objects must affect full-scale objects.  
Hint: create a script to connect the miniature objects with the full-scale objects.  
Changes to the position and orientation of the miniature object must be reflected  
in the connected full-scale object.  
Further Reading  
Sherman, W. R., Craigm A. B. (2003). Understanding Virtual Reality:  
Interface, Application, and Design (1st ed.). Morgan Kaufmann. ISBN: 978-  
Chapter 6: Interacting with the Virtual World