Computer Graphics for Virtual
and Augmented Reality
Lecture 06 – Introduction to
Augmented Reality
Edirlei Soares de Lima
<edirlei.lima@universidadeeuropeia.pt>
What is Augmented Reality?
•
From Reality to Virtual Reality:
Mixed Reality
Real
World
Augmented
Reality (AR)
Virtual
Reality (VR)
Virtual
World
–
Augmented reality combines real and virtual
images.
•
Interactive in real-time
•
Virtual objects are fixed in the space of the real
world.
Augmented Reality
•
Main characteristics:
–
–
–
Combination of real and virtual elements
Real-time updates at interactive speed
Interaction and integration between objects
•
•
Goal of AR:
–
Make users believe that the virtual elements are part of the real world.
Interaction and navigation:
–
Virtual Reality: users interact and navigate using real world metaphors
–
Augmented Reality: users interact and navigate in the real world
Augmented Reality
•
AR uses a feedback loop between human user and computer
system.
–
The user observes the AR display and controls the viewpoint.
–
The system tracks the user’s viewpoint, registers the pose in the real
world with the virtual content, and presents situated visualization.
Registration
Situated
of virtual
content
visualization
Virtual content
SPATIAL MODEL
User input
and camera
movement
Pose
tracking
Real world model
Applications of Augmented Reality
•
•
Visualize building projects
Interior design and decoration
Applications of Augmented Reality
•
•
Manufacturing and maintenance
Logistics operations
Applications of Augmented Reality
•
•
Discrepancy Analysis
Infrastructure Inspection
Applications of Augmented Reality
•
•
Advertising
Live language translation
Applications of Augmented Reality
•
•
Sports and news
Tourism
Applications of Augmented Reality
•
•
Medicine
Fashion and dressing
Applications of Augmented Reality
•
•
Parking Assistant
Navigation
Applications of Augmented Reality
•
Games
•
Instructions and guides
Strong vs. Weak AR
•
•
Weak AR
–
–
–
–
Imprecise tracking
No knowledge of the environment
Limited interactivity
Handheld AR
Strong AR
–
Very accurate tracking
–
Seamless integration into the real
world
–
–
Natural interaction
Head mounted AR
Computer Graphics for AR
•
•
AR requires less computer graphics power
–
Only the virtual objects need to be rendered in real-time
AR requires better calibration and registration
–
Tracking must be precise to guarantee that objects stay the same
places in the real world while the users moves.
•
AR requires a good composition of real and virtual elements
–
A seamless integration of the virtual objects into the real world
requires color and light adjustments.
Technologies for VR Systems
•
•
Display
–
–
–
Optical see-through
• Additive blending
Video see-through
•
Computer calculated blending
Projection-based
•
Multiplicative blending
Tracking
–
–
–
Localization
• Finding the location
Alignment
•
Registering virtual objects to the real world
Interaction
•
Touching and manipulating virtual objects
Display Technologies for VR
See-Through
Displays
Monocular
Binocular
Video
Optical
Video
Optical
See-Through
See-Through
See-Through
See-Through
Single
Camera
Dual
Camera
Monoscopic
Overlays
Stereoscopic
Overlays
Example
Products
Monoscopic
Overlays
Stereoscopic
Overlays
Monoscopic
Overlays
Stereoscopic
Overlays
E.g.: smartphone- or E.g.: Microvision
tablet-based
hand-held AR
Also: Google Glass in
VST mode
Nomad,
DigiLens DL40,
TacEye ST,
E.g.: Vuzix iWear VR920
with Possible, but no E.g.: Canon COASTAR,
E.g.: Microsoft HoloLens,
Epson Moverio BT-200,
E.g.: Trivisio
ARVision
Vuzix M2000AR
iWear CamAR
clear advantage Vuzix Wrap 1200DXAR E.g.: Lumos DK-40 Vuzix STAR 1200XLD
Optical See-Through Display
•
•
Optical see-through head mounted display
The real world is visible
–
Only virtual objects are rendered and displayed
–
Uses an optical element to combine a user’s view
of the real world with computer-generated
images
Generated Virtual Imagery
Pose Sensors
(
potentially stereoscopic)
Real World
Optical
Combiners
Optical See-Through Display
Optical See-Through Display
•
•
Characteristics:
–
Additive optical combination
–
Accommodation issues
•
Focus on both real and virtual objects
–
Typically, poor field-of-view
Examples:
Optical See-Through Display
•
Composition in optical see-through displays:
–
Combination of light from display and real world
–
Does not occlude and cannot show black
+
=
Optical See-Through Display
•
Composition in optical see-through displays:
–
There are some solutions for occlusion
• Kiyokawa et al. An Occlusion-Capable Opt
Co-located Collaboration, 2003.
Optical See-Through Display
•
•
Accommodation problem:
–
The lens projects the display at a
certain distance
–
The user’s eyes must focus on both
real and virtual objects at a distance.
Multi distance displays:
–
Varying lenses
–
Multiple waveguides + different lenses
•
Magic leap: 6 guides = 2 depths
–
Challenges
•
Transparency
•
Transition between depths
Video See-Through Display
•
Real world is shown as video
–
Captures the real world with a video camera
–
Overlay (combine) video feed with virtual objects
Pose Sensors (optional)
Image
Sensors
Digital
Combiner
Monitors
Generated
Virtual Imagery
(
potentially
stereoscopic)
Video See-Through Display
Video See-Through Display
•
Characteristics:
–
Good video quality is required
–
Resolution/color/contrast in
camera and display
–
Latency/refresh-rates in camera
and display
–
–
–
–
Matchable time delays
Cyber sickness problems
Field-of-view
Frustum and perspective
must match between camera
and computer graphics
Image Quality Comparison
•
Image quality in optical see-through displays is higher for the
real world, but generally inconsistent.
Video See-Through Display
•
Video Monitor AR:
–
Non-co-located display
–
Augmented mirror or TV shows
Projection-Based AR
•
•
Spatial AR
–
User does not wear equipment.
Project virtuality onto real objects
–
May apply compensation for uneven geometry, color and lighting
Generated
Virtual Imagery
Projector
Image
Sensors
Pose Sensors
(optional)
Projection-Based AR
Projection-Based AR
•
Re-map projection from a point-of-view
–
Can potentially improve view for a group of viewers (any
number of simultaneous viewers)
–
–
–
–
Full resolution of the real world
Nothing to wear
Sensitive to lighting and material conditions
Occlusion and geometry issues
Display Systems Classification
Technologies for VR Systems
•
•
Display
–
–
–
Optical see-through
• Additive blending
Video see-through
•
Computer calculated blending
Projection-based
•
Multiplicative blending
Tracking
–
–
–
Localization
• Finding the location
Alignment
•
Registering virtual objects to the real world
Interaction
•
Touching and manipulating virtual objects
Localization
•
Localization:
–
–
–
–
Finding the location and surroundings
Required for looking up models, annotation...
Geolocation: GPS, WiFi, etc.
Local fine-tuning by other tracking methods.
Alignment
•
•
Alignment:
–
Registering virtual objects to the
real world (position and
orientation)
Requires high fidelity tracking:
–
Use knowledge from the real
world
–
Estimate the camera posture
relative the real world
Integration and Interaction
•
•
Integration:
–
–
–
–
Virtual objects optically interacting
Correct lighting conditions
Occlude virtual objects according real-world objects
Cast shadows onto real objects (and vice versa)
Interaction:
–
–
–
Touching and manipulating virtual objects
Virtual objects colliding with real objects
Real objects affected by virtual
Optical Interaction
•
•
Virtual objects occluding real objects
–
Easy with video see-through
Real objects occluding virtual objects
–
Mask using model of real world
–
Requires a model of the real world
Physical Interaction
•
•
Virtual objects colliding with real world
objects
–
Real world knowledge + alignment
–
Requires a model of the real world
Real objects affected by virtual objects
–
Motors, pneumatics, etc...
Further Reading
•
Schmalstieg, D., Hollerer, T. (2016). Augmented Reality: Principles and
Practice (1st ed.). Addison-Wesley Professional.
–
–
Chapter 1: Introduction to Augmented Reality
Chapter 2: Displays