Narrative Interaction

Narratives and Interactive

Storytelling

Lecture 04 – Narrative Interaction

Edirlei Soares de Lima

<edirlei.lima@universidadeeuropeia.pt>

Narrative Interaction

•

What is narrative interaction?

–

Is how users interact and change narratives.

–

There are many ways to interact with narratives (e.g.: traditional GUI

interfaces, speech recognition, virtual reality interaction, hand-drawn

sketches, social networks, …)

Traditional GUI Interfaces

•

Users are more used to this type of

interface.

Limits user interaction in multi-user

settings.

Multimodal Interface

•

Movement recognition

Voice recognition

Mobile interaction

Emotion recognition

Machine Learning

–

Video

Lima, E.S., et al. Multimodal, Multi-User and Adaptive Interaction

for Interactive Storytelling Applications. X Brazilian Symposium on

Computer Games and Digital Entertainment (SBGames 2011).

Paper and Pencil

Users can switch between places by changing

the paper shown to the camera.

Stories are graphically represented in

augmented reality over the paper.

The system recognizes hand-drawn

sketches and converts them to 3D

objects in the virtual world.

Webcam

Users can change the decisions

made by virtual characters by

sketching objects on the paper.

Marker

The entire world may comprise

several sheets of paper.

Sheet of Paper

Pencil

Users act as gods of the

virtual world.

–

Video

Lima, E.S., et al. Draw Your Own Story: Paper and Pencil Interactive

Storytelling. International Conference on Entertainment Computing

(

ICEC 2011).

Paper and Pencil

Users can switch between places by changing

the paper shown to the camera.

Stories are graphically represented in

augmented reality over the paper.

The system recognizes hand-drawn

sketches and converts them to 3D

objects in the virtual world.

Webcam

Users can change the decisions

made by virtual characters by

sketching objects on the paper.

Marker

The entire world may comprise

several sheets of paper.

Sheet of Paper

Pencil

Users act as gods of the

virtual world.

–

Video

Lima, E.S., et al. Draw Your Own Story: Paper and Pencil Interactive

Storytelling. International Conference on Entertainment Computing

(

ICEC 2011).

Paper and Pencil

•

Pre-processing phase:

•

Support Vector Machine (SVM):

o Input: numerical features extracted from the sketches.

o Output: vocabulary of drawings recognized by the

system.

Interactive Comics

•

Interactive Objects:

interactive

ob_Ej_fe_f_ec_c_tt_:

World State:

healthy(anne)

healthy(john)

healthy(jimmy)

¬

at(antidote, hospital_room)

wasinfected(emma)

aatt((aannttiiddoottee,, hhoossppiittaall__rroooomm))

at(jimmy, house)

at(emma, house)

at(anne, house)

at(john, hospital_room)

Social Interaction

•

Interaction through Social Networks

•

Interaction By Comments

Interaction By Preferences

Interaction By Poll

Lima, E.S., et al. Social Interaction for Interactive Storytelling.

International Conference on Entertainment Computing (ICEC 2012).

Natural Language Interaction

•

Syntax Parser: Stanford Parser¹

Example: “Draco should kill Marian!”

S

VP

.

NP

nsubj(kill-3, Draco-1)

aux(kill-3, should-2)

root(ROOT-0, kill-3)

dobj(kill-3, Marian-4)

NNP

Draco

MD

VP

!

should VB

kill

NP

NNP

Marian

Phrase Structure Tree

Typed Dependencies

¹http://nlp.stanford.edu/software/lex-parser.shtml

Natural Language Interaction

•

Extracting “subject – direct object” relationships:

Example: “Draco should kill Marian!”

“

Draco”

noun

Typed Dependencies

nsubj(kill-3, Draco-1)

aux(kill-3, should-2)

root(ROOT-0, kill-3)

dobj(kill-3, Marian-4)

verb

“kill”

kill(Draco, Marian)

first-order

logic sentence

“Marian”

noun

nsubj = nominal subject

dobj = direct object

Natural Language Interaction

•

Anaphora resolution:

Example: “Brian saves Marian and marries her.”

first-order

logic sentence

“

noun

Brian”

Typed Dependencies

“saves”

save(Brian, Marian)

nsubj(saves-2, Brian-1)

nsubj(marries-5, Brian-1)

root(ROOT-0, saves-2)

dobj(saves-2, Marian-3)

conj_and(saves-2, marry-5)

dobj(marries-5, her-6)

verb

“

Marian”

noun

first-order

logic sentence

“Brian”

“marries”

marry(Brian, Marian)

noun

verb

“her”

“Marian”

pronoun

noun

Stanford Deterministic Coreference Resolution System

Natural Language Interaction

•

Omitted Subject:

Example: “Kill the princess!”

Typed Dependencies

root(ROOT-0, Kill-1)

det(princess-3, the-2)

?

first-order

logic sentence

dobj(Kill-1, princess-3)

“kill”

kill(*, Marian)

verb

“Marian”

noun

Interaction by Preferences

•

Natural Language Processing

–

Sentiment Analysis

Interaction by Preferences

•

List of words – each word ꢀ_ꢁ is associated with a numerical

score ꢀ_ꢁ ∈ −1.0, +1.0 .

ꢂ

ꢋꢌ ꢃꢄ ꢅ_ꢆ > ꢍ then ꢅ_ꢆ = positive commentary

ꢊ

1

ꢃꢄ ꢅ_ꢆ = ෍ ꢀ_ꢁ^ꢂ ꢋꢌ (ꢀ_ꢁ ∈ ꢅ_ꢆ)

ꢇ

ꢋꢌ ꢃꢄ ꢅ_ꢆ < ꢎ then ꢅ_ꢆ = negative commentary

ꢏꢄℎꢐꢑꢒꢋꢓꢐ ꢅ_ꢆ = neutral commentary

ꢁ

ꢈꢉ

Example: C₁= “No!! I love the princess… :(”

Words score:

“

no” = -1.0

love” = +0.8

:(” = -1.0

−

1.0 + 0.8 − 1.0

3

ꢎ = −0.3

ꢍ = +0.3

ꢃꢄ ꢅ_ꢉ =

= −0.4

C₁= negative commentary

Mobile Interaction

Lima, E.S., et al. A Multi-User Natural Language Interface for Interactive Storytelling in TV and

Cinema. XI Brazilian Symposium on Computer Games and Digital Entertainment (SBGames 2012).

Project Task:

Interaction with Comics

StorylineA – Without user interaction

…

^Pi

P_i₊₁

Storyline B – User interacts and changes the girl’s decision

^Pi

P_i₊₁

Windows: http://www.inf.puc-rio.br/~elima/is/ComicsViewer.zip

MacOS: http://www.inf.puc-rio.br/~elima/is/ComicsViewer.app.zip

Comics Viewer: Context Overview

•

General structure:

All events of the story

and the connections

between the events.

<

StoryNetwork>

...

</StoryNetwork>

Variables of the world

state.

...

</Variables>

<

Interactions>

Definition of the user

interaction points.

...

</Interactions>

<

Resources>

Images to represent

characters, objects and

environments.

...

</Resources>

Operators>

<

...

<

/ComicsViewer>

/Operators>

Definition of how each

event is visually

represented.

<

Comics Viewer: Interactions

•

<

Interactions:

Interactions>

<

Interaction type = "balloon" event = "ID" text = "TEXT"

thoughtx = "TX", thoughty = "TY", balloonx = "BX",

balloony = "BY" mx = "MX", my = "MY", defaultoption = "ID"

defaultid = "ID">

<

.

Option text = "TEXT" nextevent = "ID" currentevent = "ID"/>

..

<

.

/Interaction>

..

<NotInteracted resource = "RES" x = "X" y = "Y" scale = "S"

nextevent = "ID" currentevent = "ID"/>

<Interacted resource = "RES" x = "X" y = "Y" scale = "S"

nextevent = "ID" currentevent = "ID"/>

<

.

/Interaction>

..

</Interactions>

Comics Viewer: Thought Balloons

•

Thought balloon example:

<

Interaction type = "balloon" event = "N6" text = "Which path should

I take?" thoughtx = "260", thoughty = "35", balloonx = "300",

balloony = "47" mx = "220", my = "75", defaultoption = "N22"

defaultid = "2">

<Option text = "Uncle Wolf's house" nextevent = "N32"

currentevent = "3"/>

</Interaction>

Comics Viewer: Interactive Objects

•

Interactive object example:

<

NotInteracted resource = "antidotenormal" x = "40" y = "140"

scale = "0.5" nextevent = "N4" currentevent = "1"/>

Interacted resource = "antidoteinteracted" x = "40" y = "140"

scale = "0.5" nextevent = "N5" currentevent = "1"/>

<

</Interaction>

Comics Viewer: Variables

•

Examples of variables:

</Variables>

•

Changing variables in thought balloons:

</Interaction>

Comics Viewer: Variables

•

Examples of variables:

</Variables>

•

Changing variables in interactive objects:

<

NotInteracted ... setvar = "hasantidote = true"/>

Interacted ... setvar = "hasantidote = false"/>

</Interaction>

Comics Viewer: Variables

•

Examples of variables:

</Variables>

•

Using variables to create conditions:

<Edges>

...

<

Edge ... condition = "hasantidote == true"/>

Edge ... condition = "hasantidote == false"/>

...

</Edges>

Comics Viewer: Variables

•

Examples of variables:

</Variables>

•

Using variables to change characters:

...

<Event id = "N7" event = "meet(Little girl, #villain#, the

crossroad)"/>

<

.

Event id = "N22" event = "meet(Little girl, #villain#, the

woods)"/>

Event id = "N32" event = "meet(Little girl, #villain#, villain’s

house)"/>

..

</Events>

Project Assignment 4

4

) Add interaction to the interactive narrative of your project

(the same narrative created in the last Project Assignments)

using the Comics Viewer system.