Overview
The generation of engaging visual representations for interactive storytelling represents a key challenge for the evolution and popularization of interactive narratives. While interactive storytelling systems typically adopt computer graphics to represent virtual story worlds, which facilitates dynamic content generation, the visual quality of real-time rendered graphics remains inferior to live-action film.
This project explored a new approach to create more engaging interactive narratives using real actors and settings instead of computer-generated graphics. The research developed methods for video-based interactive storytelling that maintain the logical structure and interactivity of computational narratives while achieving the visual quality and appeal of live-action cinema.
Research Challenges
- How can video compositing techniques achieve interactive frame rates for real-time narrative generation?
- What methods can automatically edit and combine video segments to maintain temporal and spatial continuity?
- How can intelligent agents replicate the roles of filmmaking professionals in automated video production?
- What approaches enable video-based systems to work with plots generated by nondeterministic planning algorithms?
- How can interactive broadcast television support viewer choices while respecting air time constraints?
Approach
This research developed methods for creating interactive narratives using pre-recorded video with real actors, combining the visual appeal of live-action cinema with the interactivity of computational storytelling systems. The approach addresses both the technical challenges of real-time video processing and the creative challenges of maintaining narrative coherence across multiple story paths.
A key technical contribution involved developing video compositing algorithms that operate at interactive frame rates. Using techniques such as chroma key (green screen) and real-time video compositing, the system can dynamically combine visual elements to create scenes that respond to user choices while maintaining cinematic quality. The method ensures compatibility with plots generated by nondeterministic planning algorithms.
The research introduced a system of intelligent agents that automate filmmaking processes. These agents perform roles analogous to human filmmaking professionals: a Scene Composer agent handles real-time video compositing, while an Editor agent maintains temporal and spatial continuity by automatically selecting appropriate transitions between video segments, avoiding jump cuts, and presenting story events through smooth, continuous sequences.
The work also explored interactive television applications, developing narratives where multiple viewers can make individual choices while still finishing the story at the same time, respecting broadcast air time constraints. This "synchronized choice" approach enabled truly interactive experiences within the constraints of traditional television broadcasting.
Key Contributions
Developed video compositing algorithms that operate at interactive frame rates, enabling real-time combination of video elements with real actors to create dynamic interactive narratives with cinematic quality.
Created AI agent systems that replicate filmmaking professional roles, including scene composition and automatic video editing that maintains temporal and spatial continuity while avoiding jump cuts.
Developed methods for interactive broadcast television that allow multiple viewers to make individual story choices while completing the narrative simultaneously, respecting air time constraints of broadcast media.
Designed video-based dramatization architectures compatible with nondeterministic planning algorithms, enabling procedurally generated narratives to be presented through live-action video sequences.
Interactive Narratives
Several interactive narrative prototypes were produced to validate and demonstrate the video-based interactive storytelling approach:
🏆 The Princess Kidnapping ITU Innovation Award
An interactive narrative designed for interactive broadcast television. Viewers can choose their own path through the story while still finishing at the same time, respecting broadcast air time principles.
Received Honorable Mention for "Innovation" from the International Telecommunication Union (ITU) in the 1st ITU IPTV Application Challenge (2011).
🏆 Raider of The Lost Videogame ITU Interactivity Award
An interactive narrative that proposes a new "movie-game" genre for interactive television. The story structure and cinematography put viewers directly in the action, allowing them to interact as video game players while watching.
Received Honorable Mention for "Interactivity" from the International Telecommunication Union (ITU) in the 2nd ITU IPTV Application Challenge (2012).
Little Gray Planet - Interactive TV Comics
An educational interactive story about environmental preservation set in a future plagued by pollution. Users follow the protagonist Pauline through different scenes, making ecologically friendly choices that visibly improve the environment or face the consequences of harmful actions.
Modern Little Red Riding Hood
A modern, comic adaptation of the Little Red Riding Hood fairy tale that validates the video-based approach. The system generates diverse story variations with outcomes ranging from traditional fairy tale plots to unconventional and humorous alternatives, demonstrating the flexibility of the approach.
Related Publications
-
Edirlei Soares de Lima; Bruno Feijó; Antonio L. Furtado. Video-based Interactive Storytelling Using Real-time Video Compositing Techniques. Multimedia Tools and Applications, Volume 77, Issue 2, p. 2333–2357, 2018. [DOI]
-
Edirlei Soares de Lima. Video-Based Interactive Storytelling. PhD Thesis. Pontifícia Universidade Católica do Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil, August, 2014. [PDF]
-
Edirlei Soares de Lima; Bruno Feijó; Antonio L. Furtado; Cesar T. Pozzer; Angelo E. M. Ciarlini. Automatic Video Editing For Video-Based Interactive Storytelling. Proceedings of the 2012 IEEE International Conference on Multimedia and Expo (ICME 2012), Melbourne, Australia, July 2012. pp. 806-811. ISBN: 978-0-7685-4711-4. [DOI] [PDF]