topical media & game development
1
multimedia augmented theatre
In june 2003, I was asked to advise on the use
of VR in a theatre production of the Odyssee.
Lacking experience in this field,
I accepted the invitation to participate with some reluctance,
since at the time I didn't have any clue what the VR
for the theatre production should look like.
Nevertheless, I took the invitation as a challenge
and started looking for appropriate hardware,
bothering collegues for information on mixed reality
art productions, and downloading code to explore
software technologies.
Many hurdles ware to be taken.
We had to deal with organizational issues,
such as finding the money for financing the actual production
(which unfortunately never came through),
finding the right people (students, in our case) to select material and
contribute to the code; aesthetic issues, in particular
to determine which approach to take to reach at an effective solution;
and not in the least technical issues, to realize the
production on a sufficiently efficient low-cost platform.
2
context -- the Odyssee theatre production
The
Odyssee.
theatre production was initiated by
Ground Control,
as a successor of previously succesful theatrical spectacles,
including an open air performance of
Faust.
In effect, two performances of the Odyssee were planned,
an out-door (external) version, involving real ships
at the shore of a lake, and an in-door (internal) version,
to be played in temporarily empty office buildings.
The in-door version is meant to give
a more psychological rendering of the Odyssee
[Metamorphose],
where the travels of Ulysses are experienced by the audience
as a confrontation with themselves.
Our contribution was asked for the in-door version,
to enhance the experience of the audience with
additional VR.
The Odyssee is a wellknown account of the travels of Ulysses
leaving Troje, in 24 episodes ending in his return to Ithaca and
his reunion with Penelope.
The actual theatre production takes 12 parts which are played
in 12 successive rooms through which the audience,
subdivided in small groups,
is guided one room after another for about five minutes per room.
Our initial idea was to add information in the form of text and
images, to direct the interpretation of the audience
towards a particular perspective.
In that beginning stage, somewhat optimistically, we
planned to offer multiple perspectives to each
participant, in an individualized manner,
dependent on the actual focus of attention of the individual participant.
initial ideas -- VR and augmented reality:
Our first problem was to find suitable hardware,
that is see-through goggles.
Searching the Internet gave us the name of a relatively nearby
company, Cyber Mind NL,
that specialized in entertainment VR solutions.
Both price-wise and in terms of functionality semi-transparent
see-through glasses appeared to be no option,
so instead we chose for simple LCD-projection goggles
with a (head-mounted) low-resolution camera.
This solution also meant that we did not need
expensive head orientation tracking equipment,
since we could, in principle, determine focus using
captured image analysis solutions such as provided by the AR Toolkit.
Moreover, captured video feed ensured the
continuity and reactiveness needed for
a true (first-person perspective) VR experience.
Augmented or mixed reality
is an interesting area of research
with many potential applications.
However, in the course of the project we dropped our ambition
to develop personalized presentations using
image analysis, since we felt that the technology
for doing this in a mixed reality theatre setting is simply not ripe,
and instead we concentrated on using the captured
video feed as the driver for text and image presentation.
In addition, we developed image manipulation techniques
to transform the (projection of the) captured video,
to obtain more implicit effects, as to avoid the explicit
semantic overload resulting from the exclusive use of text
and images.
technological constraints -- the DirectX platform:
After a few experiments with the AR Toolkit, it soon appeared
that the frame rate would not be sufficient, on the type
of machines our budget would allow for.
Moreover, reading the AR Toolkit mailing list,
marker tracking in a theatrical context
seemed to be more or less unfeasible.
So, we shifted focus to the
DirectX SDK 9,
both for video capture and projection in 3D.
The DirectX toolkit is a surprisingly functional, and very rich
technology for multimedia applications,
supporting streamed video, including live capture,
3D object rendering and precise synchronisation
between multimedia content-related events.
At that time, and still at the time of writing,
our own intelligent multimedia technology
was no option, since it does not allow for
using live video capture and is also lacking in down-to-the-millisecond
synchronisation.
After exploring texture mapping images copied
from the incoming captured video stream,
we decided to use the
VMR-9 video mixing renderer
introduced in DirectX 9,
that allows for allocating 3D objects as its rendering surface,
thus avoiding the overhead of
explicit copies taken from a video processing stream
running in a separate thread. See section 4.3.
Although flexible and efficient,
DirectX is a low-level toolkit,
which means that we had to create our own
facilities for processing a scenegraph,
world and viewpoint transformations, and, even more importantly,
structuring our mixed reality presentations in time.
3
structuring time -- maintaining 'see-through' aesthetics
One of the problems we encountered in discussing what
we conveniently may call the VR with the producer of
the Odyssee theatre performance was the high expectancy
people have of VR, no doubt inspired by movies
as the Matrix and the like.
In mixed reality applications, manipulating persons,
warps in space, and basically any intensive image
analysis or image manipulation is simply not possible
in real time.
Moreover, there is a disturbing tendency
with the layman to strive for semantic overload
by overlaying the scene with multiple images and lines of text,
thus obscuring the reality captured by the camera
and literally blocking the participants view
and awareness of the scene.
Basically, as a guideline, we tend to strive for
70% visibility of the scene,
20% image or projection transformations
and only 10% of information in the form of text and images.
The total duration of our presentation is only 2 minutes,
or 118 seconds to be precise.
We made a subdivision in 4 scenes, with transitions
inbetween, hierarchically ordered in a tree-like structure.
Initially, we abstracted from the actual duration,
by taking only the fraction of the time passed
(in relation to the total duration) as an indication
for which scene to display.
However, when the development reached its final
stages, we introduced actual durations that allowed
us to time the sequence of scenes to the tenth
of a second.
In addition, we used multiple layers of presentation,
roughly subdivided in background captured image,
the transformed captured image projected on 3D objects,
and, finally, pictures and text.
These layers are rendered on top of eachother,
triggered in a time-based fashion, semi-independent
of one another.
The frame rate varies between 20 and 30, dependent
on the number of images simultaneously used
for texturing.
Our final mixed reality theatre application may be considered
a prototype,
awaiting to be put to the test by the audience.
4
lessons learned -- our explorations revisited:
Altogether, the development of the mixed reality
theatre application has been quite an experience,
in multiple ways.
Not in the least it has been (and still is) a challenge
to explain the possibilities of mixed reality
applications to the layman, that do not take
the abstractions we use in our daily academic
life for granted.
To be frank, it also has opened my eyes to what some
consider 'politically incorrect' technology,
in other words Microsoft DirectX,
originally developed as game technology,
and no doubt a rich toolbox for real life multimedia
applications.
Reinventing the wheel is not as simple as it seems.
Nevertheless, developing scenegraph processing facilities
and the appropriate timing mechanisms for controlling
the mixed reality presentation was, apart from
being a rekindling of basic skills, a learnful experience.
Finally, even before putting the application
to the test,
considering the aesthetics of mixed reality
theatre productions,
it may be called an eye-opener to realize
how important reality is,
and how meaningless explicit semantics
(in the form of text and images) may become.
Rather, our explorations were an incentive
to further explore more implicit (graphical)
modifications of the captured scene to convey meaning.
5
example(s) -- pizza boy
Did you like to go to a theatre play, when you were sixteen?
Peter van Kessel, one of the speakers in our visual design
develeped a game pizza boy to overcome
the resistance of secondary school students to theatre.
The game, developed by Peter's agency
Headland,
introduced elements of the theatre play in a game setting.
The player must deliver pizzas, riding a scooter, and
finds him/herself in a situations where he/she
must enact a role or perform some actions, related to the theatre
play.
Visiting the theatre, there is a MIDI-enabled scooter
with which the students can play the game, in preparation
for the actual play (image on the right in the figure above).
Peter reported that the game did indeed help
to overcome the scepticism these yougsters had with theatre.
The game was implemented by
the Little Chicken Game Company,
with
Virtools.
It was available for download at the website of
the governmental department that initiated
the development of the game,
but became such a huge success, also with
other people than the original target group,
that it had to be taken offline.
research directions -- computational art
I started studying computer music in 1979.
For about four years, I worked on a system for real-time sound synthesis
and algorithmic composition.
Some years later, I wrote an essay that I presented in 1986 at the First
Symposium for Electronic Arts, in Utrecht.
The essay was the published in Leonardo, [Art].
Now, almost 20 years later, I reread the abstract, which I OCRed from the original hardcopy:
Computational Art
The author conducts a simple thought experiment investigating the existence
and scope of 'computational art': the utilization of the computer in the visual arts and music.
In the experiment he sets the task of constructing an artifact that is capable of
producing works of art. Since it appears that the artifact needs at least the capability of
imagination, he queries the nature of images and imagery and argues that imagination
is strongly intentional. Next he introduces the concept of notational systems,
since they seem to govern the artistic activity of (not exclusively) machines.
Confronted with the question of whether we are able to develop a computational analogue for taste,
he finds that notational systems prove to be necessary for mediating the method of production
of an artwork and the appraisal of its artistic value.
Furthermore, the author shows that there are certain epistemological limits to the creativity of
an imaginative device.
Although the outcome of this hypothetical construction task clearly
denies the possibility of an autonomously creative artifact,
there seems to be no reason to worry about the opportunities for computational art:
the computer appears to be a unique tool in exploring the possibilities of artistic production,
guided by artists.
Today, despite the immense increase in computational power
and the overwhelming amount of multimedia peripherals, the computer is still not able
to produce art autonomously.
But as a tool it is about to take over the entire industry,
ranging from traditional publishing to film production and distribution.
And, perhaps more tangible, as a tool for the creation of media content
it is becoming ever better!
(C) Æliens
04/09/2009
You may not copy or print any of this material without explicit permission of the author or the publisher.
In case of other copyright issues, contact the author.
