multimedia @ VU
game @ VU / panorama @ VU
The PANORAMA interface is like a virtual gallery, where on the wall and on the floor information about social awareness is presented.
Both the frames on the wall and the floor are moving,
to indicate the passing of time.
The speed as well as the direction of movement may be adjusted.
For example, when many casual encounters
the speed may be increased to indicate hectic activity.
The ViP system is a highly complex presentation system,
based on DirectX9.
It was originally developed for an augmented reality
theatre production, to provide an overlay
of text, images, 3D models and visual effects over live
video feed, capturing the acts played on stage.
For the PANORAMA interface, we use the extended version of ViP,
that allows for the projection of multiple live video feeds,
digital video clips, and sequences of images on objects
in an immersive 3D space.
In the virtual gallery screenshot,
these streaming images are represented by the nine squares
aligned to the bottom of the screen.
In the context of PANORAMA, we may regard the streaming image
feeds as channels, that can, dependent on the
design considerations discussed below, be mapped
on arbitrary models in 3D space,
of which the frames on the wall are just a simple example.
The ViP system allows for a variety of visual effects,
including texture mapping of image feeds on 3D objects,
overlays of multiple image textures,
as well as for example particle systems with streaming
image feeds projected on sprites.
The ViP system also supports the use of vertex and pixel
shaders for the application of both
geometrical and color morphing.
For the control of what visual content is displayed
and its behavior in time, such as for example the life time
and velocity of elements in a paricle system,
the ViP system provides a scripting and programming interface,
as well as direct interaction with the screen.
The interpretation of user actions is highly programmable.
For example, clicking on one of the nine squares
aligned with the bottom of the screen may result
in displaying the content for that channel
fullscreen, or may result in the activation of a
command that displays an object or particle system
with the content of that channel projected on it.
Commands like these may also be given using a standard
network messaging protocol, such as TCP/IP,
thus allowing for the communication facilities
needed to reflect social awareness in the
Although there are only nine channels displayed
in the virtual gallery, the ViP system allows
for an arbitrary number of channels,
that is streaming image feeds.
Obviously, to control the complexity of
the visual display, we are faced with a design problem,
which may be summarized as follows:
Moreover, as will be discussed below,
the design problem also includes the selection of
image material, the 3D models and effects that populate
the PANORAMA space,
and possibly also the choice of what metaphor or type
of immersive space to use for the PANORAMA interface.
- how do we allocate the channels to, respectively,
casual encounters and self-reflection,
- what representation do we select for
projecting a channel in the PANORAMA space, and
- what desgn criteria do we apply for
combining the simultaneous presentation of multiple channels.
We have chosen for the ViP system primarily because
of its visual salience,
providing us with an immersive 3D space with multiple (live)
video feeds and a large variety of visual effects.
As a disadvantage of ViP, however, we may mention
the development effort it requires,
that is it requires programming in C++, as well as shader
programming in HLSL, and the use of the notoriously
complex DirectX API.
For exploring the functionality of the PANORAMA system,
or parts thereof, we will also consider the use
of alternative technologies such as X3D/VRML,
since it allows for fast prototyping,
or of-the-shelf game engines such as the Half-Life 2 Source engine,
which provides convenient tools for the
construction of immersive virtual spaces as well as many built-in visual effects.
Broadly speaking, the PANORAMA system receives explicit and implicit awareness information from staff-members and transforms this information as an art display located in the staff room. It provides a continuous flow of information