topical media & game development

talk show tell print

print / present


digital content creation

post-modern design allows for sampling

learning objectives

After reading this chapter you should be able to mention some basic rules of digital content creation, discuss what criteria your portfolio should meet, describe how you would approach the design of a logo, explain the notion of user-centered design, and characterize the issues that play a role in dveloping multimedia for theatre.

Whether your ambition is to become a professional designer or not, also for students of information science and computer science, a course in visual design is a must, I think.

In this chapter, we will treat various aspects of digital content creation. The first section discusses how to approach visual design and gives a number of basic design assignments, that can be used to get experience with visual design. Section 2 discusses the issue of workflow and tools, and investigates how design fits in with the process of developing multimedia applications. In the final section, I will elaborate on a theatre project I was involved in, for which I had to develop an augmented reality application.



visual design

When you are trained as a visual artist, as I once was (in the pred-digital era), you must do many basic exercises with form and color, learn the skills of drawing and painting, including sill lifes and portraits. And, to graduate you must make an exhibition of your work over the years.

In the digital era, things have changed. The skills that you need to learn, which include the use of tools for digital content creation, as well as the artistic goals, I would say. What has not changed, however, is the need for basic exercises and the presentation of your work, that is the creation of a portfolio.

In this section, I will give an outline of the course visual design, that I started in februari 2005. This outline is meant to give you a general idea of how to approach design, and to give you some hints on how to acquire the skills needed to act as a designer. If you have the opportunity to take a course in visual design, then don't hesitate and do it!




The overall goal of the visual design course is to establish some basic aesthetic awareness, by providing suitable exercises and assignments. In addition, the student is supposed to become familiar with the craft of design, which necessarily, but not exclusively, involves the use of tools and techniques.

To accomodate for the various interests and backgrounds of the students, we distinguish between several tracks or perspectives, as summarized below:

track(s) -- perspective

  • styling -- concept and presentation
  • digital content -- material, animation
  • tech track -- special effects
To illustrate the various perspectives, I invited guest speakers who showed their work and talked about their approach. A selection of their works is hown in this book. The process of design is very complex, ranging from conceptual explorations and sketching up to the stage of finalizing delivery. It is also very personal. However, as each of the speakers testified to, a significant portion of the time and effort goes into negotiating with the client. Whether it involves taking photographs for an advertisement or setting up a campaign, it takes a lot of back and forth to get an idea of what the client wants. In the agency of one of the speakers Mark Veldhuijzen van Zanten, they have created roles to help eachother come up with the right ideas.

  • e-motionist -- make emotion, rhythm and movement flow together
  • chaoticus -- who sees chaos within order
  • formologist -- who approaches the fabrication of forms as an art
  • infonaut -- who moves in the twilight zone of information and meaning
  • transformator -- who transforms images and concepts into new matter
Although slightly ironic, and in practice not so clear-cut, these roles give you an idea what cognitive modes are involved in bringing an original concept to a stage of finalization.

Coming up with an idea and sketching require a more reflective cognitive mode, whereas finalizing a design requires a more experience-related cognitive mode. It is important to choose the right tools to work with, dependent on which phase you're in, for example paper and pencil when you are still sketching and Photoshop or maya when you put your ideas into production.




However interesting the process may be, design is not about process but about product(s). Such a product may be, dependent on what you ar good at, one of the following:


  • web site -- e.g. conference, campaign (browse)
  • 2D/3D animation -- promotion/ad (temporal sequence)
  • virtual space -- game/infotainment (navigate)
  • ebook -- story (sequential experience)
As indicated between brackets, each of the products favors a particular mode of interaction. Although interaction is not an aspect of visual design as such, it is an important aspect to take into account. In section 9.2, we will look in more detail at the issue of interaction and usability in general.

One easily overlooked issue in a design project, is the creation of a portfolio. There may easily be some confusion here with regard to what should be considered the product of design, one of the items in the list above, or your portfolio. The answer is simple. Both! As a record of the process of design, the portfolio is itself a product of design.

portfolio -- design as a product

  • concept(s)
  • sketches & explorations
  • finalized products
  • evaluation & reflection
What criteria should a portfolio meet? Well, nowadays it is not only common to have your portfolio on the web, I have been told that you cannot do without it. So, first of all, your portfolio should be web-friendly. And your work should not be too many clicks away! And, secondly, it should give sufficient insight in what you have to offer, so that a potential client can decide whether it is worth the effort to contact you.

In our visual design course, I require that the portfolio contains a description of the concept(s) underlying the design, sketches and explorations as well as the finalized products. It is also required, after all it is an academic course, to provide an evaluation and some reflection in the form of an essay on a topic such as 2D versus 3D aesthetics, animation techniques, or (more theoretical), elements of style or theories of creativity.

  In my perception a portfolio is about the
  past and I feel much more related to the
  present especially in my work as a
  photographer. To me the relation with
  the present and the subject/assignment directs
  my creativity, the experiences over the 
  past draw my skills. My challenge is to be
  truly inspirred, to be present in the present.
However, as illustrated by the motivation Jaap Stahlie gives with his portfolio (above), it is perhaps wise not to overdo it!

There are basic exercises, obligatory for all students, and a final assignment, where you have a choice between three productions, each with a different supervisor. In addition, as explained in the guidelines, all students must write an essay, and give a presentation in class. For deadlines, see the schedule. There will be periodic checks on the status of your work. Each year there will be recommended themes.


In the visual design course there are basic exercises, obligatory for all students, and a final assignment, where the student has a choice between three productions.

basic exercises

  1. develop a logo
  2. create a sign
  3. design a collage
  4. write a story
For the final assignment, there is a choice between the following assignments: developing a house style, creating a non-linear visual story, and designing a suitable game environment For this assignment) the students are allowed to work in groups. However, the contribution of each individual must be reflected in his/her own portfolio.




The first requirement when working in an area such as visual design is that you acquire sufficient self-discipline to find the challenge in the assignments and to complete the tasks involved. Since a department of computer science is not the natural habitat for a course in visual design, I have laid down some strict rules:


  • be present -- 2 omissions max.
  • be in time -- hard deadlines
  • be online -- have your portfolio available
  • be creative -- don't steal without a reason/mentioning
  • be smart -- there is no 2nd chance
These rules may well apply when you work, after you graduated, as an individual designer/developer or as a member of a team in some agency. To finish this section, I may remark that design is an interesting field, full of implicit (not always so obvious) wisdom and apparent paradoxes. Whatever you do, deliver! Silence is lethal. And, as another item of colloquial wisdom, be authentic, but only if possible!



research directions -- on creativity

One of the assignments in the visual design course is an essay. As one of the recommended topics we have a reflection on theories of creativity. As such it is not a training in creativity. This section contains some random thoughts on the processes and products of design, and ends with the provoking statement there is no theory of creativity. You may, however, try to find some counter-arguments, for example in the line of  [Hewlett and Selfridge-Field (1998)].


Multimedia is a promising technology, and (nowadays) affordable. So we see that multimedia (which includes 3D-graphics, video and sound) is increasingly being used, also in information visualisation. But what is it good for? To quote  [Klabbers (2006)]:

multimedia's promise is terribly generalized,

it simply lets you do anything.

As with any new technology, the early multimedia productions (in particular CDROM and CD-I) were not optimal with respect to (aesthetic) quality. To quote  [Klabbers (2006)], again:

shovelware -- multimediocrity

... far from making a killing, it looked as if the big boys ... had killed the industry by glutting the market with inferior products.

Perhaps the industry in the late eighties did not have the right business model. But, then again, what are the chances of multimedia in our time. One more quote from  [Klabbers (2006)]:

if multimedia is comparable to print then yes, we'd be crazy to expect it to mature in a mere ten years.

eliminating complexity

So now, in the new millenium, we are (sadder and wiser) in a position to approach the effective deployment of multimedia afresh. What we look for is aesthetic quality. How do we find it? Easy enough, just be authentic.

"Learning how to not fool ourselves is, I'm sorry to say, something that we haven't specifically included in any particular course that I know of. We just hope you've caught it by osmosis."

Richard Feynman

Authentic in creating multimedia means, apart from not fooling yourselves, that you must become aware of the message orinformation you want to convey and learn to master the technology to a sufficient degree. But beware, an effective multimedia presentation is not the same as scientific argumentation:

the media equation

We regularly exploit the media equation for enjoyment by the willing suspension of our critical faculties. Theatre is the projection of a story through the window of a stage, and typically the audience gets immersed in the story as if it was real.

These quotes, as well as the following one have been taken from an online essay on eliminating complexity which provides an argument against inessential gadgets and spurious complexity and bells and whistles in whatever you can think of, including user interfaces and scientific theories. Back to the subject, what does master the technology to a sufficient degree mean? Just remember that what you do is a form of engineering.


"engineering is the art of moulding materials we do not wholly understand ... in such a way that the community at large has no reason to suspect the extent of our ignorance."

A. R. Dykes.

In other words, learn the tool(s) that you are using to a degree that you master the basics and easily cut through its apparent magic.

theories of creativity

Producing multimedia, in whatever form, has an element of craftsmanship. But, given the need for aesthetic quality, whatever way you approach it, there is an element of creativity. That means, you're in for a challenge. And, to quote  [Klabbers (2006)],

The best thing is to empower yourself. But before you can do that, you need to understand what you are doing -- which is a surprisingly novel thing to do.

Now it is tempting to look for a set of guidelines and rules that give you a key to creativity. So let me be straight with you:

there is no theory of creativity

On the other hand, there are techniques for producing ideas. And some recommend a sequence of steps, such as:


browse, explore; chew it over; incubation, let it rest; illumination (YES); verification,does it work?

And in addition, still following  [Klabbers (2006)], there are some general rules:

general rules

  • if you aim to please everybody, you will please nobody
  • constraints come with the territory, you must learn to love them
  • emotional charge is the key to success
Now, if you'd ask me, I would say, just

make your virtual hands dirty.



designing the user experience

In a time in which so much information is available as in ours, we may make statements like

postmodern design

... postmodern design is of a highly reflective nature ... appropriating design of the past ... in other words, sampling is allowed but no plagianarism

One interpretation might be that it is impossible to be original. But another interpretation might be that it is undesirable to be (totally) original. In other words, as discussed in section 2.3, it is necessary that your design contains references to not only some real situation but also to other work, so that it can be understood and experienced by the user/spectator/client whatever you like to call the people that look at your work. As observed in  [Sherman and Craig (2003)], designing for multimedia does not take into account only the technological or aesthetic issues, but also constraints on what people can perceive and what the experiential context is in which the work is presented, which may be re-phrased more plainly as what expectations the user has.

game design

Let us consider how these observations affect one of the project assignments for our visual design course. Also for game design, there are several options, dependent on the track the student is most comfortable with.

game design

  • style -- develop concept, plot and visual assets for a game of choice
  • content -- develop environments, models and animations for a game of choice
  • effects -- develop models, textures and special effects (shaders) for a game of choice
To explain, style may be considered to involve the whole gamut of concepts, plot and genre, as well as the visual assets or props of the game, those things by which the game differentiates itself from other games. Since the reader of this book will probably be more familiar with games than the author, there is no need to expand on these issues. Content is concerned with the actual game environment, including the models and animations. Finally, effects, to simplify things, is everything else, those things that are visual but does not belong to the story line or game environment.

Games, perhaps more than any other multimedia application, are appealing, not because they are useful, although they might be, but because the user gets emotionally involved, not to say addicted. Now, following  [Norman (2004)],

did you ever wonder why cheap wine tastes better in fancy glasses?

Exactly, because cheap glasses do not give us the same emotion. It is, indeed, a matter of style!

Obviously, games are played for fun. As applications, games may be classified as seductive, which is, see section 2.3, stronger than persuasive.  [Norman (2004)] distinguishes between four categories of pleasure.


  • physio-pleasure -- of the body
  • socio-pleasure -- by interaction with others
  • psycho-pleasure -- due to use of the product
  • ideo-pleasure -- reflecting on the experience
In other words, games are seductive, or fun to play, because they arouse any combination of pleasure from the categories above. Which combination depends on the kind or genre of game. Quoted from  [Norman (2004)], but originally from  [Wolf] we can list, not exhaustively, the following genres of video game:


Abstract, Adaptation, Adventure, Artificial Life, Board Games, Capturing, Card Games, Catching, Chase, Collecting, Combat, Demo, Diagnostic, Dodging, Driving, Educational, Escape, Fighting, Flying, Gambling, Interactive Movie, Management Simulation, Maze, Obstacle Course, Pencil-and-Paper Games, Pinball, Platform, Programming Games, Puzzle, Quiz, Racing, Role-Playing, Rhythm and Dance, Shoot Em Up, Simulation, Sports, Strategy, Table-Top Games, Target, Text Adventure, Training Simulation, and Utility.

When you develop a game it is good to reflect on what genre your game belongs to, because that will directly affect the user's expectation when playing the game, due to the conventions and rules that exist within a particular genre. For video games, which can be characterized as a mixture of interactive fiction with entertainment, interaction is evidently another distinguishing factor in determining the success of the game.

Corresponding to the kind of pleasure a user may experience,  [Norman (2004)] distuinguishes between three levels of design:

levels of design

  • visceral -- what appeals to our intuition (affordance)
  • behavioral -- is all about use (performance)
  • reflective -- its all about message, culture and meaning
Of these, the latter two should be rather obvious, although we will elaborate on the notion of usability later on. But what does affordance mean, and how is it related to our intuition.



affordance -- ecology of behavior

The notion of affordance has a long history. According to Don Norman, the word "affordance" was invented by the perceptual psychologist J. J. Gibson, to refer to the properties of the world that 'invite' actions, as for example a char invites one to sit. Originally, however, the notion of affordance dates back to the beginning of the 20th century, when it was used in phenomenologist philosophy to describe how the world around us presents itself as meaningful. Affordance, in other words, is a concept that explains why it seems natural for us to behave in a particular way, either because it is innate, as the reflex to close one's eyes by sudden exposure to light, or because we have learned that behavior, as for example surfing the web by clicking on links. In game or product design, thinking about 'affordance' may help us to find the most natural way to perform certain actions. Natural is in this context perhaps not the best phrase. What we must take into account is what is perceived as an affordance, and how actions fit in with what we may call an exology of behavior (with the system).

How does this diversion in abstract philosophy help us design better games? To answer this question, I'd like to recount my visit at the Virtual Humans Workshop, held in october 2004 at the Institute of Creative Technologies, in Los Angeles. Of interest in particular is the ICT Games Project:

ICT Games Project

The goal of the ICT games project is to develop immersive, interactive, real time training simulations to help the Army create a new generation of decision-making and leadership-development tools.

As further explained on the website: with the cooperation of the U.S. Army Research, Development and Engineering Command, Simulation Technology Center (RDECOM STC), Training and Doctrine Command (TRADOC), and commercial game development companies, ICT is creating two training simulations that are intended to have the same holding power and repeat value as mainstream entertainment software.

The two training applications developed by ICT are:

  • Mission Rehearsal Exercise -- to solve a potential conflict after a car accident
  • Language Training Simulation -- to learn how to contact local leaders in arabic
The mission rehearsal exercise is situated in former Yugoslavia. The trainee is confronted with the situation after a car accident in which a boy got injured. The mother of the boy is furious, and a potentially hostile crowd is waiting. An army convoy is on its way to a nearby airport, and needs to pass the crossing where the accident took place. The trainee must decide what to do and give appropriate orders, knowing that the wrong decision may lead to serious trouble.

The language training simulation is situated in the Middle-East, and is meant to teach the trainee not only some basic arabic but also proper ways of conduct, in conformance with local customs to gain confidence.

Both applications are highly realistic, with impressive graphics. A lot of attention has been devoted to creating the models and environments. Both simulations are implemented using the Unreal game engine. The both support speech input. The challenge in both simulation games was to come up with a natural way to indicate to the trainee what options for actions were available. Natural means, in this context, that it should fit within the simulation or game environment. Obviously, a menu or a row of pushbuttons does not fit naturally within such an environment, and would break what we have previously, in section 2.3 called 'immersion'.

I was invited at ICT for the Virtual Humans Workshop because of my involvement with embodied conversational agents (ECAs), as discussed in section 8.3. The topic of the workshop was, among others, to investigate whether the notion of affordance could help in analyzing and evaluating the interaction of a user/trainee with the simulation game. These were the questions we tackled:

Virtual Humans Workshop

  • Is it more appropriate to construct a frame of analysis that encompasses both user and ECA in a single interaction graph?
  • Is it fitting to think in terms of a fixed graph that the user comes to recognize, or is the graph itself a dynamic structure?
  • Is it even appropriate to focus on "affordances to act," or is it more fitting to consider cues that influence the mental interpretations that lead to action (e.g., affordances of control, affordances of valence of potential outcomes, etc.)? How does this relate to intrinsic motivation?
This workshop was a follow-up on a seminar in Dagstuhl on Evaluating Embodied Conversational Agents, where we discussed the tipics of interaction and affordance in a special interest group. In the research directions, I will further discuss an evaluation study that we did on agent-supported navigation in a virtual environment.

Back to our question, how can affordance help us in designing a game? In the mission rehearsal exercise, described above, it would be much more easy to have a menu with all available options listed. However, such amenu would defeat the purpose of the simulation, since such menus will not likely occur in real life. Immersion is, in other words, necessary to maintain the emotional involvement with the application, and affordance is the key to immersion. But, although it sounds like an answer, it does rather lead to another question, how can we define the usability of a game?



usability and fun

In interaction design there is a clear, yet unresolved, tension between usability and fun. Usability is, despite the many disputes, a well-defined notion:

usability (ISO DIS 9241-11)

... the effectiveness, efficiency and satisfaction with which specified users can achieve particular goals in particular environments ...

This is the ISO DIS 9241-11 definition, cited from  [Faulkner (2000)]. In sectiona 10.3 we will further investigate usability as a means to evaluate systems from an interaction perspective. Now, I wish to focus on why artefacts or games might be appealing even if these same aspects may compromise usability in the traditional interpretation.

In describing a fancy juice squeezer, designed by Philip Starck  [Norman (2004)] observes, following  [KS], that is:

emotional involvement

  • entices by diverting attention -- unlike the common
  • delivers surprising novelty -- not identifiable to its function
  • goes beyond obvious needs and expectations -- it becomes something else
  • creates an instinctive response -- curiosity and confusion
The phrase satisfaction in the definition of usability above seems somewhat meagre to explain the emotional involvement with games, and even inappropriate as one realizes that, in the mission rehearsal exercise, frustration might actually be beneficial for the learning experience.



example(s) -- visual sensations

The dutch visual sensations festival is an annual contest for VJs. In 2005, in cooperation with the festival, a parallel seminar seminar was held discussing the topic of the history of VJ-ing, a aplenary discussion of the relation between club-VJs and the established art circuit. In addition there were two guest speakers, Geert Mul and Micha Klein, both visual artists who also have a ten-years experience as VJ.



Above is another work of Geert Mul, in cooperation with DJ Speedy J. It was shown a dance event in cooperation with Rotterdam Maritime Museum. On the right, the cranes are swinging on the rhythm of the music.

The portfolio of Geert Mul starts with a quote from  [Arnheim (1957)]:

form and content

Very often people assume that "form" as a concept is the opposite of something called "content". This assumption implies that a poem or a musical piece or a film is like a jug. An external shape, the jug, contains something that could just as easily be held in a cup or pail. Under this assumption, form becomes less important than whatever it is presumed to contain.

We do not accept this assumption. If form is the total system, which the viewer attributes to the film, there is no inside or outside. Every component functions as part of the overall pattern that is perceived. Thus we shall treat as formal elements many things that some people consider content. From our standpoint, subject matter and abstract ideas all enter into the total system of the artwork ( .... )

I totally agree with this. And perhaps this is why I have a preference for artworks that are slightly out of the main stream of tradional art.

research directions -- engaging with fictional characters

What do you need to evaluate your game or multimedia application? There are many ways to gain insight in how your system is being used, see section 10.3. But if you want to establish functional properties of a multimedia application, for example the effectiveness of using an agent in navigating a virtual environment, in a scientifically more rigorous way, you need to have:

experimental validation

  • a theory -- in our case: PEFiC
  • a test scenario -- for example, memory tasks in a digital dossier
  • the technology -- to realize applications
In this section, I will briefly describe our efforts in experimentally validating the use of ECAs in virtual environments. As technology, we use our intelligent multimedia technolgy, described in sectiona 8.3 and appendix E. So what must be explained is the theory we adopt and the test scenarios we use. PEFiC is a theory developed by Johan Hoorn and Elly Konijn, to explain Perceiving and Experiencing Fictional Characters, see  [PEFIC]. The PEFiC theory may serve as the basis for the experimental evaluation of user responses to embodied agents. In summary, PEFiC distinguishes between three phases, encoding, comparison and response, in analyzing the user's behaviour towards an agent. Encoding involves positioning the agent (or fictitious character) on the dimensions of ethics (good vs bad), aesthetics (beauty vs ugliness) and epistemics (realistic vs unrealistic). Comparison entails establishing personal relevance and valence towards the agent. Response, finally, determines the tendency to approach or avoid the character, in other words involvement versus distance.

In general, having a virtual environment, there is, for developing test scenarios, a choice between:

validation scenario(s)

  • navigation -- pure interactivity
  • guided tours -- using some narrative structure
  • agent-mediated -- navigation and guided tours
For our application, a virtual environment of an artist's atelier, we have three experimental conditions, navigation without an agent, navigation with a realistic agent and navigation with a cartoon-like (unrealistic) agent. To ensure that these conditions can be compared, the acrtual information encountered when using the application is in all conditions the same.

The independent variable in our experiment, the degree of realism of the agent, corresponds with the epistemic and to some extent the aesthetic dimension of appraisal in the PEFiC theory. As dependent variables we have, among others, user satisfaction, believability, that is estimated usefulness of the agent, and also the extent to which the relevant information is retained.



The application is a digital dossier for the Dutch artist Marinus Boezem. The spatial metaphor we used for the dossier is the artist's atelier. We created a virtual environment containing a display of the artworks, in 3D, a file cabinet with textual information, a workbench for inspecting the artist's material, and a video projector, with which the user can display a video-recorded interview with the artist.

The actual task to be performed by the user is to learn what constraints do apply to the installation of one of the artworks, Stone and Feather:

Stone and Feather

  • feather: 70 cm, from ostrich, curved
  • stone: 13.5 cm, white marble
  • position: alignment with pedestal, no glue
  • environment: 50 lux of light max.
The items mentioned in this list must be reproduced by the user in a subsequent memory test, and in another experiment the user must be able to choose the right materials and reconstruct the artwork.

Our assumption in designing this test scenario was that the gestural nature of positioning the artwork will be favorable for the condition with a gesturing agent, whereas believability will be positively affected by the degree of realism of the agent.



multimedia augmented theatre

In june 2003, I was asked to advise on the use of VR in a theatre production of the Odyssee. Lacking experience in this field, I accepted the invitation to participate with some reluctance, since at the time I didn't have any clue what the VR for the theatre production should look like. Nevertheless, I took the invitation as a challenge and started looking for appropriate hardware, bothering collegues for information on mixed reality art productions, and downloading code to explore software technologies. Many hurdles ware to be taken. We had to deal with organizational issues, such as finding the money for financing the actual production (which unfortunately never came through), finding the right people (students, in our case) to select material and contribute to the code; aesthetic issues, in particular to determine which approach to take to reach at an effective solution; and not in the least technical issues, to realize the production on a sufficiently efficient low-cost platform.



context -- the Odyssee theatre production

The Odyssee. theatre production was initiated by Ground Control, as a successor of previously succesful theatrical spectacles, including an open air performance of Faust. In effect, two performances of the Odyssee were planned, an out-door (external) version, involving real ships at the shore of a lake, and an in-door (internal) version, to be played in temporarily empty office buildings. The in-door version is meant to give a more psychological rendering of the Odyssee  [Entanaclaz (2003)], where the travels of Ulysses are experienced by the audience as a confrontation with themselves. Our contribution was asked for the in-door version, to enhance the experience of the audience with additional VR.

The Odyssee is a wellknown account of the travels of Ulysses leaving Troje, in 24 episodes ending in his return to Ithaca and his reunion with Penelope. The actual theatre production takes 12 parts which are played in 12 successive rooms through which the audience, subdivided in small groups, is guided one room after another for about five minutes per room. Our initial idea was to add information in the form of text and images, to direct the interpretation of the audience towards a particular perspective. In that beginning stage, somewhat optimistically, we planned to offer multiple perspectives to each participant, in an individualized manner, dependent on the actual focus of attention of the individual participant.

initial ideas -- VR and augmented reality:

Our first problem was to find suitable hardware, that is see-through goggles. Searching the Internet gave us the name of a relatively nearby company, Cyber Mind NL, that specialized in entertainment VR solutions. Both price-wise and in terms of functionality semi-transparent see-through glasses appeared to be no option, so instead we chose for simple LCD-projection goggles with a (head-mounted) low-resolution camera. This solution also meant that we did not need expensive head orientation tracking equipment, since we could, in principle, determine focus using captured image analysis solutions such as provided by the AR Toolkit. Moreover, captured video feed ensured the continuity and reactiveness needed for a true (first-person perspective) VR experience.

Augmented or mixed reality is an interesting area of research with many potential applications. However, in the course of the project we dropped our ambition to develop personalized presentations using image analysis, since we felt that the technology for doing this in a mixed reality theatre setting is simply not ripe, and instead we concentrated on using the captured video feed as the driver for text and image presentation. In addition, we developed image manipulation techniques to transform the (projection of the) captured video, to obtain more implicit effects, as to avoid the explicit semantic overload resulting from the exclusive use of text and images.

technological constraints -- the DirectX platform:

After a few experiments with the AR Toolkit, it soon appeared that the frame rate would not be sufficient, on the type of machines our budget would allow for. Moreover, reading the AR Toolkit mailing list, marker tracking in a theatrical context seemed to be more or less unfeasible. So, we shifted focus to the DirectX SDK 9, both for video capture and projection in 3D. The DirectX toolkit is a surprisingly functional, and very rich technology for multimedia applications, supporting streamed video, including live capture, 3D object rendering and precise synchronisation between multimedia content-related events. At that time, and still at the time of writing, our own intelligent multimedia technology was no option, since it does not allow for using live video capture and is also lacking in down-to-the-millisecond synchronisation.

After exploring texture mapping images copied from the incoming captured video stream, we decided to use the VMR-9 video mixing renderer introduced in DirectX 9, that allows for allocating 3D objects as its rendering surface, thus avoiding the overhead of explicit copies taken from a video processing stream running in a separate thread. See section 4.3. Although flexible and efficient, DirectX is a low-level toolkit, which means that we had to create our own facilities for processing a scenegraph, world and viewpoint transformations, and, even more importantly, structuring our mixed reality presentations in time.



structuring time -- maintaining 'see-through' aesthetics

One of the problems we encountered in discussing what we conveniently may call the VR with the producer of the Odyssee theatre performance was the high expectancy people have of VR, no doubt inspired by movies as the Matrix and the like. In mixed reality applications, manipulating persons, warps in space, and basically any intensive image analysis or image manipulation is simply not possible in real time. Moreover, there is a disturbing tendency with the layman to strive for semantic overload by overlaying the scene with multiple images and lines of text, thus obscuring the reality captured by the camera and literally blocking the participants view and awareness of the scene. Basically, as a guideline, we tend to strive for 70% visibility of the scene, 20% image or projection transformations and only 10% of information in the form of text and images.

The total duration of our presentation is only 2 minutes, or 118 seconds to be precise. We made a subdivision in 4 scenes, with transitions inbetween, hierarchically ordered in a tree-like structure. Initially, we abstracted from the actual duration, by taking only the fraction of the time passed (in relation to the total duration) as an indication for which scene to display. However, when the development reached its final stages, we introduced actual durations that allowed us to time the sequence of scenes to the tenth of a second. In addition, we used multiple layers of presentation, roughly subdivided in background captured image, the transformed captured image projected on 3D objects, and, finally, pictures and text. These layers are rendered on top of eachother, triggered in a time-based fashion, semi-independent of one another. The frame rate varies between 20 and 30, dependent on the number of images simultaneously used for texturing. Our final mixed reality theatre application may be considered a prototype, awaiting to be put to the test by the audience.



lessons learned -- our explorations revisited:

Altogether, the development of the mixed reality theatre application has been quite an experience, in multiple ways.

Not in the least it has been (and still is) a challenge to explain the possibilities of mixed reality applications to the layman, that do not take the abstractions we use in our daily academic life for granted.

To be frank, it also has opened my eyes to what some consider 'politically incorrect' technology, in other words Microsoft DirectX, originally developed as game technology, and no doubt a rich toolbox for real life multimedia applications.

Reinventing the wheel is not as simple as it seems. Nevertheless, developing scenegraph processing facilities and the appropriate timing mechanisms for controlling the mixed reality presentation was, apart from being a rekindling of basic skills, a learnful experience.

Finally, even before putting the application to the test, considering the aesthetics of mixed reality theatre productions, it may be called an eye-opener to realize how important reality is, and how meaningless explicit semantics (in the form of text and images) may become. Rather, our explorations were an incentive to further explore more implicit (graphical) modifications of the captured scene to convey meaning.



example(s) -- pizza boy

Did you like to go to a theatre play, when you were sixteen? Peter van Kessel, one of the speakers in our visual design develeped a game pizza boy to overcome the resistance of secondary school students to theatre. The game, developed by Peter's agency Headland, introduced elements of the theatre play in a game setting. The player must deliver pizzas, riding a scooter, and finds him/herself in a situations where he/she must enact a role or perform some actions, related to the theatre play. Visiting the theatre, there is a MIDI-enabled scooter with which the students can play the game, in preparation for the actual play (image on the right in the figure above). Peter reported that the game did indeed help to overcome the scepticism these yougsters had with theatre.

The game was implemented by the Little Chicken Game Company, with Virtools. It was available for download at the website of the governmental department that initiated the development of the game, but became such a huge success, also with other people than the original target group, that it had to be taken offline.

research directions -- computational art

I started studying computer music in 1979. For about four years, I worked on a system for real-time sound synthesis and algorithmic composition. Some years later, I wrote an essay that I presented in 1986 at the First Symposium for Electronic Arts, in Utrecht. The essay was the published in Leonardo,  [Eliens (1988)]. Now, almost 20 years later, I reread the abstract, which I OCRed from the original hardcopy:

Computational Art

The author conducts a simple thought experiment investigating the existence and scope of 'computational art': the utilization of the computer in the visual arts and music. In the experiment he sets the task of constructing an artifact that is capable of producing works of art. Since it appears that the artifact needs at least the capability of imagination, he queries the nature of images and imagery and argues that imagination is strongly intentional. Next he introduces the concept of notational systems, since they seem to govern the artistic activity of (not exclusively) machines. Confronted with the question of whether we are able to develop a computational analogue for taste, he finds that notational systems prove to be necessary for mediating the method of production of an artwork and the appraisal of its artistic value. Furthermore, the author shows that there are certain epistemological limits to the creativity of an imaginative device. Although the outcome of this hypothetical construction task clearly denies the possibility of an autonomously creative artifact, there seems to be no reason to worry about the opportunities for computational art: the computer appears to be a unique tool in exploring the possibilities of artistic production, guided by artists.

Today, despite the immense increase in computational power and the overwhelming amount of multimedia peripherals, the computer is still not able to produce art autonomously. But as a tool it is about to take over the entire industry, ranging from traditional publishing to film production and distribution. And, perhaps more tangible, as a tool for the creation of media content it is becoming ever better!

development(s) -- dialectic(s) of awareness

There is an overload of information. Usually we don't pay much attention to this. Increasingly, we rely on search (engines) to find the information, when needed. But often our search is a bit disappointing, simply because there are too many results. If only there was more focus in search.

To assist you in proving focussed search, using google, that limits search to your site, look at the following fragement of code.


  <form action="" method="GET">
  <input name="q" value=" ">
  <input type="text" name="q" size=40>
  <input type="submit" value=" "> 
Adapt the url, and put it in your web page, and, voila, your site is search-enabled.

On a deeper level, we may wonder why we are so impatient, in a hurry to search for information, and often not taking the time to properly digest it. In  [Eliens & Vyas (2007)] we wrote: In the course of our field study for the PANORAMA system, we tried to establish what relation users would have to the system, not only in the way they interact with it, but also in terms of what role the system plays in their lives, and when and how they would be aware of the systema. Due to the intrinsic properties of the PANORAMA system, as a system meant to support social awareness in a work environment, we could not assume direct focussed attention. Instead, we must take the various forms of awareness or attention into account.

Our thoughts in this direction were triggered by a lecture of Linda Stone (former vice-president of Microsoft) at the Crossmedia Week September 2006 in Amsterdam, entitled Attention -- the Real Aphrodisiac. In that lecture Linda Stone made a distinction between applications popular before 1985, applications which were in general meant for self-improvement, for example language-learning, applications that were popular between 1985 and 2005, applications that she characterized as supporting continuous partial awareness, such as email and news-feeds, and applications of the period thereafter, from now into the future, which may be characterized as applications that allow the user to be creative, take part in a community, and are in other words more focussed and less dependent on the external environment.

Admittedly, it takes a few more steps to formulate a theory of the dialectics of awareness. However, with the function of the PANORAMA system in mind, we may make, following  [Benjamin (1936)], some interesting distinctions between the experience of art and architecture. Where art is usually experienced in a delimited time span, and is similarly delimited in space, that is the position of the observer, architecture is everywhere and always there. As a consequence, art receives focussed attention and may be appreciated with reflective distance, whereas architecture is often not perceived consciously, but merely present and subject to an almost sub-conscious sensibility, which is only brought to the focus of attention when it is either aesthetisized, for example when taking photographs, or when something surprising is sensed, for example in the change of skyline in New York.

As argued in  [Hallnäss and Redström (2002)], many of the new interactive systems, whether in the category of ambient media, ubiquitous computing or calm technology, will fall somewhere inbetween the spectrum spanned by art and architecture, or more likely even alternate between the forms of awareness associated with respectively art and architecture.

In designing the new interactive systems and games, we need to be explicitly concerned with the actual phases of awareness that occur, simply because it is not clear what role these systems play in our life. When introducing a new system or artefact, we may distinguish between the following phases:

phases of awareness

  • initiation -- appeal to curiosity
  • promotion -- raising interest
  • progression -- prolonged involvement
As designers we must ask ourselves the following questions. How do we appeal to the users' curiosity, so that our system is noticed? How do we get a more sustained interest? How de we get the user to interact with or contribute to the system? And, how do we obtain prolonged involvement, and avoid boredom? These questions are not simple to answer, and require also an understanding of the actual context in which the system is deployed as well as an understanding of the level of (aesthetic) literacy of the user(s).

Aesthetic awareness is common to us all,  [Saw (1971)]. Having an understanding of aesthetic awareness, can we isolate the relevant design parameters and formulate rules of composition that may help us in developing interactive applications? According to our philosophical credo,  [Eliens (1979)], no! However, the history of art clearly shows the impact of discoveries, such as the discovery of perspective, as well as conventions in the interpretation of art, as for example in the iconic representation of narrative context in 17th century Dutch painting. Moreover, the analysis of the visual culture of mass media may also give us better understanding of the implied meaning of compositional structures.

The notion of perspective, described in  [Alberti (1435)], is an interesting notion in itself, since it describes both the organisation of the image as well as the optimal point of view of the viewer. The normal perspective as we know it is the central perspective. However, there are variants of perspective that force the viewer in an abnormal point of view, as for example with anamorphisms.

Perspective had an enormous impact on (western) art and visual culture. It defines our notion of naturalist realism, and allowed for the development of the panorama as a mass medium of the 19th century,  [Grau (2003)]. Art that deviated from central perspective, such as cubism or art from other cultures, was often considered naive. Photography and its pre-cursors had a great impact on the perfection of perspectivist naturalism, and what is called photorealism became the touchstone of perfection for early computer graphics,  [Bolter and Grusin (2000)].

Apart from perspective, other conventions regulate the composition of the 2D image, in particular, following  [Kress and van Leeuwen (1996)], the information value related to where an object is placed in the image, and the salience of the object, determined by its relative size, being foreground or background, and visual contrast. Also framing is used to emphasize meaning, as for example in the close-up in a movie shot. In analysing a large collection of image material,  [Kress and van Leeuwen (1996)], somewhat surprisingly found that lef/right positioning usually meant given versus new, top/bottom positioning ideal versus real, and centre/margin positioning important versus marginal. It is doubtful whether these meaning relationships hold in all cultures, but as a visual convention it is apparently well-rooted in western visual culture.




9. digital content creation



projects & further reading

As a project, you may develop a dialog engine for non-linear interactive story telling or a collage generator, that produces artworks from a collection of images.

You may further explore the various presentation platforms, and assess the tradeoffs with respect to the support they offer for authoring.

For further reading, I suggest to study interaction design patterns. It is also worthwhile to get some books on modern art, to gain some knowledge about art and design.

the artwork

  1. street logos -- images from  [Manco (2004)].
  2. photograph of oilpaint box.
  3. Mark Veldhuijzen van Zanten -- the six roles in their agency.
  4. Mark Veldhuijzen van Zanten -- to design for the salon, periodic lounge evenings in musea and art institutes.
  5. Geert Mul -- interactive multimedia installation.
  6. Geert Mul -- multimedia installation in dutch consulate in India.
  7. website of Institute of Creative Technologies, showing scenes from Mission rehearsal Exercise (MRE).
  8. street logos -- more images from  [Manco (2004)].
  9. website for Visual Sensations, a yearly VJ contest in the Netherlands, developed by the agency of mark Veldhuijzen van Zanten.
  10. Geert Mul -- Harbour Sound & Vision, 1999
  11. screenshots from virtual atelier of Marinus Boezem.
  12. left: don't spit, a chines poster against spitting during the SARS period, taken from dutch newspaper; right: filmteckarna,  [Wiedermann (2004)].
  13. sketches -- from filmteckarna,  [Wiedermann (2004)].
  14. sketches -- from filmteckarna,  [Wiedermann (2004)].
  15. sketches -- from filmteckarna,  [Wiedermann (2004)].
  16. game -- pizza boy, developed by Headland, see 9.3.
  17. signs -- health and safety,  [ van Rooijen (2003)], p. 258, 259
The artwork for this chapter is meant to emphasize context. The street logos opening this chapter, as well as the work of Mark veldhuijzen van Zanten and Geert Mul, must be experienced in a context to fully appreciate their meaning.

Also for the MRE application, it is the context, in this case the stress and anxiety of a war situation, that determines the impact. The photorealistic graphic style of MRE, wellknown by the trainees from other games, is meant to strengthen the experience of immersion. Notice that the street logos assume an almost iconic character.

(C) Æliens 23/08/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.