topical media & game development

talk show tell print

entertainment and experience

The question of what is multimedia is rather elusive. We may, nevertheless, look at how the phrase multimedia is used, and how the concept multimedia is related to other concepts. as in the concept graphs that may be obtained with the Visual Thesaurus, providing as input multimedia. We then see that the notion of multimedia is related to systems, in particular interactive and hypermedia systems, and indirectly also to the notion of transmission, which will even become more apparent when we inspect the graph for the concept of medium, depicted in figure X below.

However, although this gives us some indication of how to position multimedia in the larger area of computer applications, in particular when exploring the systems node, it does not so much tell us what multimedia is all about.

From the perspective of human cognition, we may look at how multimedia contributes to our understanding of ourselves and the world around us. Traditionally, three levels of cognitive functioning are distinguished,  [Education], corresponding with three levels of meaning:

levels of meaning

Multimedia is clearly (most strongly) related to the sensory/iconic level, although for games one could say there is also a strong relation with the actionary level, and to some extent (for both multimedia and games) with the symbolic level.

For a more serious and deep understanding of how multimedia artefacts provide meaning and what role they play in our daily life, or how that meaning is affected by social contexts, we need to take recourse to semiotic theory, which is now one step too far, buth which we will look at in chapter 12.

Antother perspective from which to understand the meaning of multimedia, is to look at the function of media in our society, or, in other words, how multimedia is situated in our cultural institutions.

Consider this quote from the preface of all of all MIT books in the Leonardo series:

cultural convergence

The cultural convergence of art, science, and technology provides ample opportunity for artists to challenge the very notion of how art is produced and to call into question its subject matter and its function in society.

Although the quote is about art, it is essentially related to multimedia, to the extent that the quote refers to media art. The MIT Media Lab is one of the worlds most famous institutes in the field of multimedia. The Leonardo series is a collection of authoritive books on multimedia and related topics, which includes  [DeepTime],  [VirtualArt],  [InformationArts].

To understand the position of (computer supported) media in our society, we may observe following  [DeepTime]: there are two forces, political and technological, and there is, currently, a trend towards standardization and uniformity

standardization and uniformity

  1. Telematic media were incorporated very quickly in the globalization strategies of transnational corporations and their political administrators and they became increasingly dependent on existing power structures.
  2. At the other end of the scale, there were individuals, or comparatively small groups, who projected great hopes onto these networks as a testing ground for cultural, artistic and political models that would give greater prominence and weight to divergence and plurality.
This reflects what  [DeepTime] calls the advanced media paradox, facilitating heterogeneity and immersion on the one hand, and striving for universalisation on the other hand, as demanded by the centers of technological and political power.

Leaving the socio-political arena, we may in some sense predict the tension between convergence and divergence, by looking at the meaning context of the concept of convergence, again using the Visual Thesaurus, where we find that not only notions such as overlap and occurrence are related to it, but also the complementary concept of divergence. However, instead of speculating on the meaning of words, it might be more worthwhile to look at what we may consider to be the recent history of multimedia, entertainment.


In november 2000, a theme issue of the Scientific American appeared, featuring a number of articles discussing (digital) entertainment in the era of digital convergence. Let's start with a quote:

Scientific American (november 2000)

The barriers between TV, movies, music, videogames and the Internet are crumbling.

Audiences are fetting new creative options.

Here is what entertainment could become if the technological and legal hurdles can be cleared ...

Moreover, the editors made some wildly speculative claims, such as digitizing everything audio and video will disrupt the entertainment industry's social order, and the whole concept of holding a CD or movie in your hand will disappear once d-entertainment is widely available. To some extent this seems already to be true, as for example the music industry can painfully testify to.

Underlying the importance of entertainment in the era of digital convergence is the premisse governing an entertainment economy, which may be stated as

there is no business without show business

Additionally, the authors of the introduction to the theme issue speculate that the creation of content will be democratized, due to the availability of low cost digital movie cameras and PC video editors. Producing a video movie is now possible for just a few thousend euro or dollars. However, given the aesthetic ignorance of the average individual making video movies, it seems doubtful that this will hold true for entertainment in general.

In that same issue of the Scientific American, Gloria Davenport, a pioneer in the field of multimedia, presents list of applications characterizing the evolution of digital entertainment,  [Entertainment]:

evolution of digital entertainment

  • 1953: Winky Dink (CBS) -- interactive television, drawing exercise
  • 1972: Pong (Atari) -- ping-pong on computer screen
  • 1977: Adventure -- text-based interactive fiction
  • 1983: Dragon's Liar -- laser-disc technology 3D game
  • 1989: SimCity -- interactive simulation game
  • 1989: Back to the Future -- the Ride
  • 1993: Doom -- 3D action game
  • 1995: The Spot -- interactive web-based soap opera (Webisodic)
  • 1999: IMAX3D -- back to Atlantis (Las Vegas)
  • 2000: Big Brother -- TV + around the clock Web watch + voting
  • 2001: FE Sites -- fun enhanced web sites

It is interesting to note that Big Brother, which was originally created by a Dutch team, has become a huge success in many countries. Although the integration with the web was limited, it may be seen as the start of a number of television programs with web-based interaction facilities.

digital experience

The list compiled by Gloria Davenport suggests, a convergence towards an 'ultimate digital experience', Now, what does digital experience mean?

In a special issue of the Communications of the ACM, about the next 1000 years of computing, Ramesh Jain makes the following observation,  [Experience]:

experience is fundamental to human life

The desire to share experiences will be the motivating factor in the development of exciting multimedia technology in the foreseeable future.

Considering the variety of means we have at our disposal to communicate, as reflected in the list below, we may wonder whether our current technology really stands out as something special.

communication technology

  • oral -- communicate symbolic experiences
  • writing -- record symbolic experiences
  • paper -- portability
  • print -- mass distribution
  • telegraph -- remote narrow communication
  • telephone -- remote analog communication
  • radio -- analog broadcasting of sound
  • television -- analog A/V broadcasting
  • recording media -- analog recording
  • digital processing -- machine enhancement
  • internet -- multimedia communication
According to Ramesh Jam, internet-based multimedia communication differs from earlier communication technology in that it somehow frees the message from the medium. Reflecting on Marshall McLuhan phrase -- the medium is the message -- he observes that:

the medium was the message when only one medium could be used to communicate messages.

Now, that the Internet allows the synthesis and rendering of information and experiences using whatever is the most appropriate media to convey the message, the message is, as Jain phrases it, just the message, and the medium is just the medium. In other words, the medium itself does not seem to constrain what message can be conveyed. Looking at the documentary Fahrenheit 9/11 though, we may seriously doubt whether this is true. Although it is possible to gain knowledge about the alliances that underly politics, even in the age of the internet, the television campaigns seem to be more dominant in affecting the general publics opionion about global politics than anything else, due to the conventional formats of presentation and editing.



Let's once more look at a graph, above, indicating the concept relations for the notion of medium. What strikes me as important are the relations with the distinct concepts of substance, communication, environment, and intermediate. In some respects the notion of medium, underlying the plural use of it in multimedia is comparable to the notion of ether, which was once seen as a vehicle for the transport of broadcasted information. But I also like to stress the 'substantial' aspect of multimedia, as a material for design and creation, similar to paint.

The basic issue here is what is a medium and how does it affect, or even shape our experience(s). Following Ramesh Jain, we may speculate that the range of sensory information offered by multimedia applications may become much richer than is currently the case, and we may then predict that there will be a tremendous progress in presentation technology, multisensory presentation technology! Clearly, from a technological perspective there seems to be no limit, except those imposed by our own phantasy. However, it should be equally obvious that compelling experiences rely on carefully staged presentations, and as such require an entirely new discipline of design.


VR for pain reliefimage delivery system


example(s) -- VR for pain relief

The research project fMRI Research on Virtual Reality Analgesia at the Human Interaction Laboratory (Washington) has explored the use of VR to reduce the agonay of taking MRI scans. The U.W Radiology Digital Imaging Science Centers wide field of view magnet-friendly virtual reality image delivery system makes it possible for volunteers and patients to have the illusion of going into virtual reality during fMRI brain scans. As explained on the website, the image on the left above, shows a woman in virtual reality during an fMRI brain scan, looking into a custom magnet-friendly virtual reality goggles. VR images from projectors in another room are carried to the participant in the form of light (photons, no electrons) via optic fiber image guides. The participant has the illusion of going inside the virtual world, allowing researchers to measure what happens to her brain when she reports reductions in pain during VR. The white cage-like structure around the womans head, in the image on the right, shows fMRI receiver coils used by the fMRI brain scanner to collect the information about changing patterns of brain activity.

Another project investigating the use of VR techniques for pain distraction can be found at the site of the Virtual Environments of the Georgia Institute of Technology, Atlanta.

research directions -- the face of cyberspace

The notion of cyberspace was introduced in William Gibson's novel Neuromancer, that appeared in the early 1980's, signifying a vast amount of (digital) data that could be accessed only through a virtual reality interface that was controlled by neuro-sensors. Accessing data in cyberspace was not altogether without danger, since data protection mechanisms (including firewalls, as we call them nowadays) were implemented using neuro-feedback. Although the vision expressed in Neuromancer is (in our days) still futuristic, we are confronted with a vast amount of information and we need powerful search engines and visualisation techniques not to get lost. So what is the reality of cyberspace today?

... cyberspace is a construct in terms of an electronic system.

as observed by Vivian Sobschack, 1996, quoted from  [History], p. 321. On reflection, our (electronic) world of today might be more horrendous than the world depicted in Neuromancer. In effect,


television, video cassettes, video tape-recorder/players, video games, and personal computers all form an encompassing electronic system whose various forms interface to constitute an alternative and absolute world that uniquely incorporates the spectator/user in a spatially decentered, weakly temporalized and quasi-disembodied state.

All these gadgets make us dizzy, stoned with information and fried by electro-magnetic radiation. However, the reality of everyday computer use is (fortunately?) less exciting than the images in Neuromancer suggest. User interfaces are usually tiresome and not at all appealing. So except for the fanatic, the average user does easily get bored. Would this change when virtual reality techniques are applied pervasively? What is virtual reality?

virtual reality

virtual reality (is) when and where the computer disappears and you become the 'ghost in the machine' ...

In other words, virtual reality is a technology that provokes immersion, sensuous immersion, supported by rich media and powerful 3D graphics. In our age of information, we may wonder how all that information should be presented. Rephrasing the question, we may ask what are the limits of the digital experience, or more importantly, what should be the norm: 3D virtual environments, plain text, or some form of XP?

(C) Æliens 04/09/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.