topical media & game development

talk show tell print

print / present


digital culture

life is becoming digital

learning objectives

After reading this chapter you should be able to define the notion of multimedia, recount the history of digital entertainment, explain the concept of digital convergence, discuss the future of cyberspace, and speculate about the commercial viability of mobile multimedia.

We live in the digital era,  [Negroponte (1995)]. We are surrounding ourselves with gadgets and we are consuming immense amounts of information, that is increasingly being delivered to us via the Internet. We play games, and we still watch (too much) television.

Some of us watch televion on our PCs, and may be even looking forward to watch television on their mobile phone. This is multimedia. For others, the PC is still a programmable machine. Being able to program it might earn you a living. Understanding multimedia, however, might even provide you with a better living.

In this chapter, we study what trends may currently be observed in the creation and delivery of multimedia information, and we explore what impact the digital revolution may have from a commercial perspective.



entertainment and experience

The question of what is multimedia is rather elusive. We may, nevertheless, look at how the phrase multimedia is used, and how the concept multimedia is related to other concepts. as in the concept graphs that may be obtained with the Visual Thesaurus, providing as input multimedia. We then see that the notion of multimedia is related to systems, in particular interactive and hypermedia systems, and indirectly also to the notion of transmission, which will even become more apparent when we inspect the graph for the concept of medium, depicted in figure X below.

However, although this gives us some indication of how to position multimedia in the larger area of computer applications, in particular when exploring the systems node, it does not so much tell us what multimedia is all about.

From the perspective of human cognition, we may look at how multimedia contributes to our understanding of ourselves and the world around us. Traditionally, three levels of cognitive functioning are distinguished,  [Bruner (1972)], corresponding with three levels of meaning:

levels of meaning

  • actionary level -- action and movements
  • sensory/iconic level -- images and impressions
  • symbolic level -- language and mathematics
Multimedia is clearly (most strongly) related to the sensory/iconic level, although for games one could say there is also a strong relation with the actionary level, and to some extent (for both multimedia and games) with the symbolic level.

For a more serious and deep understanding of how multimedia artefacts provide meaning and what role they play in our daily life, or how that meaning is affected by social contexts, we need to take recourse to semiotic theory, which is now one step too far, buth which we will look at in chapter 12.

Antother perspective from which to understand the meaning of multimedia, is to look at the function of media in our society, or, in other words, how multimedia is situated in our cultural institutions.

Consider this quote from the preface of all of all MIT books in the Leonardo series:

cultural convergence

The cultural convergence of art, science, and technology provides ample opportunity for artists to challenge the very notion of how art is produced and to call into question its subject matter and its function in society.

Although the quote is about art, it is essentially related to multimedia, to the extent that the quote refers to media art. The MIT Media Lab is one of the worlds most famous institutes in the field of multimedia. The Leonardo series is a collection of authoritive books on multimedia and related topics, which includes  [Zielinski (2006)],  [Grau (2003)],  [Wilson (2002)].

To understand the position of (computer supported) media in our society, we may observe following  [Zielinski (2006)]: there are two forces, political and technological, and there is, currently, a trend towards standardization and uniformity

standardization and uniformity

  1. Telematic media were incorporated very quickly in the globalization strategies of transnational corporations and their political administrators and they became increasingly dependent on existing power structures.
  2. At the other end of the scale, there were individuals, or comparatively small groups, who projected great hopes onto these networks as a testing ground for cultural, artistic and political models that would give greater prominence and weight to divergence and plurality.
This reflects what  [Zielinski (2006)] calls the advanced media paradox, facilitating heterogeneity and immersion on the one hand, and striving for universalisation on the other hand, as demanded by the centers of technological and political power.

Leaving the socio-political arena, we may in some sense predict the tension between convergence and divergence, by looking at the meaning context of the concept of convergence, again using the Visual Thesaurus, where we find that not only notions such as overlap and occurrence are related to it, but also the complementary concept of divergence. However, instead of speculating on the meaning of words, it might be more worthwhile to look at what we may consider to be the recent history of multimedia, entertainment.


In november 2000, a theme issue of the Scientific American appeared, featuring a number of articles discussing (digital) entertainment in the era of digital convergence. Let's start with a quote:

Scientific American (november 2000)

The barriers between TV, movies, music, videogames and the Internet are crumbling.

Audiences are fetting new creative options.

Here is what entertainment could become if the technological and legal hurdles can be cleared ...

Moreover, the editors made some wildly speculative claims, such as digitizing everything audio and video will disrupt the entertainment industry's social order, and the whole concept of holding a CD or movie in your hand will disappear once d-entertainment is widely available. To some extent this seems already to be true, as for example the music industry can painfully testify to.

Underlying the importance of entertainment in the era of digital convergence is the premisse governing an entertainment economy, which may be stated as

there is no business without show business

Additionally, the authors of the introduction to the theme issue speculate that the creation of content will be democratized, due to the availability of low cost digital movie cameras and PC video editors. Producing a video movie is now possible for just a few thousend euro or dollars. However, given the aesthetic ignorance of the average individual making video movies, it seems doubtful that this will hold true for entertainment in general.

In that same issue of the Scientific American, Gloria Davenport, a pioneer in the field of multimedia, presents list of applications characterizing the evolution of digital entertainment,  [Davenport (2000)]:

evolution of digital entertainment

  • 1953: Winky Dink (CBS) -- interactive television, drawing exercise
  • 1972: Pong (Atari) -- ping-pong on computer screen
  • 1977: Adventure -- text-based interactive fiction
  • 1983: Dragon's Liar -- laser-disc technology 3D game
  • 1989: SimCity -- interactive simulation game
  • 1989: Back to the Future -- the Ride
  • 1993: Doom -- 3D action game
  • 1995: The Spot -- interactive web-based soap opera (Webisodic)
  • 1999: IMAX3D -- back to Atlantis (Las Vegas)
  • 2000: Big Brother -- TV + around the clock Web watch + voting
  • 2001: FE Sites -- fun enhanced web sites

It is interesting to note that Big Brother, which was originally created by a Dutch team, has become a huge success in many countries. Although the integration with the web was limited, it may be seen as the start of a number of television programs with web-based interaction facilities.

digital experience

The list compiled by Gloria Davenport suggests, a convergence towards an 'ultimate digital experience', Now, what does digital experience mean?

In a special issue of the Communications of the ACM, about the next 1000 years of computing, Ramesh Jain makes the following observation,  [Jain (2000)]:

experience is fundamental to human life

The desire to share experiences will be the motivating factor in the development of exciting multimedia technology in the foreseeable future.

Considering the variety of means we have at our disposal to communicate, as reflected in the list below, we may wonder whether our current technology really stands out as something special.

communication technology

  • oral -- communicate symbolic experiences
  • writing -- record symbolic experiences
  • paper -- portability
  • print -- mass distribution
  • telegraph -- remote narrow communication
  • telephone -- remote analog communication
  • radio -- analog broadcasting of sound
  • television -- analog A/V broadcasting
  • recording media -- analog recording
  • digital processing -- machine enhancement
  • internet -- multimedia communication
According to Ramesh Jam, internet-based multimedia communication differs from earlier communication technology in that it somehow frees the message from the medium. Reflecting on Marshall McLuhan phrase -- the medium is the message -- he observes that:

the medium was the message when only one medium could be used to communicate messages.

Now, that the Internet allows the synthesis and rendering of information and experiences using whatever is the most appropriate media to convey the message, the message is, as Jain phrases it, just the message, and the medium is just the medium. In other words, the medium itself does not seem to constrain what message can be conveyed. Looking at the documentary Fahrenheit 9/11 though, we may seriously doubt whether this is true. Although it is possible to gain knowledge about the alliances that underly politics, even in the age of the internet, the television campaigns seem to be more dominant in affecting the general publics opionion about global politics than anything else, due to the conventional formats of presentation and editing.



Let's once more look at a graph, above, indicating the concept relations for the notion of medium. What strikes me as important are the relations with the distinct concepts of substance, communication, environment, and intermediate. In some respects the notion of medium, underlying the plural use of it in multimedia is comparable to the notion of ether, which was once seen as a vehicle for the transport of broadcasted information. But I also like to stress the 'substantial' aspect of multimedia, as a material for design and creation, similar to paint.

The basic issue here is what is a medium and how does it affect, or even shape our experience(s). Following Ramesh Jain, we may speculate that the range of sensory information offered by multimedia applications may become much richer than is currently the case, and we may then predict that there will be a tremendous progress in presentation technology, multisensory presentation technology! Clearly, from a technological perspective there seems to be no limit, except those imposed by our own phantasy. However, it should be equally obvious that compelling experiences rely on carefully staged presentations, and as such require an entirely new discipline of design.


VR for pain reliefimage delivery system


example(s) -- VR for pain relief

The research project fMRI Research on Virtual Reality Analgesia at the Human Interaction Laboratory (Washington) has explored the use of VR to reduce the agonay of taking MRI scans. The U.W Radiology Digital Imaging Science Centers wide field of view magnet-friendly virtual reality image delivery system makes it possible for volunteers and patients to have the illusion of going into virtual reality during fMRI brain scans. As explained on the website, the image on the left above, shows a woman in virtual reality during an fMRI brain scan, looking into a custom magnet-friendly virtual reality goggles. VR images from projectors in another room are carried to the participant in the form of light (photons, no electrons) via optic fiber image guides. The participant has the illusion of going inside the virtual world, allowing researchers to measure what happens to her brain when she reports reductions in pain during VR. The white cage-like structure around the womans head, in the image on the right, shows fMRI receiver coils used by the fMRI brain scanner to collect the information about changing patterns of brain activity.

Another project investigating the use of VR techniques for pain distraction can be found at the site of the Virtual Environments of the Georgia Institute of Technology, Atlanta.

research directions -- the face of cyberspace

The notion of cyberspace was introduced in William Gibson's novel Neuromancer, that appeared in the early 1980's, signifying a vast amount of (digital) data that could be accessed only through a virtual reality interface that was controlled by neuro-sensors. Accessing data in cyberspace was not altogether without danger, since data protection mechanisms (including firewalls, as we call them nowadays) were implemented using neuro-feedback. Although the vision expressed in Neuromancer is (in our days) still futuristic, we are confronted with a vast amount of information and we need powerful search engines and visualisation techniques not to get lost. So what is the reality of cyberspace today?

... cyberspace is a construct in terms of an electronic system.

as observed by Vivian Sobschack, 1996, quoted from  [Briggs and Burke (2001)], p. 321. On reflection, our (electronic) world of today might be more horrendous than the world depicted in Neuromancer. In effect,


television, video cassettes, video tape-recorder/players, video games, and personal computers all form an encompassing electronic system whose various forms interface to constitute an alternative and absolute world that uniquely incorporates the spectator/user in a spatially decentered, weakly temporalized and quasi-disembodied state.

All these gadgets make us dizzy, stoned with information and fried by electro-magnetic radiation. However, the reality of everyday computer use is (fortunately?) less exciting than the images in Neuromancer suggest. User interfaces are usually tiresome and not at all appealing. So except for the fanatic, the average user does easily get bored. Would this change when virtual reality techniques are applied pervasively? What is virtual reality?

virtual reality

virtual reality (is) when and where the computer disappears and you become the 'ghost in the machine' ...

In other words, virtual reality is a technology that provokes immersion, sensuous immersion, supported by rich media and powerful 3D graphics. In our age of information, we may wonder how all that information should be presented. Rephrasing the question, we may ask what are the limits of the digital experience, or more importantly, what should be the norm: 3D virtual environments, plain text, or some form of XP?

technological developments

Let's see if we are able to give a more precise characterization of digital convergence. In their introduction to the theme issue of the Scientific American, Forman and SaintJohn locate the beginning of digital convergence, historically, at the 1939 New York World Fair, and more in particular the RCA Pavillion, which should be considered as the formal debut of television broadcast. They observe that


the receiver at the RCA Pavillon was way ahead of its time, it was a combination of television - radio - recorder - playback - facsimile - projector ...

Moreover, they remark that this in hindsight suggests that we humans have a fundamental desire to merge all media in one entity.

By way of definition we may state, following Forman and SaintJohn, that digital convergence is:

digital convergence

the union of audio, video and data communication into a single source, received on a single device, delivered by a single connection

And, as they say, predicted for decades, convergence is finally emerging, albeit in a haphazard fashion.

Taking a somewhat closer look, we may discern subsidiary convergences with respect to content, platform and distribution:

subsidiary convergences

  • content -- audio, video, data
  • platform -- PC, TV, internet, game machine
  • distribution -- how it gets to your platform
Here, Forman and SaintJohn continue by speculating that if compatibility standards and data protection schemas can be worked out, all d-entertainment will converge into a single source that can shine into your life on any screen, whereever you are ... However, observe that the number of competing standards and architectures is enormous, and that apart from the technical issues involved it is not entirely clear what business model should underly such convergence. In computer shops, there PCs with TV receivers are sold in the range of 1000-2000 euro. This does not include the screen. They come with either the XP Home or Windows Media Center. One of the first in this line of machines, in the higher prices range, was the Sony W1.



TV or PC

It is fair to say that no device has changed the way we live so dramatically as television. Television, for one, has altered the way we furnish our living rooms, not to speak about the time we waste watching the thing. Comparing the graphs for television and communication, we immediately see that their underlying concepts are very different. And more specifically, the association of television with a phrase such as idiot box may raise doubt whether the promise of convergence, which does include communication as an essential feature, will ever become a reality.

Now, we may wonder what interactive television and enhanced televison have to offer us. Looking back, we may observe that it takes some time for the new possibilities to catch on. For example, interactive television was introduced in 1970, but apparently people did not want to communicate with the broadcaster. As another example of enhanced televison, take Big Brother. Although many people watched Big Brother when it first appeared on television, the willingness of the audience to react other than by phone was (apparently) somewhat disappointing. Perhaps, in the Netherlands this was due to the fact that only a fraction of the PC owners was, at that time, permanently online.

In spite of the failed experiments, Forman and SaintJohn state, somewhat optimistically, that the convergence of digital content, broadcast distribution and display platforms create the big convergence of d-entertainment and information with feedback supporting human interactivity.

Before looking at digital television more closely, let's summarize what digital convergence involves:


  • content -- 2D/3D graphics, data, video, audio
  • distribution -- broadcast, wireless, DVD, internet, satelite, cable
  • platform -- PC, television, game machine, wireless data pad, mobile phone

This summary indicates the technical opportunities, and the possible functinal extensions that may enhance our use of television, computer, game console and mobile phone. As concerns digital television, we may come up with some immediate advantages, such as enhanced resolution, a multiplication of channels, and (more relevant to the issue of convergence) interactive television.


exposition on the history of TV in Institute for Time-based Arts/Montevideo


To get you failiar with some common acronyms, when speaking about (digital) television, we must make a further distinction between:


  • HDTV -- high definition television
  • SDTV -- standard definition television
  • ITV -- interactive television
As further discussed in chapter 3, we have (standard) codecs for d-TV, in particular MPEG-2, for recording digital video, and MPEG-4, for high-quality streaming video on the internet, both from the Motion Picture Expert Group, that enable the effective delivery of digital video, possibly in combination with other content.

Unfortunately, experts disagree on what might become the most suitable appliance or platform to consume all those digital goodies. Here are some possible choices:

a killer d-TV appliance ...

  • personal television -- TiVo, Replay-TV (MPEG-2 cache)
  • game machine -- Sony PS 2/3, X-Box
Will we prefer to watch stored video, instead of live televison broadcasts? Will the Internet be able to compete with traditional television broadcasting. Will DelayTV or Replay-TV, which allows you to watch previous broadcasts at a time that suits you become popular? Will an extended game machine or PC replace your television? Currently,we must observe that streaming media (still) have rather poor resolution.

Leaving game machines aside, will it then be the TV or PC that will become our platform of choice? Forman and SaintJohn observe:

TV or PC

The roadblock to the Entertainment PC could be the PC itself. Even a cheap TV doesn't crash or freeze. The best computers still do.

However, they conclude that it might make sense to adopt a programmable PC that can support competing TV standards, rather than construct a stack of TV peripherals. Nevertheless, there are a number of problems that occur when we (collectively) choose for the PC as our platform for d-entertainment. Should we have thin clients, for example based on the Sun/Java platform or so-called fat clients based on some version of Microsoft windows> How do we handle the fact that the current internet protocols are not robust, and how can we provide what is known as quality of service? Should we adopt any of the proprietary architectures and codecs, such as RealVideo, QuickTime, Windows media, or should we adhere to an open standard such as MPEG-4?

Evidently, the situation becomes even more complex when we just consider the range of alternatives for connectivity, that is for possible ways of distributing contents:


  • telephone network -- from 0.5 - 2 Mbps to 60 Mpbs (2.5km)
  • broadcast TV -- 6 MHz / 19 Mbps (4 channels MPEG HDTV)
  • cable TV -- hybrid fiber-optic coaxial cable 6 Mbps
  • fixed wireless -- 2 Mbps (radiotowers + rooftop antenna), phones/handhelds
  • satellite -- downloads to 100kbps, modem for uploads ...
Most probably, convergence with respect to distribution will not result in one single way of being connected, but rather a range of options from which one will be selected transparently, dependent on content and availability.

Let's stay optimistic, and ask ourselves the following question:

digital convergence

what will we do with convergence once we have it?

One possible scenario, not too unlikely after all, is to deploy it for installing computing devices everywhere, to allow for, to name a few, smart houses, smart clothes, or, in other words, to create a smart world. I wonder what a smart world will look like. In the end we will have to wait and see, but whatever will emerge


we will watch

That is to say, it is not likely that we will have a world without television. Television as we are used to it seems to be the dominant paradigm for d-entertainment, for both the near and distant future.


Berkeley meshSan Francisco viewaugmented terrain map


example(s) -- visible world

Just imagine that every visible place on earth would be accessible in a virtual world. Researchers of the Georgia Institute of Technology, Atlanta, have developed software for the semi-automated construction of detailed interactive urban environments, that takes data from multiple sources, including geo-corrected imagery from aerial photography and satelites and ground-based close-ups,  [Rosenblum and Macedonia (2002)].

The challenge here is to collect data from multiple sources and convert this into models, and perhaps even more difficult, to make the models visible so that they can be navigated in an interactive fashion. Recently, the Georgia group teamed up with a group from Berkeley to develop more complex models (images on the left), and together they are working on automating the extraction of information from aerial pictures (image on the right), in particular the detection of groups of trees, and height estimation.

There are many applications for such technology, including urban planning, emergency response, tourism and entertainment, military operations, traffic management, construction and maintenance, mobile services, citizen-government relations, and (not in the least) games.

The next step might be to connect the cameras, that are already there in many of these places, to the model, to observe what happens there in real life. But, somehow, this vision becomes frightening.

However, if you want to give it a try yourself, and populate the virtual globe with your own creations, go download the viewer and editing tool from Google Earth:

Google Earth

and read the tutorials!

research directions -- technological determinism

Although there are many technical issues involved in (digital) multimedia, as exemplified in the issues that play a role in digital convergence, a technical perspective alone does not suffice. Each technological innovation has its consequences on our social life. Conversely, each trend in society might result in the adoption or development of new technology. Looking at the history of the media, we may observe that media become materials in our social processes. Or, as phrased in  [Briggs and Burke (2001)]:

media as materials

each medium of communication tended to create a dangerous monopoly of knowledge

For example ( [Briggs and Burke (2001)], p. 8) for Christians, images where both a means of conveying information and a means of persuasion, that is part of the rethorics of institutionalized religion.

Looking at our age, and the media that have come into existence in the previous century (radio, television, ...),  [Briggs and Burke (2001)] observe that:

technological determinism

technological determinism was not the answer, ... more attempts were to be made to provide answers about the social consequences of television than had ever been asked about radio.

In effect, underlying all developments in the media (including the computer) we may assume a basic need for information. A rather problematic need, for that matter:


Information became a major concern anywhere during the late 1960 and 1970s where there was simultaneous talk both of 'lack of information' and 'information saturation'.

 [Briggs and Burke (2001)], p. 555

Nowadays, we regard information as a commodity. Train schedules, movies, roadmaps, touristic information, stock prices, we expect it all to be there, preferably online, at no cost. No information, no life. Information drives the economy. Upwards and downwards!

multimedia applications

In many stores there is a multimedia section. In some stores you will see B-movies being announced as multimedia topper. In other stores, the multimedia sections has a large offering of computer peripherals, ranging from DVD-RW drives to webcams and TV on PC hardware. Elsewhere you may buy authoring packages to organize your cell-phone photos, your family photo and video album, to create your personal archive on DVD. All this might make you wonder whether multimedia is serious business. See figure (a) and (b), illustrating our personal memex, as explained below.

But more seriously, what is the commercial impact multimedia and in particular digital convergence may have? And, perhaps equally important, why should we be interested in this from, I must say, an academic perspective?

In this last section of the introductory chapter, we will look at some popular press items related to new media, mixed media (in particular the merge of TV and internet) and mobile multimedia. We will then briefly reflect on what significance these issues have from our academic perspective.



new media

As you may be read in the newspapers in the beginning of this century, large investments have been made (by both cable and telephone companies) to improve the technological infrastructure for the new media. Simultaneously, joint ventures have arisen between content developers and providers, as with the Dutch Endemol company.

Now, what does the popular press have to say about all these developments. Here is one comment, from a Dutch newspaper:

Peter Greven 23/3/2001 (Volkskrant)

new media sucks


people like new technology.

they don't like new media.

The translation from Dutch is, admittedly, mine. It says, in other words, that people like to receive the old stuff on new gadgets, but that they are not willing to pay for any new sort of services. For example, when considering the smart video recorder, that uses a disk cache for storing MPEG coded versions of broadcasts, just think of other gadgets and services that didn't make it or that are encountering problems in being accepted. Some famous examples from the past are the videofoon, videotext, cd-i, and DCC. Perhaps the reason for these failures is the trial-and-error method,, also referred to as the spaghetti method, that is being followed in developing new media. As characterized by Jan van Dijk, of a dutch university in the east of the Netherlands (Twente), the spaghetti method consists of throwing a plate against the wall, and see what will stick to the wall. In other words, just throw your product on the market and see whether it will stick. Perhaps that is not the right method to be followed. But can you think of a better one?

In many cases 'the market', that is the people using a service, do not behave as expected. For example in Sweden, the upload of material far exceeded download, which is contrary to the assumptions underlying ADSL.

TV meets the Web

At first sight it seems promising to develop mixed media. As an example, a dutch agency announced services to support the integration of TV and the Web, promising the integration of

streaming media (audio and video), interactive gaming, virtual reality and 3D animation, interactive TV programming, interactive advertising, video on-demand, webcasting and multimedia

In 2000 they issued a report sketching the European broadband landscape. Quoting from this report: The advent of broadband Internet access, which has been available in the US for some time but is only now beginning to make inroads into Europe, makes a whole range of new services possible.

As download speeds have increased and more bandwidth has become available, the possibility of delivering screen-based content such as films, television programs and music has moved a step closer to mass market usage. With respect to the adoption of cable or DSL in Europe, they observe that despite the fact that cable companies have gained firm ground, there is an even larger number of conventional telephone lines, around 180 million. In contrast, there are only 15 million cable subscribers, giving DSL a large potential audience. Matthijs Leendertse, co-author of the report, observes: Gaining competitive advantage and future revenue in Europe's broadband landscape will depend heavily on a company's ability to offer integrated services: access (fixed and wireless) and content. It is virtually impossible at this point for one single company to offer these services on a pan-European level. This means that companies need to find partners to fill the gaps in their offerings. Let me assure you, at the moment you will be reading this the battle is still going on!

mobile multimedia

Let's look at another potential hype. In 2000, Webnoize published a report (by Matt Bailey), entitled Wireless Entertainment: What Is It Worth?, which introduces the wireless web, and predicts that young media junkies will demand music videos and animations, and listen to wirelessly streamed music.

The intent of the report is to investigate whether investments in the mobile entertainment are justified. The report examines how providers of music and video services can benefit from the wireless delivery of multimedia. Using survey evidence, pricing information from new wireless networks and interviews with industry visionaries, the report analyzes supply and demand to build an economic and business model for mobile multimedia.

Apart from the need to invent some business model, there are a number of strategic questions to be answered in order to estimate the risk of making investments in this direction. Following Bailey, we may list questions such as:

strategic questions

  • how quickly will wireless connectivity speeds improve?
  • what is the demand for services that deliver music and video to wireless devices?
  • how can suppliers of multimedia services monetize demand for wireless access?
  • how much will it cost to stream multimedia content to wireless devices now and in 2006?
  • are consumers willing to compromise quality for lower cost?
And more. If you are interested whether anyone is willing to take such risks and invest in mobile multimedia, just look at what players were involved.

the players

Alltel, AT&T Wireless, AtomShockwave, Cingular Wireless, Clear Channel, HitHive, Ifilm, Infinity, KDDI, Liquid Audio, LMIV, Mannesmann,, MTV, NetCom, Myplay, Nortel Networks, NTT DoCoMo, Omnitel, Sprint, Telefonica, Telstra, Vitaminic, Verizon Wireless, Virgin Megastores, Vodafone, Voicestream.

Now make up your mind, and ask yourself the question whether multimedia is worth your (intellectual) investment.


Vasulka Objects (1)Vasulka Objects (2)


the academic perspective

Being sensitive to hype is only too human. So also academics may be fascinated by new trends. and get distracted by rumors on the market. Breaking loose from this fascination, we may ask ourselves what are the real issues, and what makes multimedia interesting. Let me start with answering the latter question first. As an academic subject, multimedia is interesting because it offers such an intriguing mix of subjects, including multimedia technology, exploratory design and scientific validation. Commercially, it is safe to say that the volume of entertainment related multimedia content, including games, music and infotainment is substantial, and hence its economic interest is indisputable. But what are the real issues?

One of the examples of multimedia applications I will present in the last chapter is an application in the domain of cultural heritage. For this domain we have developed so-called digital dossiers containing a representation of the work(s) of a particular artist as well as information that characterizes the work in its historic and cultural context, needed for the re-exposition or installation of the work. Problems facing the developer of a digital dossier cover the interaction of the user with the dossier, the presentation of both textual and multimedia information, and facilities for search and navigation. And there are technical issues, such as which codecs to select for the videos and how to manage the content included in the dossier. Developing a dossier is not as one might naively think the creation of content only, but rather involves designing the functionality of the application as well.

Generalizing from the domain of cultural heritage to the area of infotainment and multimedia information systems, where an integrated presentation of textual and multimedia information must be achieved, we may boldly state that designing the functionality of the application is the most crucial issue, and as such of primary academic interest. All other topics, including multimedia technology, compression algorithms, software engineering, multimedia platform support and information retrieval techniques, may be regarded in some loose sense to be subservient to the issue of design.

digital art

As the illustrations in the text testify, another personal motivation for being involved in multimedia comes from the area of digital art. And, with students I observe a similar interest in the potential digital content authoring offers as a vehicle for personal expression.

One of the artists of which I included material in this book is Woody Vasulka, who was a pioneer in the early days of video and computer art. In an interview, held in 1985 with Rene Coelho, the founder of Montevideo, Vasulka explained his fascination with the scan processor and later the video computer by stating that it allowed him to invent the image. Still, howver, as he said, in some sense traditional painting acted as a visual reference system by which to judge the images produced with the new technology. Later in the interview, he observed that after some time he became bored with the images produced this way, and he started to feel the need to include more narrative in his work. His wife, Steina Vasulka, with whom he founded the Kitchen, a gathering place for new media artists in New York in the 1970s, remarked that in the early phase she was struck by the fact that the material was so friendly, that is how easy it was to express your ideas.

These words suffice to emphasize the importance of the motivation you might get out of the material, to be susceptible to as Brancusi phrases it the rethorics of the material, even when you are an academic.


(a) personal events(b) gadgets


example(s) -- personal memex

Just imagine that your would store all your photographs, SMS messages, emails, and in addition to that record your physiological condition, using body-wearable sensors. These data can then be uploaded to your PC, and later to a mass storage server, so that they can be used in your medical dossier, to improve your performance in your favorite sport, or to augment your memory when recollecting stories about your holidays or travels. Impossible? Not at all, Disk space will be cheap. Your body may act as a network to connect the body wearable devices, and, after all, most of the gadgets do already exist! Besides, the idea is not new. See section 2.2 for early visions of the memex.

research directions -- the information society

There is no doubt about it, we live in an information society. But do we know what an information society is?

In  [Briggs and Burke (2001)] (p. 187), the functions of the media are summarized as

functions of media

information, education, entertainment

So, perhaps, we could better state that we live in a media society. So far, in the latter part of the previous century, television has dominated our lives, and observe that (following Ernie Kovack, cited from  [Briggs and Burke (2001)]):


television is a medium 'because it is neither rare nor well done'

Back to the main issues, what is an information society? According to  [Briggs and Burke (2001)]:

information society

the new term 'information society' gave form to a cluster of hitherto more loosely related aspects of communication -- knowledge, news, literature, entertainment, all exchanged through different media and different media materials -- paper, ink, canvas, paint, celluloid, cinema, radio, television and computers.

From the 1960s onwards, all messages, public and private, verbal and visual, began to be considered as 'data', information that could be transmitted, collected, recorded, whatever their point of origin, most effective through electronic technology.

So, from the varieties of perspectives we have discerned, including technological perspectives, societal perspectives and psychological perspectives, we must investigate the problem of communication:


  • what -- content
  • who -- control
  • whom -- audience (how many)
That is, simply, who says what to whom in what channel with what effect?! The remainder of the book will, however, will treat these issues mainly from a technological perspective. In the chapters that follow, we will enquire after the technological assumptions that make an information society possible.

development(s) -- convergence in second life

As indicated in the preface, developments in new media never stop. On the contrary. At the time of writing, I consider both the (bewildering) offerings at youtube and the (outright surprising) growth of secondlife to be (the) most spectacular. What to do with youtube is another story. Let's start with Second Life.

I first heard about Second Life in the crossmedia week PICNIC'06. When I was asked, in december 2006, to join a project to bring our university into Second Life, I was happy to do so. The result of our efforts is well expressed in the following quote from  [Eliens et al. (2007c)]:


Second Life seems to be overtaking the world. In the whole range of cummunity-building platforms, Second Life stands out as an immersive 3D world with an almost stunning adoption, by both individuals, companies and institutions, followed attentively by the Press. Not entirely without an understanding of the value of press coverage, the VU University Amsterdam decided to create presence in Second Life, by creating a virtual campus, to realize a (virtual) community of learners,  [Eliens et al. (2007)]. And, indeed, we succeeded in being the first university in The Netherlands with presence in Second Life and, as hoped, this was covered in the 8 o'clock nation-wide TV news.


(a) visitors outside(b) visitors inside


It is indeed rewarding, once and a while, to enter the media circus. But of course, as (more or less) respectable academics, we are more serious than that. So,  [Eliens et al. (2007c)] continued: More substantial than getting into a nation-wide television broadcast, however, is our aim to communicate our institutional goals, creating a community of learners, by creating a virtual campus in Second Life, offering an information portal as well as a meeting point, in a media platform that is widely adopted by our target community. Virtual presence in Second Life, obviously, is not enough. The relatively long history of virtual worlds has shown that lack of interesting content and functionality easily leads to boredom, desinterest, and hence churn, users dropping off. As a consequence, there is a need for sustainable functionality, that both motivates people to come back and participate, and, otherwise why choose Second Life, makes essential use of the 3D immersive environment offered by Second Life. In  [Eliens et al. (2007c)], we further wrote: we will explore how to use web services in meaningful compositions or mashups to enhance our presence in Second Life, and create a community where visitors actively participate in both education and research, in other words, indeed, a community of learners.

How we did or would like to explore that will be dealt with later, see part v. Now it might be more worthwhile to look at what made Second Life, so suddenly, so popular, and what the future of Second Life, and possibly the Web, which according to some is threatened to be overtaken by Second Life, might look like.

In  [Eliens et al. (2007)], we discussed the success factors of Second Life. We asked ourselves: will Second Life become the new (immersive 3D) mass medium of our participatory culture of the 21th century, as once the immersive panorama was the propaganda/art medium for the masses in the 19th century? Cf.  [Grau (2003)]. In thinking about possible reasons why Second Life is so successful, we observed that Second Life does provide:

success factors (1/2)

  • convergence of social networking and content creation
  • immersive networked 3D environment
  • inclusion of elementary economic principles
However, we also see that other factors may contribute to the success of Second Life, such as:

success factors (2/2)

  • don't miss the boat effect
  • free and easy accessible 3D design tool set
  • adoption by big companies like IBM, Reebok, ...
  • marketing of Second Life by Linden Lab (?)
  • the promise to make (real) money (?)
According to Philip Rosedale, CEO of Linden Lab, (interview in .NET magazine, issue 158, January 2007) the success of SL is due to the fact that (1) it offers a set of capabilities, which are in many different ways superior to the real world, (2) the decision to allow residents to own the intellectual property rights to their creations and (3) because Second Life is full of creative possibilities, and opportunites for innovation.

In order to establish what constitutes the success of Second Life in a more rigorous manner, we must subject Second Life to a (game) reference model as introduced in  [Juul (2005)], see also section 12.1, which we have also applied to (serious) service management games in  [Eliens & Chang (2007)], see section 11.1. A first tentative characterization of Second Life according to our reference model would be:

reference model

  • rules -- construct and communicate!
  • outcome -- a second world
  • value -- virtual and real (monetary) rewards
  • effort -- requires elementary skills
  • attachment -- a virtual identity
  • consequences -- transfer to first life
Second Life clearly has a wider scope and more freedom than just gaming. Apart from elementary rules, that more or less require of the (serious) visitor to construct and communicate, there are almost no fixed rules, no in-game strategies, but many opportunities for inter-personal contact and the establishment of relations world-wide, affecting (possibly) the Second Lifer's first life (consequences). Whether Second Life will turn out to be a veritable media-supported augmentation of our first life, cf.  [Zielinski (2006)], remains to be seen. Chances are also that Second Life will end up as another item on the dead media project(s) list, see section 11.1, to be replaced by an alternative participatory framework or environment.

The reference model we introduced in  [Eliens et al. (2007)], will be explained in more detail in chapter 12. Obviously, to establish the value of Second Life, or any game whatsoever, in a more objective manner, some reference model is needed. Here it suffices to see how the reference model expresses why second life might be worthwhile, poosibly as an alternative to first life, which evidently is not at all times rewarding or even satisfactory!

So far, our story about how we brought the VU on or into (its) Second Life is full of the techno-euforia so often encountered in these stories. But, in all our enthusiasm, as we did indeed enjoy the project and the challenge to be on time, on time to get into the 8 o'clock news, we do well at taking a step back, and look at our exercise from a more critical perspective.

At some point in my career I gave a talk entitled what led me to multimedia?, with as a subtitle and where does multimedia lead me?, indicating my doubts bout how multimedia would fit in the academic curriculum, and for that matter, in my own (academic) career. Generalizing this perspective, we may ask ourselves what led us to second life, and weher will second life lead us?

With critical perspective, I mean the perspective of critical theory, which is characterized in  [Wilson (2002)] as the discipline(s) that

critical theory

attempt(s) to link the arts, literature, media studies, politics, sociology, antropology, philosophy and technology in an interdisciplinary search for relevant concepts and frameworks with which to understand the current world.

Often the findings of (researches from) critical theory may sound poetic in themselves, for example, quoting Lev Manovich's Labor of Vision from  [Wilson (2002)]:

contemporary perception(s)

... in the contemporary world, the perceptual task has changed, in both leisure and work, to monitor data displays, ready for events.

Nevertheless, such statements indicate how much the current (ICT & media) developments affect our lives, and even touch upon epistemological issues questioning what is our reality. See section 11.3 for more on this topic.

A more direct and perhaps sobering analysis is given in Richard barbrook's and Andy cameron's essay The Californian Dream which according to  [Wilson (2002)] states:

californian dream(s)

... the new faith has emerged from a bizarre fusion of the cultural bohemianism of San Francisco with the high-tech industries of Silocon Valley...

and, to further de-construct the digital utopianism:

... the californian ideology promiscuously combines the freewheeling spirit of the hippies and the entrepeneurial zeal of of the yuppies

Now you may find the implied accusation of american imperialism exaggerated, but still, as testified by recent attempts to give the creative industry in the Amsterdam area a push, the bay area (San Francisco and Silicon Valley) still play a leading role and act as model for the combination of creativity, innovation and entrepeneurship around the world!. Another implication is also worthwhile to consider, since it seems to directly affect our personal life, as members of the digital class. As phrased in  [Wilson (2002)]:

digital class

... the shadow side of the digital class's freedom and individuality is a lack of connection ... and an unrealized acceptance of work as the main life value.

In the light of these remarks it is doubtful whether it is really such a privilige to belong to the digital class and/or to work in the new media or creative industry. That is a matter of personal choice, I would see, and I can only encourage anyone to get (academic) training and pursue a career in this area. After all, that is my living!

In the line of critical theory, it is at this moment appropriate to look at another quote, again taken from  [Wilson (2002)], a quote that exemplifies cyberfeminist critique;


... empowerment of women in the field of new electronic media can only result from the demystification of technology and the appropriation of access to (these) tools.

However, it would be fundamentally wrong, in my opinion, not to extend this emancipatoric advice to both genders, of all cultures. Access to tools and de-mystification of technology seems to be benifial for all, and possibly the only way to survive in our (information) society that is increasingly being dominated by (digital) culture. Or was it lack of culture?




1. digital convergence



projects & further reading

As a project, consider the development of a Java-based mobile game using J2ME, see  [Morrison (2005)], or a web-based game using Visual Basic .NET, see  [Santos Lobao and Hatton (2003)].

You may further explore multiplatform game development, and find arguments to choose for either Java-based or managed code based implementations.

For further reading, I advice to have a look at the special issues of the Scientific American,  [American], and the CACM on the next 1000 years of computing,  [CACM (2001)], and, for getting an idea where this all leads to, Schneidermann's Leonardo's laptop,  [Shneiderman (2003)]. For Second Life, see  [Rymaszweski et al. (2007)].

the artwork

  1. photographs of art works by Marina Abramovic, Art must be beautiful, Blue period, Dissolution, Dozing consciousness, In between, with (pending) permission from Montevideo. See also section 10.2.
  2. medium, according to the Visual Thesaurus.
  3. fMRI Research on Virtual Reality Analgesia, see section 1.1.
  4. television and communication, according to the Visual Thesaurus.
  5. TV Today, exhibition at Montevideo, februari 2005.
  6. visible world -- taken from  [Rosenblum and Macedonia (2002)], see section 1.2.
  7. personal event database and personal gadgets, from Freeband project.
  8. Thomas Lips 1975, Thomas Lips 1993, from Marina Abramovic, with permission from Montevideo.
  9. scanlines from Woody Vasulka, 197x, with permission from the artist.
  10. VU @ SecondLife, taken from  [Eliens et al. (2007)].
  11. signs -- people,  [ van Rooijen (2003)], p. 254, 256.
The work of Marina Abramovic has a strong existential flavor. It has also served as the material for a case study in developing a digital artist dossiers, the abramovic dossier, discussed in section 10.2. The work of Woody Vasulka is of a more experimental character, and shows the joy of discovering the possibilities of the, at the time, new electronic and digital tools and materials.

(C) Æliens 23/08/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.