more than the art of turning base metals into gold, alchemy is a system of cosmic symbolism

perfect solutions

digital convergence


perspectives -- digital convergence


the artwork

  1. alchemy -- an illustration from a book about alchemy, from which also the quote is taken, the quote is explained in the afterthoughts.
  2. signs -- ancient chemical symbols,  [Signs], p. 171, 172.
  3. photographs -- Jaap Stahlie, from portrait series.


digital culture

life is becoming digital

digital convergence

learning objectives

After reading this chapter you should be able to define the notion of multimedia, recount the history of digital entertainment, explain the concept of digital convergence, discuss the future of cyberspace, and speculate about the commercial viability of mobile multimedia.

We live in the digital era,  [Negroponte (1995)]. We are surrounding ourselves with gadgets and we are consuming immense amounts of information, that is increasingly being delivered to us via the Internet. We play games, and we still watch (too much) television.

Some of us watch televion on our PCs, and may be even looking forward to watch television on their mobile phone. This is multimedia. For others, the PC is still a programmable machine. Being able to program it might earn you a living. Understanding multimedia, however, might even provide you with a better living.

In this chapter, we study what trends may currently be observed in the creation and delivery of multimedia information, and we explore what impact the digital revolution may have from a commercial perspective.


levels of meaning

cultural convergence

The cultural convergence of art, science, and technology provides ample opportunity for artists to challenge the very notion of how art is produced and to call into question its subject matter and its function in society.

standardization and uniformity

  1. Telematic media were incorporated very quickly in the globalization strategies of transnational corporations and their political administrators and they became increasingly dependent on existing power structures.
  2. At the other end of the scale, there were individuals, or comparatively small groups, who projected great hopes onto these networks as a testing ground for cultural, artistic and political models that would give greater prominence and weight to divergence and plurality.

Scientific American (november 2000)

The barriers between TV, movies, music, videogames and the Internet are crumbling.

Audiences are fetting new creative options.

Here is what entertainment could become if the technological and legal hurdles can be cleared ...

Underlying the importance of entertainment in the era of digital convergence is the premisse governing an entertainment economy, which may be stated as

there is no business without show business

evolution of digital entertainment

  • 1953: Winky Dink (CBS) -- interactive television, drawing exercise
  • 1972: Pong (Atari) -- ping-pong on computer screen
  • 1977: Adventure -- text-based interactive fiction
  • 1983: Dragon's Liar -- laser-disc technology 3D game
  • 1989: SimCity -- interactive simulation game
  • 1989: Back to the Future -- the Ride
  • 1993: Doom -- 3D action game
  • 1995: The Spot -- interactive web-based soap opera (Webisodic)
  • 1999: IMAX3D -- back to Atlantis (Las Vegas)
  • 2000: Big Brother -- TV + around the clock Web watch + voting
  • 2001: FE Sites -- fun enhanced web sites

experience is fundamental to human life

The desire to share experiences will be the motivating factor in the development of exciting multimedia technology in the foreseeable future.

communication technology

  • oral -- communicate symbolic experiences
  • writing -- record symbolic experiences
  • paper -- portability
  • print -- mass distribution
  • telegraph -- remote narrow communication
  • telephone -- remote analog communication
  • radio -- analog broadcasting of sound
  • television -- analog A/V broadcasting
  • recording media -- analog recording
  • digital processing -- machine enhancement
  • internet -- multimedia communication

the medium was the message when only one medium could be used to communicate messages.

... cyberspace is a construct in terms of an electronic system.


television, video cassettes, video tape-recorder/players, video games, and personal computers all form an encompassing electronic system whose various forms interface to constitute an alternative and absolute world that uniquely incorporates the spectator/user in a spatially decentered, weakly temporalized and quasi-disembodied state.

virtual reality

virtual reality (is) when and where the computer disappears and you become the 'ghost in the machine' ...


the receiver at the RCA Pavillon was way ahead of its time, it was a combination of television - radio - recorder - playback - facsimile - projector ...

digital convergence

the union of audio, video and data communication into a single source, received on a single device, delivered by a single connection

subsidiary convergences

  • content -- audio, video, data
  • platform -- PC, TV, internet, game machine
  • distribution -- how it gets to your platform


  • content -- 2D/3D graphics, data, video, audio
  • distribution -- broadcast, wireless, DVD, internet, satelite, cable
  • platform -- PC, television, game machine, wireless data pad, mobile phone


  • HDTV -- high definition television
  • SDTV -- standard definition television
  • ITV -- interactive television

a killer d-TV appliance ...

  • personal television -- TiVo, Replay-TV (MPEG-2 cache)
  • game machine -- Sony PS 2/3, X-Box

TV or PC

The roadblock to the Entertainment PC could be the PC itself. Even a cheap TV doesn't crash or freeze. The best computers still do.


  • telephone network -- from 0.5 - 2 Mbps to 60 Mpbs (2.5km)
  • broadcast TV -- 6 MHz / 19 Mbps (4 channels MPEG HDTV)
  • cable TV -- hybrid fiber-optic coaxial cable 6 Mbps
  • fixed wireless -- 2 Mbps (radiotowers + rooftop antenna), phones/handhelds
  • satellite -- downloads to 100kbps, modem for uploads ...

digital convergence

what will we do with convergence once we have it?


we will watch

Google Earth

media as materials

each medium of communication tended to create a dangerous monopoly of knowledge

technological determinism

technological determinism was not the answer, ... more attempts were to be made to provide answers about the social consequences of television than had ever been asked about radio.


Information became a major concern anywhere during the late 1960 and 1970s where there was simultaneous talk both of 'lack of information' and 'information saturation'.

 [Briggs and Burke (2001)], p. 555

Peter Greven 23/3/2001 (Volkskrant)

new media sucks


people like new technology.

they don't like new media.

streaming media (audio and video), interactive gaming, virtual reality and 3D animation, interactive TV programming, interactive advertising, video on-demand, webcasting and multimedia

strategic questions

  • how quickly will wireless connectivity speeds improve?
  • what is the demand for services that deliver music and video to wireless devices?
  • how can suppliers of multimedia services monetize demand for wireless access?
  • how much will it cost to stream multimedia content to wireless devices now and in 2006?
  • are consumers willing to compromise quality for lower cost?

the players

Alltel, AT&T Wireless, AtomShockwave, Cingular Wireless, Clear Channel, HitHive, Ifilm, Infinity, KDDI, Liquid Audio, LMIV, Mannesmann,, MTV, NetCom, Myplay, Nortel Networks, NTT DoCoMo, Omnitel, Sprint, Telefonica, Telstra, Vitaminic, Verizon Wireless, Virgin Megastores, Vodafone, Voicestream.

functions of media

information, education, entertainment


television is a medium 'because it is neither rare nor well done'

information society

the new term 'information society' gave form to a cluster of hitherto more loosely related aspects of communication -- knowledge, news, literature, entertainment, all exchanged through different media and different media materials -- paper, ink, canvas, paint, celluloid, cinema, radio, television and computers.

From the 1960s onwards, all messages, public and private, verbal and visual, began to be considered as 'data', information that could be transmitted, collected, recorded, whatever their point of origin, most effective through electronic technology.


  • what -- content
  • who -- control
  • whom -- audience (how many)


Second Life seems to be overtaking the world. In the whole range of cummunity-building platforms, Second Life stands out as an immersive 3D world with an almost stunning adoption, by both individuals, companies and institutions, followed attentively by the Press. Not entirely without an understanding of the value of press coverage, the VU University Amsterdam decided to create presence in Second Life, by creating a virtual campus, to realize a (virtual) community of learners,  [Eliens et al. (2007)]. And, indeed, we succeeded in being the first university in The Netherlands with presence in Second Life and, as hoped, this was covered in the 8 o'clock nation-wide TV news.

success factors (1/2)

  • convergence of social networking and content creation
  • immersive networked 3D environment
  • inclusion of elementary economic principles

success factors (2/2)

  • don't miss the boat effect
  • free and easy accessible 3D design tool set
  • adoption by big companies like IBM, Reebok, ...
  • marketing of Second Life by Linden Lab (?)
  • the promise to make (real) money (?)

reference model

  • rules -- construct and communicate!
  • outcome -- a second world
  • value -- virtual and real (monetary) rewards
  • effort -- requires elementary skills
  • attachment -- a virtual identity
  • consequences -- transfer to first life

critical theory

attempt(s) to link the arts, literature, media studies, politics, sociology, antropology, philosophy and technology in an interdisciplinary search for relevant concepts and frameworks with which to understand the current world.

contemporary perception(s)

... in the contemporary world, the perceptual task has changed, in both leisure and work, to monitor data displays, ready for events.

californian dream(s)

... the new faith has emerged from a bizarre fusion of the cultural bohemianism of San Francisco with the high-tech industries of Silocon Valley...

and, to further de-construct the digital utopianism:

... the californian ideology promiscuously combines the freewheeling spirit of the hippies and the entrepeneurial zeal of of the yuppies

digital class

... the shadow side of the digital class's freedom and individuality is a lack of connection ... and an unrealized acceptance of work as the main life value.


... empowerment of women in the field of new electronic media can only result from the demystification of technology and the appropriation of access to (these) tools.

projects & further reading

As a project, consider the development of a Java-based mobile game using J2ME, see  [Morrison (2005)], or a web-based game using Visual Basic .NET, see  [Santos Lobao and Hatton (2003)].

You may further explore multiplatform game development, and find arguments to choose for either Java-based or managed code based implementations.

For further reading, I advice to have a look at the special issues of the Scientific American,  [American], and the CACM on the next 1000 years of computing,  [CACM (2001)], and, for getting an idea where this all leads to, Schneidermann's Leonardo's laptop,  [Shneiderman (2003)]. For Second Life, see  [Rymaszweski et al. (2007)].

the artwork

  1. photographs of art works by Marina Abramovic, Art must be beautiful, Blue period, Dissolution, Dozing consciousness, In between, with (pending) permission from Montevideo. See also section 10.2.
  2. medium, according to the Visual Thesaurus.
  3. fMRI Research on Virtual Reality Analgesia, see section 1.1.
  4. television and communication, according to the Visual Thesaurus.
  5. TV Today, exhibition at Montevideo, februari 2005.
  6. visible world -- taken from  [Rosenblum and Macedonia (2002)], see section 1.2.
  7. personal event database and personal gadgets, from Freeband project.
  8. Thomas Lips 1975, Thomas Lips 1993, from Marina Abramovic, with permission from Montevideo.
  9. scanlines from Woody Vasulka, 197x, with permission from the artist.
  10. VU @ SecondLife, taken from  [Eliens et al. (2007)].
  11. signs -- people,  [ van Rooijen (2003)], p. 254, 256.

everything must be intertwinkled

hypermedia information spaces

learning objectives

After reading this chapter you should be able to define information spaces in a precise manner, position the hypertextual capabilities of the web in a historical perspective, explain the difference between multimedia and hypermedia, and argue why computational support for narrative structure in multimedia applications is desirable.

However entertaining it might be presented to you, underlying every multimedia presentation there is an information space. That is to say, irrespective of the medium, there is a message. And being confronted with a message, we might want to inquire for more information.

In this chapter, we will define the notion of information space more precisely.

We will extend this definition to include information hyperspaces, by looking at the history of hypertext and hypermedia.

Finally, we will discuss visualisation as a means to present (abstract) information in a more intuitive way, and we will reflect on what is involved in creating compelling multimedia.

Current day multimedia information systems distinguish themselves from older day information systems not only by what information they contain, that includes multimedia objects such as images and sounds, but also by a much more extensive repertoire of query mechanisms, visual interfaces and rich presentation facilities. See  [Chang and Costabile (1997)].

S.K. Chang and M.F. Costabile -- Visual Interfaces to Multimedia Databases

The Handbook of Multimedia Information Management

multimedia information systems

  • storage technology -- multimedia databases
  • wideband communication -- distribution accross networks
  • parallel computing -- voice, image and video processing
  • graphic co-processors -- visual information with high image quality

multimedia applications

geographical information systems, office automation, distance learning, health care, computer aided design, scientific visualization, and information visualization.

multimedia databases

  • the size of data,
  • synchronization issues,
  • query mechanisms, and
  • real time processing.

an information space is a representation of the information stored in a system or database that is used to present that information to a user.

we must distinguish between a visual information space (for presentation), a logical information space (in which we can reason about abstract information objects) and a physical information space (where our concrete multimedia objects are stored).

  • physical information space -- images, animations, video, voice, ...
  • logical information space -- abstract database objects
  • presentational information space -- to present information to the user

a logical information space is a multidimensional space where each point represents an object from the physical information space (read database).

  • information object -- a point in the (logical) information space
  • query -- an arbitrary region in this information space
  • clue -- a region with directional information, to facilitate browsing

XML is a set of rules (you may also think of them as guidelines or conventions) for designing text formats that let you structure your data.

XML in 10 points

  1. XML is for structuring data
  2. XML looks a bit like HTML
  3. XML is text, but isn't meant to be read
  4. XML is verbose by design
  5. XML is a family of technologies
  6. XML is new, but not that new
  7. XML leads HTML to XHTML
  8. XML is the basis for RDF and the Semantic Web
  9. XML is license-free, platform-independent and well-supported

related technologies

  • Xlink -- hyperlinks
  • XPointer -- anchors and fragments
  • XSL -- advanced stylesheets
  • XSLT -- transformation language
  • DOM -- object model for application programmer interface
  • schemas -- to specify the structure of XML documents


  • separate data from presentation
  • transmit data between applications

information hyperspace

the logical information space may further be structured in a logical information hyperspace, where the clues become hyperlinks that provide directional information, and the information space can be navigated by the user following directional clues.


  • 1945 -- Vannevar Bush (Memex) -- as we may think,  [ Bush (1995)]
  • 1963 -- Douglas Engelbart (Augment) -- boosting the human intellect  [Engelbart (1963)]
  • 1980 -- Ted Nelson (Xanadu) -- everything is intertwinkled,  [Nelson (1980)]

  • flash 1: we are in trouble (human mankind)
  • flash 2: we need to boost mankind's ability to deal with complex urgent problems
  • flash 3: aha, graphic vision surges forth of me ...
  • flash 4: hypermedia -- to augment the human intellect
  • flash 5: augment (multimedia) workstation -- portal into an information space

hypermedia systems

  • components -- text, graphics, audio, video
  • links -- relations between components
  • presentation -- structured display

A curriculum promotes a false simplification of any subject, cutting the subject's many interconnections and leaving a skeleton of sequence which is only a charicature of its richness and intrinsic fascination.

classification of hypermedia systems

  • macro-literary systems -- publishing, reading, criticism
  • problem exploration tools -- authoring, outlining, programming
  • browsing systems -- teaching, references, information
  • general hypermedia technology -- authoring, browsing, collaboration
  • embedded hypermedia -- CASE, decision support, catalogs


  • content -- text, graphics, video, program
  • attributes -- semantic description
  • anchors -- (bi-directional) links to other documents
  • presentation -- display characteristics

(CMIF) multimedia model

  • data block -- atomic component
  • channel -- abstract output device
  • synchronization arc -- specifying timing constraints
  • event -- actual presentation

Amsterdam Hypermedia Model

  • contents -- data block
  • attributes -- semantic information
  • anchors -- (id, value)
  • presentation -- channel, duration, ...

research issues

  • search and query -- for better access
  • composition -- for imposing structure
  • virtual structures -- on top of existing structures
  • computation -- for flexibility and interaction
  • versioning -- to store modification histories
  • collaborative work -- sharing objects with multiple users
  • extensibility and tailorability -- to adapt to individual preferences


Grasping the whole is a gigantic theme, intellectual history's most important.

Ant vision is humanity's usual fate; but seeing the whole is every thinking person's aspiration.

David Gelernter, Mirror Worlds 1992

data types

  • 1-D linear data -- text, source code, word index
  • 2-D map data -- floor plan, office layout
  • 3-D world -- molecules, schematics, ...
  • temporal data -- 1 D (start, finish)
  • multi-dimensional data -- n-dimensional (information) space
  • tree data -- hierarchical
  • network data -- graph structure

  • interactive -- overview first, zoom and filter, then details on demand
  • storytelling -- as a paradigm for information presentation

Whatever your target audience, whatever your medium, whatever your message, you have to be convincing if not compelling.


  • a communication process in which the communicator seeks to elicit a desired response from his receiver
  • a conscious attempt by one individual to change the attitudes, beliefs or behaviours of another individual or group individual through the transmission of some messages.


  • immediacy -- a tendency towards transparent immersion, and
  • hypermediacy -- the presence of referential context

Virtual Reality won't merely replace TV. It will eat it alive.


  • epistemological: transparency, the absence of mediation
  • psychological: the medium has disappeared, presence, immersion


  • epistemological: opacity, presence of the medium and mediation
  • psychological: experience of the medium is an experience of the real

Convergence is the mutual remediation of at least three important technologies -- telephone, televison and computer -- each of which is a hybrid of technical, social and economic practice, and each of which offers its own path to immediacy.

The telephone offers the immediacy of voice or the interchange of voices in real-time.

Television is a point-of-view technology that promises immediacy through its insistent real-time monitoring of the world.

The computer's promise of immediacy comes through the combination of three-dimensional graphics, automatic (programmed) action, and an interactivity that television can not match.

As they come together, each of these is trying to absorb the others and promote its own version of immediacy.


(p. 27) ... merging previously disparate technologies of communication and representation into a single medium.

The networked computer acts like a telephone in offering one-to-one real-time communication, like a television in broadcasting moving pictures, like an auditorium in bringing groups together for lectures and discussion, like a library in offering vast amounts of textual information for reference, like a museum in its ordered presentation of visual information, like a billboard, a radio, a gameboard and even like a manuscript in its revival of scrolling text.


  • procedural -- 'programmed media' ...
  • participatory -- offering agency


  • spatial -- explorable in (state) space
  • encyclopedic -- with (partial) information closure

multimedia authoring

  • narrative format
  • procedural authorship

web 2.0

video sharing / online gaming / social networking

daft punk -- technologic (cn / jp)

  Buy it, use it, break it, fix it.
  Trash it, change it, melt -- upgrade it.
  Change it, point it, zoom it, press it.
  Snap it, work it, quick -- erase it.
  Write it, out it, paste it, save it.
  Load it, check it, quick -- rewrite it.
  Plug it, play it, burn it, rip it.
  Drag and drop it, zip -- unzip it.
  Look it, fill it, curl it, find it.
  View it, coat it, jam -- unlock it.
  Surf it, scroll it, pose it, click it.
  Cross it, crack it, twitch -- update it.
  Name it, rate it, tune it, print it.
  Scan it, send it, fax -- rename it. 
  Touch it, bring it. Pay it, watch it.
  Turn it, leave it, stop -- format it.


  • substituting a single pragmatism for ideal design
  • using light weight programming models

web 2.0 design pattern(s)

  • web 1.0 -- the web as platform
  • web 2.0 -- architecture of participation
  • web 3.0 -- data is the (intel) inside

Learnlog: XML Is The Fabric Of Web 2.0 Applications

  • the client side is AJAX (Asynchronous Javascript and XML)
  • the server application typically exposes data through XML
  • the interaction model is web services
  • mashups combine multiple webservices to create new types of applications


  • representation -- encoding in a particular format
  • state -- data encapsulated in an object
  • transfer -- using HTTP methods

the most powerful mashups don't just mix code and data, they mix cultures.

which provides a challenge that trancends all issues of mere technological correctness.



  • /seen?user=SomeAvatar -- records the presence of SomeAvatar
  • /touched?user=SomeAvatar -- invokes flickr API with users tag
  • /set_tag?user=SomeAvatar&tag=FavoriteTag -- records SomeAvatar's favourite tag

flash/quicktime in SL

Quicktime supports Flash, but only up to Flash version 5. We're up to version 9 on that now! Luckily, I have been dabbling with Flash since the early days, so already knew how to do this 'the old way'... So, Flash is doing all the work. No LSL at all... I heart feeds. Did I say 'I heart feeds?

Referring to section 7.4 for a more detailed discussion, we may observe that there is no meaning in merely putting things together. Without mechanisms of personalization and recommendation we would simply be flooded by data and information, in a way that even search would not be able to cope with. Context, narratives and personalized presentation(s), notions from the past, reappear as keywords for the future of the web 2.0 and beyond.

projects & further reading

As a project, I suggest the development of a virtual tour in a city, museum or other interesting locatoion.

You may further explore the implementation of traversal within a context, taking into account the history of navigation when backtracking to a particular point, issues in hyperlinking and interaction in multimedia applications, and computational support for narratives.

For further reading I advice you to take a look at the history of hypermedia and the web, using online material from the W3C, or the history of media as accounted for in  [Briggs and Burke (2001)] and  [Bolter and Grusin (2000)].

the artwork

  1. book covers --  [Weishar (1998)],  [Eco (1994)],  [Burger (1981)],  [Kunst],  [Betsky (2004)]
  2. Federico Campanale -- Oxygen, fragments from video installation, 2004
  3. Vasarely --  [Diehl 1973].
  4. Vasarely --  [Diehl 1973].
  5. Vasarely --  [Diehl 1973].
  6. Federico Campanale -- Oxygen, more fragments.
  7. student work -- from introduction multimedia 2000.
  8. Rutger van Dijk -- mobius, interactive story, opening screen, see section 2.3.
  9. edgecodes -- screenshots, see section 2.3
  10. signs -- people,  [ van Rooijen (2003)], p. 244, 245.