topical media & game development

talk show tell print

print / present


hypermedia information spaces

everything must be intertwinkled

learning objectives

After reading this chapter you should be able to define information spaces in a precise manner, position the hypertextual capabilities of the web in a historical perspective, explain the difference between multimedia and hypermedia, and argue why computational support for narrative structure in multimedia applications is desirable.

However entertaining it might be presented to you, underlying every multimedia presentation there is an information space. That is to say, irrespective of the medium, there is a message. And being confronted with a message, we might want to inquire for more information.

In this chapter, we will define the notion of information space more precisely.

We will extend this definition to include information hyperspaces, by looking at the history of hypertext and hypermedia.

Finally, we will discuss visualisation as a means to present (abstract) information in a more intuitive way, and we will reflect on what is involved in creating compelling multimedia.



information and data

Current day multimedia information systems distinguish themselves from older day information systems not only by what information they contain, that includes multimedia objects such as images and sounds, but also by a much more extensive repertoire of query mechanisms, visual interfaces and rich presentation facilities. See  [Chang and Costabile (1997)].

S.K. Chang and M.F. Costabile -- Visual Interfaces to Multimedia Databases

The Handbook of Multimedia Information Management

Preceding the advent of multimedia information systems, which include networked multimedia systems as discussed in section 6.3, we have seen advances in

multimedia information systems

  • storage technology -- multimedia databases
  • wideband communication -- distribution accross networks
  • parallel computing -- voice, image and video processing
  • graphic co-processors -- visual information with high image quality

Now, the class of multimedia information systems is, admittedly, a large one and includes applications and application areas such as:

multimedia applications

geographical information systems, office automation, distance learning, health care, computer aided design, scientific visualization, and information visualization.

Nevertheless, irrespective of what technology is used for storage and retrieval, multimedia information systems or multimedia databases impose specific requirements, with respect to: the size of data, synchronisation issues, query mechanisms and real-time processing.

multimedia databases

  • the size of data,
  • synchronization issues,
  • query mechanisms, and
  • real time processing.
Partly, these requirements concern the efficiency of storage and retrieval and partly they concern aspects of usability, that is the way information is presented to the user. In particular, we can think of a great number of query mechanisms that our multimedia information system of choice is expected to support: free text search, SQL-like querying, icon-based techniques, querying based on ER-diagrams, content-based querying, sound-based querying, query by example, and virtual reality techniques.



logical information spaces

But before thinking about the optimal architecture of multimedia information systems or the way the information is presented to the user, let's consider in what way a multimedia (information) system or presentation may be considered an information space.

As a tentative definition, let's assume that

an information space is a representation of the information stored in a system or database that is used to present that information to a user.

This may sound too abstract for most of you, so let's have a look at this defininition in more detail.

First of all, observe that when we speak of representation, and when we choose for example a visual representation, then the representation chosen might be either the users conceptualization of the database, or a system generated visualization. In principle the same holds for a text-based representation, but this is far less interesting because the options in choosing a representation and presenting it to the user are much more limited.

Unfortunately, the phrase representation is also somewhat vague. To be more precise,

we must distinguish between a visual information space (for presentation), a logical information space (in which we can reason about abstract information objects) and a physical information space (where our concrete multimedia objects are stored).

Summarizing we have:

  • physical information space -- images, animations, video, voice, ...
  • logical information space -- abstract database objects
  • presentational information space -- to present information to the user
Our visual information space, our presentation space, as you may prefer to call it, might reflect the logical information space in a symbolic manner by using diagrams, icons, text and possibly visualzations, or, going one step further, it may also mimic the logical information space by using virtual reality, as discussed in chapter 8.

Now we can give a more precise definition of the notion of information space, in particular logical information spaces:

a logical information space is a multidimensional space where each point represents an object from the physical information space (read database).

First of all, observe that when we speak of dimensions we might also speak of attributes that can take either continuous, numerical, discrete or logical values. So, concretely, these attributes may be directly or indirectly related to information stored in the database, and hence we can give a more precise definition of the notion of (multimedia) information objects, queries and cues (in the logical information space):

  • information object -- a point in the (logical) information space
  • query -- an arbitrary region in this information space
  • clue -- a region with directional information, to facilitate browsing
The notion of clue is actually quite interesting, since both examples and queries may be regarded as clues, that facilitate browsing through the contents of an information space. As an example, just think of the situation that, when looking for another notebook, you want something that is similar to the the thing you've previously seen, but that has an additional video output slot that may be connected to your TV.

Also, clues are needed to allow for query by example. In this case you need to help the user to define a query in the logical information space, so that the system can construct an optimal query to search for the desired object(s) in the physical information space.

When we regard the information retrieval problem to be the construction of the optimal query with respect to the examples and clues presented by the user, then we may characterize the optimal query as the one that will retrieve the largest number of relevant database objects within the smallest possible region in the (logical) information space.


Given the stratification, that is levels or layers, of information systems discussed above, we can think of improvements or extensions on each level. At the physical layer, for example networked multimedia, in a client/server architecture, see 6.3. At the logical layer, as an information hyper space,consisting of chunks and hyperlinks, as explained in section 2.2. And at the presentation layer a virtual reality interface, representing for example the physical location of student records, somewhere at a virtual campus [x], as further explored in chapter 8. Each of these improvements or extensions can be regarded as a technological or scientific adventure in it's own right.



example(s) -- e-flux

Do you recognize this?

When we visit a contemporary art exhibition, we find ourselves before the works, which are often quite difficult to interpret, and we observe them without understanding the process that generated them. Between a chopped-up cow immersed in formaldehyde and a replica of the Pope blindsided by a meteorite, it's legitimate to ask questions.

To provide a counter-force the exhibiton Project Room challenges the usual exhibition routine and decides to not exhibit executed art works but rather offers ten self-interviewing videos by as many artists, who speak openly about a piece they are working on, or a visionary project they want to realize, or about their creative process.

In other words, this is about works of art with no physical manifestion. It is an interesting issue whether this would still count as a work of art. And for multimedia, is there multimedia without a physical manifestation, with sensorily impressing the user/client. Do you remember the children story, the New Clothes of the Emperor?

research directions -- universal interchange

Technology changes rapidly. Just think about the development of the PC in the last two decades of the previous century. And applications change rapidly too. At the time of writing the web does barely exist for ten years. Information spaces, on the other hand, from a sufficiently abstract perspective at least, should be rather stable over time. So the question is, how can we encode information content in an application-independent way? As a remark, application-independence implies technology-independence. The answer is, simply, XML. The next question then should be, what is XML and why is it more suitable for encoding information then any of the other formats, such as for example relational tables.

The first question is not so difficult. There are many sources from where an answer may be obtained. Perhaps too many. A good place to start is the XML FAQ (Frequently Asked Questions) at the Web Consortium site:

XML is a set of rules (you may also think of them as guidelines or conventions) for designing text formats that let you structure your data.

More specifically, XML may be characterized as follows:

XML in 10 points

  1. XML is for structuring data
  2. XML looks a bit like HTML
  3. XML is text, but isn't meant to be read
  4. XML is verbose by design
  5. XML is a family of technologies
  6. XML is new, but not that new
  7. XML leads HTML to XHTML
  8. XML is the basis for RDF and the Semantic Web
  9. XML is license-free, platform-independent and well-supported
Perhaps not all of these points make sense to you at this stage. So let me first indicate that XML has in fact quite a long history. XML is the successor of SGML (the Structured Generalized Markup Language) that was developed in the 1980s to encode documents (such as airplane manuals) in an application-independent manner. SGML is not a language itself, but a descritpion of how to create a content description language, using tags and attributes (as in HTML). In fact, HTML is an application of SGML, using tags with attributes both for formatting and hyperlinks. In other words, SGML is a meta language. And so is XML. Since everything got messy on the web, XML was proposed (as a subset of SGML) to make a clear distinction between content and presentation. Presentation aspects should be taken care of by stylesheets (see below) whereas the content was to be described using and XML-based language.

Now, why is XML a suitable format for encoding data? That question is a bit harder to answer. One of the reasons to use XML might be that it comes with a powerful set of related technologies (including facilities to write stylesheets):

related technologies

  • Xlink -- hyperlinks
  • XPointer -- anchors and fragments
  • XSL -- advanced stylesheets
  • XSLT -- transformation language
  • DOM -- object model for application programmer interface
  • schemas -- to specify the structure of XML documents
These technologies (that are, by the way, still in development) provide the support needed by applications to do something useful with the XML-encoded information. By itself, XML does not provide anything but a way to encode data in a meaningful manner. Meaning, however, comes by virtue of applications that make use of the (well-structured) data.

In summary, XML and its related technologies provide the means to


  • separate data from presentation
  • transmit data between applications
Actually, the fact that XML was useful also for arbitrary data interchange became fully apperent when XML was available. To get an impression of what XML is used for nowadays, look at

This leaves us with the question of why XML is to be preferred over other candidate technolgies, such as relational databases and SQL. According to  [Kay (2001)], the answer to that question is simply that XML provides a richer data structure to encode information. In the multimedia domain we see that XML is widely adopted as an encoding format, see section 3-2. For an example you might want to have a look at MusicXML, an interchange format for notation, analysis, retrieval, and performance applications, that is able to deal with common Western musical notation as used from the 17th century onwards. In appendix XML we will explore how XML might be useful for your own multimedia application by treating some simple examples.




Given an information space we may turn it into an information hyperspace, that is, following  [Chang and Costabile (1997)],

information hyperspace

the logical information space may further be structured in a logical information hyperspace, where the clues become hyperlinks that provide directional information, and the information space can be navigated by the user following directional clues.

In other words, information is chunked, and each chunk is illustrated or made accessible by an example (hypernode).

Now, what exactly does information hyperspace mean? To answer this question, let's briefly look at the history of hypertext and hypermedia.


  • 1945 -- Vannevar Bush (Memex) -- as we may think,  [ Bush (1995)]
  • 1963 -- Douglas Engelbart (Augment) -- boosting the human intellect  [Engelbart (1963)]
  • 1980 -- Ted Nelson (Xanadu) -- everything is intertwinkled,  [Nelson (1980)]
Vannevar Bush' seminal paper As we may think may be regarded as the origin of what is known as hypertext with which, even if you don't know the phrase, every one of you is familiar, since it is (albeit in a rather simple way) realized in the web.

The phrase hypertext was invented by Ted Nelson (not patented, as far as I know), who looked for a less constraining way to organize information then was common in the educational system he grew up with. But before that, Douglas Engelbarth, who incidently invented the mouse, developed the Augment system to, as he said, boost the human intellect. What for, you may ask. Let me quote the series of flashes that Engelbarth went through, according to Dust or Magic  [Klabbers (2006)]:

  • flash 1: we are in trouble (human mankind)
  • flash 2: we need to boost mankind's ability to deal with complex urgent problems
  • flash 3: aha, graphic vision surges forth of me ...
  • flash 4: hypermedia -- to augment the human intellect
  • flash 5: augment (multimedia) workstation -- portal into an information space

classification of hypermedia

Perhaps it is good to know that Vannevar Bush wrote his article when working for an information agency in the second world war period. From that perspective, we can easily see that hypermedia (combining hypertext and multimedia) were thought of an instruments of intelligence.

Basically, hypermedia systems must be able to deal with:

hypermedia systems

  • components -- text, graphics, audio, video
  • links -- relations between components
  • presentation -- structured display
Far from being a definition, this characterization gives some insight in what functionality hypermedia systems must support. Recall that dealing with complex information is what hypermedia is all about.

Is this a natural way to deal with information? Just think about how you are taught to deal with information and how you actually go about with it. Speaking about Ted Nelson,  [Klabbers (2006)] observed that he realized that this intertwingularity was totally at odds with the education system he spent so long in and had been so uncomfortable with. Quoting Ted Nelson himself from his book Literary Machines:

A curriculum promotes a false simplification of any subject, cutting the subject's many interconnections and leaving a skeleton of sequence which is only a charicature of its richness and intrinsic fascination.

Judge for yourself. Would you prefer to have an 'immersive' course in multimedia rather than a more or less ordered collection of abstractions?

True enough, the visions of the pioneers of hypermedia where overwhelming. Nevertheless, the concept of hypermedia, that is non-linear media with machine-supported links, or 'text' as a network, found an application in a large variety of systems, see  [McKnight et al. (1991)].

classification of hypermedia systems

  • macro-literary systems -- publishing, reading, criticism
  • problem exploration tools -- authoring, outlining, programming
  • browsing systems -- teaching, references, information
  • general hypermedia technology -- authoring, browsing, collaboration
  • embedded hypermedia -- CASE, decision support, catalogs
An example of a hypermedia system that has extensively been used in education, for example biology and chemistry classes, is the Brown University Intermedia system of which supports so-called information webs, consisting of documents and links, that could both be retrieved by specifying attribute, allowing in this way for respectively both filtered content and conditional navigation. An interesting aspect of this system is that the user may create maps, that is structures containing documents and links, which form a personalized version of the web of information for a specific user, superimposed on the information space offered by the system.



Dexter Hypertext Reference Model

After many years of developing ideas and exploring implementations, one group of experts in the field came together and developed what is commonly known as the Dexter Hypertext Reference Model, named after the location, actually a pub, where the meetings were held. The Dexter model offers an abstract description of hypertext. It made a distinction between components, anchors within components and links between components, attached to anchors. The model was meant as a reference standard against which existing and future hypertext systems could be compared.

Components have the following attributes:


  • content -- text, graphics, video, program
  • attributes -- semantic description
  • anchors -- (bi-directional) links to other documents
  • presentation -- display characteristics
The Dexter Hypertext Model has been criticised from the beginning. Among others, because compound documents, that is documents having subcomponents, where not adequately dealt with. And also because it did not accomodate multimedia (such as video) content very well. In practice, however, the Dexter model has proven to be even somewhat overambitious in some respects. For example, the web does (currently) not support bi-directional links in a straightforward manner.

Amsterdam Hypermedia Model

When looking for
alternatives, a Dutch multimedia research group at CWI proposed to extend the Dexter model with their own multimedia model (CMIF), an extension for which they coined the name Amsterdam Hypermedia Model.

Let's look at the (CMIF) multimedia model first:

(CMIF) multimedia model

  • data block -- atomic component
  • channel -- abstract output device
  • synchronization arc -- specifying timing constraints
  • event -- actual presentation
What strikes as an immediate difference with respect to the hypertext model is the availability of channels, that allow for presenting information simultaneously, and so-called synchronization arcs, that allow the author to specify timing constraints. Also, events are introduced in the model to deal with user interactions.

With respect to authoring, the model supports a declarative approach to specifying sequential and parallel compounds, that is in what order specific things must be presented and what may ocuur simultaneously. Again, channels may be employed to offer a choice in the presentation, for example a dutch or english account of a trip in Amsterdam, dependent on the preferences of the (human) viewer.

The Amsterdam Hypermedia Model (AHM) extends the Dexter Hypertext Reference Model in a rather straigthforward way with channels and synchronization arcs.

Amsterdam Hypermedia Model

  • contents -- data block
  • attributes -- semantic information
  • anchors -- (id, value)
  • presentation -- channel, duration, ...
Obviously, the difference between Dexter and AHM is primarily the more precise definition of presentation characteristics, by introducing channels as in the (CMIF) multimedia model. Another (major) difference lies in the characterization of compounds. Each compound has one or more children, or subcomponents. Subcomponents may act as the source or destination of synchronization arcs. Each component obtains a start-time, that may result from parallel or sequential composition and synchronisation arcs.

Another interesting concept introduced by the Amsterdam Hypermedia Model is the notion of context. What happens when you click on a link? Does everything change or are only some parts affected? Then, when you return, does your video fragment start anew or does it take up where you left it? Such and other issues are clarified in the Amsterdam Hypermedia Model, of which we have omitted many details here.

It is perhaps interesting to know that the Amsterdam Hypermedia Model has served as a reference for the SMIL standard discussed in section 3.2. If you want to know more about the Amsterdam Hypermedia Model, you may consult  [Ossenbruggen (2001)] or  [Hardman et al. (1994)].



example(s) -- hush

In the hush we explore a variety of hypermedia applications. In fact already in 1994 we developed a SGML-based browser with applets in Tcl/Tk. Somehow, we did a lot with music with optimistic titels such as Bringing music to the Web,  [Ossenbruggen & Eliens (1994)] and more pessimistic ones such as Jamming (on) the Web,  [Eliens et al. (1997)]. The acronym hush stands for hyper utility shell. Many of the projects with hush were student projects, in which we studied operational support for hypermedia applications. Although we used SGML for markup, we did not have any specific document model, as in  [CMIF]. An overview and rationale of hush is given in  [Eliens (2000)]. A significant part of the hush software is being reused in the ViP system, that is discussed in section 4.3, albeit with an entirely different presentation technology.

research directions -- computational models

Today, hypermedia functionality is to some extent embedded in almost all applications. However, to realize the full potential of hypermedia, and in effect the networked multimedia computer, there are still many (research) issues to be resolved. To get an impression of the issues involved, have a look at the famous seven hypermedia research issues formulated by Halasz.

research issues

  • search and query -- for better access
  • composition -- for imposing structure
  • virtual structures -- on top of existing structures
  • computation -- for flexibility and interaction
  • versioning -- to store modification histories
  • collaborative work -- sharing objects with multiple users
  • extensibility and tailorability -- to adapt to individual preferences
See  [Ossenbruggen (2001)], section 2.3 for a more extensive description. Although the research issues listed above were formulated quite early in the history of hypermedia, as a reflection on the requirements for second-generation hypermedia systems, they remain valid even today. Without going into any detail with respect to the individual research issues, I rather wish to pose the grand encompassing research issue for the networked multimedia computer: What is the proper computational model underlying hypermedia or, more generally, for applications that exploit the networked multimedia computer in its full potential? Some directions that are relevant to this issue will be given in section 3-3 which deals with the multimedia semantic web.

multimedia authoring

It is tempting to identify a presentation with the information space it presents. This is what users often do, and perhaps should do. When that happens, the presentation is effective. But you must remember that the actual presentation is just one of the many possible ways to engage a user in exploring an information space. Making the choice of what to present to the user is what we understand by (multimedia) authoring.

Authoring is what we will discuss in this section. Not by giving detailed guidelines on how to produce a presentation (although you may look at the online assignment for some hints in this respect), but rather by collecting wisdom from a variety of sources.




Let's start with our explorations by looking at the problem of visualisation with a quote from David Gelernter, taken from  [Shneiderman (1997)]:


Grasping the whole is a gigantic theme, intellectual history's most important.

Ant vision is humanity's usual fate; but seeing the whole is every thinking person's aspiration.

David Gelernter, Mirror Worlds 1992

Now, consider, there are many ways in which the underlying information space may be structured, or speaking as a computer scientist, what data types may be used to represent the (abstract) information.

data types

  • 1-D linear data -- text, source code, word index
  • 2-D map data -- floor plan, office layout
  • 3-D world -- molecules, schematics, ...
  • temporal data -- 1 D (start, finish)
  • multi-dimensional data -- n-dimensional (information) space
  • tree data -- hierarchical
  • network data -- graph structure
The visualisation problem then is to find a suitable way to present these structures to the user. Basicall, following  [Shneiderman (1997)], there are two paradigms to present this information:

  • interactive -- overview first, zoom and filter, then details on demand
  • storytelling -- as a paradigm for information presentation
Storytelling may be very compelling, and does not force the user to interact. On the other hand, storytelling may lead to information consumerism alike to television enslavement.

An interaction paradigm that combines 'storytelling' with opportunities for interaction, as for example in the blendo approach discussed in section 3.2, would seem to be most favorable. Interaction then may result in either changing the direction of the story, or in the display of additional information or even transactions with a third party (for example to buy some goodies).

persuasive technology

Whatever your target audience, whatever your medium, whatever your message, you have to be convincing if not compelling.

In the tradition of rethorics, which is the ancient craft of convincing others, a new line of research has arisen under the name of persuasive technology. In the words of my collegue, Claire Dormann, persuasion is:


  • a communication process in which the communicator seeks to elicit a desired response from his receiver
  • a conscious attempt by one individual to change the attitudes, beliefs or behaviours of another individual or group individual through the transmission of some messages.
In other words, the purpose of persuasion is to accomplish one of the following goals: to induce the audience to take some action, to educate the audience (persuade them to accept to accept information or data), or to provide the audience with an experience. In the area of multimedia, one may think of many applications. Quoting Claire Dormann, in interactive media, the field of application of persuasive technology ranges from E-commerce, social marketing (like an anti-AIDS campaign) to museum exhibits. Also E-commerce provides an obvious example. To convince people to buy more, more persuasive messages and technologies are developed through the use of humorous and emotional communication, agents (such as price finders) or 3D representations of products and shops. For health campaigns (or any campaign of your choice) one can imagine 3D information spaces with agents presenting different point of views and where users are given different roles to play. In a museum you might want to highlight key points through innovative and fun interactive exhibits. Although the subject of persuasive technology is far less technology-oriented than the name suggests, multimedia (in a broad sense) form an excellent platform to explore persuasion. As concerns multimedia authoring, set yourself a goal, do the assignment, explore your capabilities, convey that message, and make the best of it.


What can you hope to achieve when working with the new media? Think about it. Are the new media really new? Does anyone want to produce something that nobody has ever seen or heard before? Probably not. But it takes some philosophy to get that sufficiently clear.

In  [Bolter and Grusin (2000)], the new media are analyzed from the perspective of remediation, that is the mutual influence of media on eachother in a historical perspective. In any medium, according to  [Bolter and Grusin (2000)], there are two forces at work:


  • immediacy -- a tendency towards transparent immersion, and
  • hypermediacy -- the presence of referential context
Put in other words, immediacy occurs when the medium itself is forgotten, so to speak, as is (ideally) the case in realistic painting, dramatic movies, and (perhaps in its most extreme form) in virtual reality. Hypermediacy may be observed when either the medium itself becomes the subject of our attention as in some genres of modern painting, experimental literature and film making, or when there is an explicit reference to other related sources of information or areas of experience, as in conceptual art, many web sites, and also in CNN news, where apart from live reports of ongoing action, running banners with a variety of information keep the viewers up to date of other news facts.

Now, the notion of remediation comes into play when we observe that every medium draws on the history of other media, or even its own history, to achieve a proper level of immediacy, or 'natural immersion'. For example, Hollywoord movies are only realistic to the extent that we understand the dramatic intent of cuts, close-ups and storylines, as they have been developed by the industry during the development of the medium. As another example, the realism of virtual reality can only be understood when we appreciate linear perspective (which arose out of realistic Renaissance painting) and dynamic scenes from a first person perspective (for which we have been prepared by action movies and TV).

Even if you may argue about the examples, let it be clear that each (new) medium refers, at least implicitly, to another medium, or to itself in a previous historic phase. So, what does this mean for new media, like TV or virtual reality?

Let's start with virtual reality.  [Bolter and Grusin (2000)] comment on a statement of Arthur C. Clarke

Virtual Reality won't merely replace TV. It will eat it alive.

by saying that ... he is right in the sense that virtual reality remediates television (and film) by the strategy of incorporation. This strategy does not mean that virtual reality can obliterate the earlier visual point-of-view technologies, rather it ensures that these technologies remain as least as reference points by which the immediacy of virtual reality is measured. So, they observe "paradoxically, then, remediation is as important for the logic of transparency as it is for hypermediacy". Following  [Bolter and Grusin (2000)], we can characterize the notions of immediacy and hypermediacy somewhat more precisely.


  • epistemological: transparency, the absence of mediation
  • psychological: the medium has disappeared, presence, immersion


  • epistemological: opacity, presence of the medium and mediation
  • psychological: experience of the medium is an experience of the real
Now, sharpen your philosophical teeth at the following statement, taken from  [Bolter and Grusin (2000)], p. 224:

Convergence is the mutual remediation of at least three important technologies -- telephone, televison and computer -- each of which is a hybrid of technical, social and economic practice, and each of which offers its own path to immediacy.

The telephone offers the immediacy of voice or the interchange of voices in real-time.

Television is a point-of-view technology that promises immediacy through its insistent real-time monitoring of the world.

The computer's promise of immediacy comes through the combination of three-dimensional graphics, automatic (programmed) action, and an interactivity that television can not match.

As they come together, each of these is trying to absorb the others and promote its own version of immediacy.

Let us once more come back to virtual reality and its possible relevance in our information age,  [Bolter and Grusin (2000)], p. 225:: in the claim that new media should not be merely archival but immersive, the rhetoric of virtual reality finally enters in, with its promise of the immediacy of experience through transparency. . So, with respect to the new media, we may indeed conclude: what is in fact new is the particular way in which each innovation rearranges and reconstitutes the meaning of earlier elements. What is new about media is therefore also old and familiar: that they promise the new by remediating what has gone before. The true novelty would be a new medium that did not refer to the other media at all. For our culture, such mediation without remediation seems to be impossible.



example(s) -- mobius

Rurger van Dijk, a former student of mine, has implemented an interactive story in flash. The story is a romance, told with images displaying scenes from the life of the players, a young man and a young women. The user can choose perspectives, either the man's or woman's, and watch the story from that point of view. The story is both non-linear and circular. The scenes can be connected in various way, and order is not compulsory.

research directions -- narrative structure

Where do we go from here? What is the multimedia computer, if not a new medium? To close this section on multimedia authoring, let us reconsider in what way the networked multimedia computer differs from other media, by taking up the theme of convergence again. The networked multimedia computer seems to remediate all other media. Or, in the words of  [Murray (1997)]:


(p. 27) ... merging previously disparate technologies of communication and representation into a single medium.

The networked computer acts like a telephone in offering one-to-one real-time communication, like a television in broadcasting moving pictures, like an auditorium in bringing groups together for lectures and discussion, like a library in offering vast amounts of textual information for reference, like a museum in its ordered presentation of visual information, like a billboard, a radio, a gameboard and even like a manuscript in its revival of scrolling text.

In  [Murray (1997)], an analysis is given of a great variety of computer entertainment applications, varying from shoot-em-up games to collaborative interactive role playing.  [Murray (1997)] identifies four essential properties that make these applications stand out against the entertainment offered by other media, which include books and TV. Two key properties determine the interactive nature of computer entertainment applications:


  • procedural -- 'programmed media' ...
  • participatory -- offering agency
All applications examined in  [Murray (1997)] may be regarded as 'programmed media', for which interactivity is determined by 'procedural rules'. With agency is meant that the user can make active choices and thus influence the course of affairs, or at least determine the sequence in which the material is experienced.



Another common characteristic of the applications examined is what  [Murray (1997)] calls immersiveness. Immersiveness is determined by two other key properties:


  • spatial -- explorable in (state) space
  • encyclopedic -- with (partial) information closure
All applications are based on some spatial metaphor. Actually, many games operate in 'levels' that can be accessedonly after demonstrating a certain degree of mastery. Networked computer applications allow for incorporating an almost unlimited amount of information. Some of the information might be open-ended, with storylines that remain unfinished. Closure, then, is achieved simply by exhaustive exploration or diminishing attention.

multimedia authoring

Coming back to the question what the 'new medium', that is the networked multimedia computer, has to offer from the perspective of multimedia authoring, two aspects come to the foreground:

multimedia authoring

  • narrative format
  • procedural authorship
The narrative format is incredibly rich, offering all possibilities of the multimedia computer, including 3D graphics, real-time sound, text. In short, everything up to virtual reality. But perhaps the most distinguishing feature of the new medium is that true authorship requires both artistic capabilities as well as an awareness of the computational power of the medium. That is to say, authorship also means to formulate generic computational rules for telling a story while allowing for interactive interventions by the user. Or, as phrased in  [Murray (1997)], the new cyberbard must create prototypical stories and formulaic characters that, in some way, lead their own life and tell their stories following their innate (read: programmed) rules. In section 8.3 and appendix C, we will present a framework that may be used as a testbed for developing programmed narrative structures with embodied agents as the main characters.

development(s) -- mashup semantic(s)

The old media have a hard time to catch up with the new media. While TV still may be considered a mass-medium, it seems to be loosing ground to online games and, indeed, In a panel of experts, gathered to discuss the notion of crossmedia, all agreed that the development(s) commonly referred to as web 2.0 are here to stay:

web 2.0

video sharing / online gaming / social networking
Not only do these application areas appeal to the user(s), but moreover they seem to be fruitful from an entrepeneurial perspective as well. In other words, there is money in it!

The spirit of the shift of culture that characterizes these developments is well expressed in the following poem/rap from a local group, called daft punk:

daft punk -- technologic (cn / jp)

  Buy it, use it, break it, fix it.
  Trash it, change it, melt -- upgrade it.
  Change it, point it, zoom it, press it.
  Snap it, work it, quick -- erase it.
  Write it, out it, paste it, save it.
  Load it, check it, quick -- rewrite it.
  Plug it, play it, burn it, rip it.
  Drag and drop it, zip -- unzip it.
  Look it, fill it, curl it, find it.
  View it, coat it, jam -- unlock it.
  Surf it, scroll it, pose it, click it.
  Cross it, crack it, twitch -- update it.
  Name it, rate it, tune it, print it.
  Scan it, send it, fax -- rename it. 
  Touch it, bring it. Pay it, watch it.
  Turn it, leave it, stop -- format it.
From a more objective perspective, we may observe that information has become a commodity, that is easily re-used, or put together in different combinations, for different purposes.

In an extremely well-readible article, entitled: What Is Web 2.0 -- Design Patterns and Business Models for the Next Generation of Software, Tim O'Reilly, ponders on the question(s), what makes these things work, and why are they profitable? When we look at many of these new applications or mashups, for example those using google maps, that these are:


  • substituting a single pragmatism for ideal design
  • using light weight programming models
In other words, where the original visions of hypertext and hypermedia suffered from megalomaniac ambitions such as boosting the human intellect, many mashups simply provide a useful service or entertaining content. And in the same vein, where software engineering principles dominated the early hypermedia systems, the new mashups are often no more than a simple hack, exploiting existing services in a clever way. With great effect!

O'Reilly also sketches the shift that characterizes the underlying economic model of these development(s), that is the growth of the original web into the web 2.0, and beyond:

web 2.0 design pattern(s)

  • web 1.0 -- the web as platform
  • web 2.0 -- architecture of participation
  • web 3.0 -- data is the (intel) inside
The gist of these characterizations should be clear, service-oriented, and with a clear eye to the data that makes service(s) worthwhile, and profitable.

In a study, investigating how to use web services to enhance Second Life,  [Eliens et al. (2007c)], we wrote: by now the phrase web 2.0 as well as applications representing it, such as Flickr and YouTube, are well established, and enjoyed by a wide community. Each day new items are added to the growing list of mashups, and the number of web services that constitute the building blocks of mashups also shows a steady growth. Mashups seem to be the easy way to start up a company, since the technology is relatively easy and, making use of appropriate services, initial investment costs can be low. Cf.  [Amazon].

What web 2.0 stands for, from a technical perspective, is succinctly expressed in Dorai's:

Learnlog: XML Is The Fabric Of Web 2.0 Applications

  • the client side is AJAX (Asynchronous Javascript and XML)
  • the server application typically exposes data through XML
  • the interaction model is web services
  • mashups combine multiple webservices to create new types of applications
And eventhough many alternative representations, such as JSON (Javascript Object Notation) are increasingly being used, all in all XML may be regarded as the interlingua of the Web 2.0.

Before taking a closer look at the communication protocol(s) underlying web 2.0 and de-construct the tight link of AJAX to HTML in-page formatting, it is worthwhile, following  [Amazon], to give an overview of a selected number of services, that may be used to create mashups:


Although mashups featuring google maps seem to be the dominant mashup type, other services such as offered by, Flickr and BBC might prove to be more worthwhile for 'serious' applications. For example, for developing e-commerce applications Amazon offers services for product operations, such as item search and similarity lookup, remote shopping carts, to create and manage purchase collections, customer content, to access information contributed by customers, and third party listings, to find related resellers. It is important to note that many of these services, as for example the shoppong cart services, may be used independently of the commercial offerings of Amazon!

Most of the service providers and services mentioned above are accessible using a choice of protocols, including WSDL, SOAP, XML-RPC and the REST protocol. The REST protocol seems to be most widespread and as we will discuss in the next section, it seems to be tho most appropriate protocol in Second Life.

REST stands for Representational State Transfer. In essence, the REST protocol uses the url as a command-line for stateless RPC invocations, which allows for services to be executed by typing in the address box of a web browser. A great tutorial about the REST protocol can be found in Joe Gregorio's column column: The Restful Web. As fully explained in  [Visser and Eliens (2000)], the phrases representation, state and transfer, respectively, stand for:


  • representation -- encoding in a particular format
  • state -- data encapsulated in an object
  • transfer -- using HTTP methods
In practice, the use of REST means that the state associated with a resource or service must be managed by the client. Together with mechanisms such as content-negotiation and URL-rewriting, REST provides a simple, yet powerful method to invoke services using HTTP requests.

A common misunderstanding is that AJAX is intimately tied to web browsers and in-page HTML formatting. This misunderstanding is due to the fact that AJAX is often used to improve the user experience of web pages bij emulating RIA (Rich Internet Applications) using DHTML and CSS. However, the real meaning of AJAX in our view is that AJAX allows for asynchronous client-controlled server requests, that are executed without an immediate visible effect for the user.

The web 2.0 offers a lively arena for consumers and developers alike, with a multitude of blogs discussing the future of the web. For example, in Dion Hinchcliffe rebuttal of Jeffrey Zeldman's Web 3.0 – Web 1.0 = Web 2.0 blog, entitled Is Web 2.0 Entering "The Trough of Disillusionment"? it is suggested that our services could even be more powerful by creating semantic mashups. Although the notion of sematic web technology is widely known and accepted, we include for reference a characterization of Nova Spivack quoted from Dan Farber and Larry Dignan's blog Web 2.0 isn’t dead, but Web 3.0 is bubbling up:

The Semantic Web is a set of technologies which are designed to enable a particular vision for the future of the Web – a future in which all knowledge exists on the Web in a format that software applications can understand and reason about. By making knowledge more accessible to software, software will essentially become able to understand knowledge, think about knowledge, and create new knowledge. In other words, software will be able to be more intelligent, not as intelligent as humans perhaps, but more intelligent than say, your word processor is today.

But even in the semantic web community the discussion whether to go for folksonomies or formal ontologies rages,  [Folk], and it is not clear at this stage what will prove to be more powerful, HTML-scraping, tags, microformats, or full ontologies.

Instead of joining this perhaps endless discussion, let us explore what is involved in incorporating web services in Second Life, and how to realize meaningful mashups in 3D virtual environments. Nevertheless, to conclude this brief overview of web services and mashups I wish to give another quote from Dorai's Learnlog, this time from Jon Udell, in his blog on his move to Microsoft:

the most powerful mashups don't just mix code and data, they mix cultures.

which provides a challenge that trancends all issues of mere technological correctness.

using web services in Second Life

Second Life offers an advanced scripting language with a C-like syntax and an extensive library of built-in functionality. Although is has support for objects, LSL (the Linden Scripting Language) is not object-oriented. Cf.  [Eliens (2000)]. Scripts in Second Life are server-based, that is all scripts are executed at the server, to allow sharing between visitors. Characteristic for LSL are the notions of state and eventhandler, which react to events in the environments.

Among the built-in functions there are functions to connect to a (web) server, and obtain a response, in particular (with reference to their wiki page):


Other functions to connect to the world include sensors, for example to detect the presence of (visitors') avatars, and chat and instant messaging functions to communicate with other avatars using scripts. In addition, LSL offers functions to control the behavior and appearance of objects, including functions to make objects react to physical laws, to apply force to objects, to activate objects attached to an avatar (as for example the phantom Mario sprites mentioned earlier), and functions to animate textures, that can be used to present slide shows in Second Life.

On the Mashable Social Networking News site a brief overview is given of the use of web services in Second Life, entitled Second Life + Web 2.0 = Virtual World Mashups. To access Second Life from outside-in (that is from a web browser), so-called slurls may be used, for example to reach VU @ Second Life, and all slurls listed in under slurlmarker may be used, also to activate in-world teleporting using scraping techniques.

As remarked in the hackdiary by Matt Biddulph, Second Life (currently) lacks the ability to parse XML or JSON, so the best way to incorporate web services is to set up a web server with adequate resources. As Matt Biddulph indicates, to access flickr photographs for a particular user (avatar), a web server may contain the following resources:


  • /seen?user=SomeAvatar -- records the presence of SomeAvatar
  • /touched?user=SomeAvatar -- invokes flickr API with users tag
  • /set_tag?user=SomeAvatar&tag=FavoriteTag -- records SomeAvatar's favourite tag
For example, in response to a 'touch' event, invoking touch results in consulting the database for the user's tag and asking the Flickr API for a random photo with that tag. It then returns a string containing the url for a particular photograph. LSL functions used in this application include sensors, to check for presence, listen functions, to respond to spoken commands, and touch events, for the physical interface. In addition to supporting strings and lists, LSL provides a perl-like split function to convert a string into a list of strings, thus allowing for processing multiple items in response to a server request.

Another example of using web services in Second Life is writing blogs from within Second Life using the BlogHUD developed by Koz Farina who also is reported to have found a flash hack that allows for reading RSS feeds. As explained by Koz Farina:

flash/quicktime in SL

Quicktime supports Flash, but only up to Flash version 5. We're up to version 9 on that now! Luckily, I have been dabbling with Flash since the early days, so already knew how to do this 'the old way'... So, Flash is doing all the work. No LSL at all... I heart feeds. Did I say 'I heart feeds?

The RSS display uses the ability to stream Quicktime video in Second Life, and again the mashup is not created in Second Life but by appropriate server support.

In a similar vein we may incorporate live streaming video, for example by using WireCast to capture and organize live camera input, possibly together the screen output of other applications such as powerpoint, which must then be sent to a streaming server supporting Quicktime, such as Apple's Darwin, which may then be accessed from Second Life to texture a display object.

Finally, as another Web 2.0 to Web 3D phenomenon, announced in New World Notes, we may mention the used of Twitter messages, that allow residents to send and receive message about ongoing activities. A similar service is reported to exist for jaiku messages.

Referring to section 7.4 for a more detailed discussion, we may observe that there is no meaning in merely putting things together. Without mechanisms of personalization and recommendation we would simply be flooded by data and information, in a way that even search would not be able to cope with. Context, narratives and personalized presentation(s), notions from the past, reappear as keywords for the future of the web 2.0 and beyond.




2. information spaces



projects & further reading

As a project, I suggest the development of a virtual tour in a city, museum or other interesting locatoion.

You may further explore the implementation of traversal within a context, taking into account the history of navigation when backtracking to a particular point, issues in hyperlinking and interaction in multimedia applications, and computational support for narratives.

For further reading I advice you to take a look at the history of hypermedia and the web, using online material from the W3C, or the history of media as accounted for in  [Briggs and Burke (2001)] and  [Bolter and Grusin (2000)].

the artwork

  1. book covers --  [Weishar (1998)],  [Eco (1994)],  [Burger (1981)],  [Kunst],  [Betsky (2004)]
  2. Federico Campanale -- Oxygen, fragments from video installation, 2004
  3. Vasarely --  [Diehl 1973].
  4. Vasarely --  [Diehl 1973].
  5. Vasarely --  [Diehl 1973].
  6. Federico Campanale -- Oxygen, more fragments.
  7. student work -- from introduction multimedia 2000.
  8. Rutger van Dijk -- mobius, interactive story, opening screen, see section 2.3.
  9. edgecodes -- screenshots, see section 2.3
  10. signs -- people,  [ van Rooijen (2003)], p. 244, 245.
The work of Vasarely has served as an example for many contemporary digital artists. It is playful, mat may be characterized also as formalist. The highly aesthetic video work of Federico Campanale who, as he told me was strongly influenced by vasarely in his early years, shows a similar combination of formalism and playfulness. The interactive story by Rutger van Dijk has a rather different atmosphere, it is highly romantic, with slick graphics. The musea sites are included to point to the existence of (an increasing number) of virtual tours.

(C) Æliens 23/08/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.