topical media & game development

talk show tell print



a multimedia semantic web?

To finish this chapter, let's reflect on where we are now with 'multimedia' on the web. Due to refined compression schemes and standards for authoring and delivery, we seemed to have made great progress in realizing networked multimedia. But does this progress match what has been achieved for the dominant media type of the web, that is text or more precisely textual documents with markup?

web content

Commonly, a distinction is made between successive generations of web content, with the first generation being simple hand-coded HTML pages. The second generation may be characterized as HTML pages that are generated on demand, for example by filling in templates with contents retrieved from a database. The third generation is envisaged to make use of rich markup, using XML, that reflects the (semantic) content of the document more directly, possibly augmented with (semantic) meta-data that describe the content in a way that allows machines, for example search engines, to process it. The great vision underlying the third generation of web content is commonly refered to as the the semantic web. which enhances the functionality of the current web by deploying knowledge representation and inference technology from Artificial Intelligence, using a technology known as the Resource Description Framework (RDF). As phrased in  [CWI], the semantic web will bring
structure to the meaningful content of web pages,
thus allowing computer programs,such as search engines and intelligent agents, to do their job more effectively. For search engines this means more effective information retrieval, and for agents better opportunities to provide meaningful services.

A great vision indeed. So where are we with multimedia? As an example, take a shockwave or flash presentation showing the various musea in Amsterdam. How would you attach meaning to it, so that it might become an element of a semantic structure? Perhaps you wonder what meaning could be attached to it? That should not be too difficult to think of. The (meta) information attached to such a presentation should state (minimally) that the location is Amsterdam, that the sites of interest are musea, and (possibly) that the perspective is touristic. In that way, when you search for touristic information about musea in Amsterdam, your search engine should have no trouble in selecting that presentation. Now, the answer to the question how meaning can be attached to a presentation is already given, namely by specifying meta-information in some format (of which the only requirement is that it is machine-processable). For our shockwaveor flash presentation we cannot dothis in a straightforward manner. But for MPEG-4 encoded material, as well as for SMIL content, such facilities are readily available.

Should we then always duplicate our authoring effort by providing (meta) information, on top of the information that is already contained in the presentation? No, in some cases, we can also rely to some extent on content-based search or feature extraction, as will be discussed in the following chapters.



Resource Description Framework -- the Dublin Core

The Resource Description Framework, as the W3C/RDF site informs us integrates a variety of applications from library catalogs and world-wide directories to syndication and aggregation of news, software, and content to personal collections of music, photos, and events using XML as an interchange syntax. The RDF specifications provide, in addition a lightweight ontology system to support the exchange of knowledge on the Web.

The Dublin Core Metadata Initiative is an open forum engaged in the development of interoperable online metadata standards that support a broad range of purposes and business models.

What exactly is meta-data? As phrased in the RDF Primer

meta data

Metadata is data about data. Specifically, the term refers to data used to identify, describe, or locate information resources, whether these resources are physical or electronic. While structured metadata processed by computers is relatively new, the basic concept of metadata has been used for many years in helping manage and use large collections of information. Library card catalogs are a familiar example of such metadata.

The Dublin Core proposes a small number of elements, to be used to give information about a resource, such as an electronic document on the Web. Consider the following example:

Dublin Core example

      <rdf:Description rdf:about="">
        <dc:title>An Introduction to the Resource Description Framework</dc:title>
        <dc:creator>Eric J. Miller</dc:creator>
        <dc:description>The Resource Description Framework (RDF) is an
         infrastructure that enables the encoding, exchange and reuse of
         structured metadata. rdf is an application of xml that imposes needed
         structural constraints to provide unambiguous methods of expressing
         semantics. rdf additionally provides a means for publishing both
         human-readable and machine-processable vocabularies designed to
         encourage the reuse and extension of metadata semantics among
         disparate information communities. the structural constraints rdf
         imposes to support the consistent encoding and exchange of
         standardized metadata provides for the interchangeability of separate
         packages of metadata defined by different resource description
         communities. </dc:description>
        <dc:publisher>Corporation for National Research Initiatives</dc:publisher>
            <rdf:li>machine-readable catalog record formats</rdf:li>
            <rdf:li>applications of computer file organization and
             access methods</rdf:li>
        <dc:rights>Copyright  1998 Eric Miller</dc:rights>
        <dc:type>Electronic Document</dc:type>
        <dcterms:isPartOf rdf:resource=""/>
Items such as title, creator, subject and description, actually all tags with the prefix dc, belong to the Dublin Core and are used to give information about the document, which incidentally concerns an introduction to the Resource Description Framework. The example also shows how rdf constructs can be used together with the Dublin Core elements. The prefixes rdf and dc are used to distinguish between the distinct namespaces of respectively RDF and the Dublin Core.

The Dublin Core contains the following elements:

Dublin Core

  • title -- name given to the resource
  • creator -- entity primarily responsible for making the content of the resource
  • subject -- topic of the content of the resource
  • description -- an account of the content of the resource
  • publisher -- entity responsible for making the resource available
  • contributor -- entity responsible for making contributions to the content of the resource
  • date -- date of an event in the lifecycle of the resource
  • type -- nature or genre of the content of the resource
  • format -- physical or digital manifestation of the resource
  • identifier -- unambiguous reference to the resource within a given context
  • source -- reference to a resource from which the present resource is derived
  • language -- language of the intellectual content of the resource
  • relation -- reference to a related resource
  • coverage -- extent or scope of the content of the resource
  • rights -- information about rights held in and over the resource
In section 10.3 we discuss an application in the domain of cultural heritage, where the Dublin Core elements are used to provide meta information about the information available for the conservation of contemporary artworks.



research directions -- agents everywhere

The web is an incredibly rich resource of information. Or, as phrased in  [IR]:

information repository

The Web is becoming a universal repository of human knowledge and culture, which has allowed unprecedented sharing of ideas and information in a scale never seen before.

Now, the problem (as many of you can acknowledge) is to get the information out of it. Of course, part of the problem is that we often do not know what we are looking for. But even if we do know, it is generally not so easy to find our way. Again using the phrasing of  [IR]:

browsing & navigation

To satisfy his information need, the user might navigate the hyperspace of web links searching for information of interest. However, since the hyperspace is vast and almost unknown, such a navigation task is usually inefficient.

The solution of the problem of getting lost in hyperspace proposed in  [IR] is information retrieval, in other words query & search. However, this may not so easily be accomplished. As observed in  [IR], The main obstacle is the absence of a well-defined data model for the Web, which implies that information definition and structure is frequently of low quality. Well, that is exactly the focus of the semanics web initiative, and in particular of the Resource Description Framework discussed above.

Standardizing knowledge representation and reasoning about web resources is certainly one (important) step. Another issue, however, is how to support the user in finding the proper resources and provide the user with assistance in accomplishing his task (even if this task is merely finding suitable entertainment).

What we need, in other words, is a unifying model (encompassing both a data model and a model of computation) that allows us to deal effectively with web resources, including multimedia objects. For such a model, we may look at another area of research and development, namely intelligent agtents, which provides us not only with a model but also with a suitable metaphor and the technology, based on and extending object-oriented technology, to realize intelligent assistance, [OO].

For convenience, we make a distinction between two kinds of agents, information agents and presentation agents.

information agent

  • gather information
  • filter and select
Information agents are used to gather information. In addition, they filter the information and select those items that are relevant for the user. A key problem in developing information agents, however, is to find a proper representation of what the user considers to be relevant.

presentation agent

  • access information
  • find suitable mode of presentation
Complementary to the information agent is a presentation agent (having access to the information gathered) that displays the relevant information in a suitable way. Such a presentation agent can have many forms. To appetize your phantasy, you may look at the vision of angelic guidance presented in  [Angelic]. More concretely, my advice is to experiment with embodied agents that may present information in rich media 3D. In section 7-3, we will present a framework for doing such experiments.



navigating information spaces

Having agents everywhere might change our perspective on computing. But, it may also become quite annoying to be bothered by an agent each time that you try to interact with with your computer (you know what I mean!). However, as reported by Kristina Höök, even annoyance can be instrumental in keeping your attention to a particular task. In one of her projects, the PERSONAS project, which stands for

PERsonal and SOcial NAvigation through information spaceS

the use of agents commenting on people navigating information space(s) is explored. As a note, the plural form of spaces is mine, to do justice to the plurality of information spaces.

As explained on the PERSONAS web site, which is listed with the acronyms, the PERSONAS project aims at:


investigating a new approach to navigation through information spaces, based on a personalised and social navigational paradigm.

The novel idea pursued in this project is to have agents (Agneta and Frieda) that are not helpful, but instead just give comments, sometimes with humor, but sometimes ironic or even sarcastic comments on the user's activities, in particular navigating an information space or (plain) web browsing. As can be read on the PERSONAS web site:

Agneta & Frieda

The AGNETA & FRIDA system seeks to integrate web-browsing and narrative into a joint mode. Below the browser window (on the desktop) are placed two female characters, sitting in their livingroom chairs, watching the browser during the session (more or less like watching television). Agneta and Frida (mother and daughter) physically react, comment, make ironic remarks about and develop stories around the information presented in the browser (primarily to each other), but are also sensitive to what the navigator is doing and possible malfunctions of the browser or server.

In one of her talks, Kristina Höök observed that some users get really fed up with the comments delivered by Agneta and Frieda. So, as a compromise, the level of interference can be adjusted by the user, dependent on the task at hand.

Agneta & Frieda

In this way they seek to attach emotional, comical or anecdotal connotations to the information and happenings in the browsing session. Through an activity slider, the navigator can decide on how active she wants the characters to be, depending on the purpose of the browsing session (serious information seeking, wayfinding, exploration or entertainment browsing).

As you may gather, looking at the presentations accompanying this introduction to multimedia and  [Dialogs], I found the PERSONAS approach rather intriguing. Actually, the PERSONAS approach is related to the area of affective computing, see  [Affective], which is an altogether different story.

The Agneta and Frieda software is available for download at the PERSONAS web site.

(C) Æliens 04/09/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.