topical media & game development

talk show tell print

I-GUARD -- Intelligent Guidance in Artist's Digital Dossiers



1a) Intelligent Guidance in Artist's Digital Dossiers


1c) Dr. A. Eliëns

Address: Vriej Universiteit, Faculty of Sciences/Informatica, De Boelelaan 1081, 1081 HV Amsterdam, email:

2) Summary:

2a) Summary

Contemporary art is an intrinsic part of our cultural heritage. Installations, performances, video and other forms of media art, have the interest of a small group of adherents, but are in comparison with more traditional art forms, far more difficult to present to a general audience. Another problem presents itself, due to the type of materials used and the context-specific aspects of these art forms, in the conservation of the works.

We focus on intelligent navigation and interaction with the the digital dossier. that contains all (multimedia) information necessary for an adequate understanding of one or more works of art.

An essential part of our research is to support guided tours, that may take the form of a walk-through in virtual space, an automated story-line in a concept map, or allow the (professional) to experiment with the (de-)construction of for example an art installation, or to experiment with exhibition parameters, such as light and space. An important research aspect here is to provide for the possibility to adapt the guided tour to the personal information need of the user.

The results of our research will on the one hand contribute to making contemporary art forms accessible to a larger audience and on the other hand are explicitly meant to support the complex task of the conservation and re-installation of works of art in an effective manner. These results are also of interest for presenting more traditional forms of cultural heritage, in particular with respect to the development of personalised guided tours.

2b) Abstract for layman (in Dutch)

Installaties, performances, video en ander vormen van media kunst, hebben de interesse van een kleine groep liefhebbers, maar zijn in vergelijk met meer traditionele kunstvormen, zoals de schilderkunst, moeilijker toegankelijk te maken voor een breed publiek. Ook doen zich, door ondermeer materiaal gebruik en de situatie-gebonden aspecten van deze kunstvormen, problemen voor bij de conservering ervan.

In ons onderzoek richten we ons op intelligente navigatie en interactie met het digitale dossier, dat de (multimedia) informatie bevat om een of meerdere kunstwerken adequaat te begrijpen.

Essentieel is de ondersteuning van guided tours, die de vorm kunnen aannemen van een 'walk through' in een virtuele ruimte, een 'story-line' in een concept map, danwel de (professionele) gebruiker in staat stellen tot interactie met het kunstwerk, zoals bijvoorbeeld de (de-)constructie of het experimenteren met exhibitie-parameters zoals licht en ruimte. Een belangrijk onderzoeks-aspect hier is het aanpassen van de guided tour aan de persoonlijke informatie behoefte van de gebruiker.

De resultaten van ons onderzoek zullen enerzijds een bijdrage leveren aan het toegankelijk maken van hedendaagse kunstvormen voor een breder publiek en zijn anderzijds expliciet bedoeld de ingewikkelde taak van de conservering en re-installatie van de kunstwerken effectief te ondersteunen. Ook voor meer traditionele vormen van cultureel erfgoed zijn deze resultaten van groot belang, met het oog op de ondersteuning van ge-personaliseerde guided tours in een media-rijke omgeving.

3) Classification:

computer science / digitale beleving

4) Composition of the Research Team:

Dr. A. Eliensmultimedia/game developmentVU/IMSE8 (supervision)
Drs. C. Visseragent technolgy/DLPVU 12 (programmer)
C. van Riel *)student MMCVU32-36 (OIO)
2 (promotor)

The research will be executed within the VU, under the daily supervision of Dr. A. Eliens
*) C. van Riel will obtain his master degree Information Science, within the specialisation Multimedia and Culture, 11 october 2006. As student/contributer to the digital dossier(s), he is the proposed candidate for the OiO position within the project.

5) Research School: SIKS

All project-members fall under SIKS (the Dutch Research School for Information and Knowledge Systems,

6) Description of the Proposed Research

Contemporary art is an intrinsic part of our cultural heritage. Installations, performances, video and other forms of media art, have the interest of a small group of adherents, but are in comparison with more traditional art forms, far more difficult to present to a general audience. Another problem presents itself, due to the type of materials used and the context-specific aspects of these art forms, in the conservation and re-installation of the works.

In our research we address the issue of providing access to these contemporary art forms from a wide variety of perspectives, ranging from the interested layman to the expert that has to deal with archiving, conserving and the possible re-installation of the art works.

Our research is centered around the notion of digital dossier, that is the collection of information needed to (properly) understand one or more works of art in the context of the oeuvre of the artist. The information consists of descriptive text, images depicting the work(s), and possibly video registrations as well as interviews with the artist. From the perspective of the expert, in charge of conserving the work(s), the dossier must contain all the information needed for (proper) re-installation.

For the presentation of the digital dossier we deploy a rich media presentation environment with which we gained ample experience in our research in intelligent multimedia, among others in previous pilots with ICN, in which we realized such environments for the Dutch artist Marinus Boezem, the Serbian-Dutch artist Marina Abramovic and the Australian media artist Jeffrey Shaw.

In our present research we will focus in particular on intelligent navigation and interaction with the digital dossier. Due to the wish and need to support the use of the dossier from a variety of perspectives, the presentation environment must offer a multitude of potential interaction scenarios, and accomodate the needs of the variety of users in an intelligent fashion.

Our research will focus in particular on the question of how to support guided tours in an adequate manner, with regard to the information need of the user. As we have demonstrated in the prototype implementations, a guided tour may take the form of a walk-through in virtual space, an automated story-line in a concept map, or allow the (professional) to experiment with the(de-)construction of for example an art installation, or to experiment with exhibition parameters, such as light and space. For adapting a guided tour to the personal information need of the user, there is a variety of solutions, as for example explored in the CATCH/CHIP project. In the I-GUARD project, we will investigate the use of filtering on concept node types in the concept map, according to the interests indicated by the user. In addition, we will explore how to utilize the navigation behavior of the user as an indication of his/her interests, to construct a personalized tour.

For our research, we will take the digital dossier developed for Marina Abramovic, in the sequel referred to as the abramovic dossier, as a starting point, since it provides the most clear and innovative solution to the problems of navigation and presentation:

  • navigation -- by means of a concept graph
  • presentation -- by means of a presentation gadget that allows for the simultaneous inspection of multiple media items
Apart from allowing the presentation of 2D information (text, images, and video), the abramovic dossier also allows for the incorporation of 3D models of artworks and installation in a seamless (immersive) way.

6a) Scientific aspects

Given the need to represent information of one or multiple works of art in a digital dossier, that allows for the presentation of that information in a rich media presentation environment, we address, from a research perspective, the following issues:

research issues

  • navigation using a concept graph
  • presentation of 2D and 3D information in a unified fashion
  • interaction with support for (personalized) guided tours
The presentation of information in a 3D environment will not be the central focus of our research, although our experience in this area is an essential pre-requisite for our research. Instead we will concentrate on navigation, presentation issues, and interaction with support for guided tours. In addition, we will tackle more broad issues of representation of information and investigate a generic approach to the construction of digital dossiers.


Given the complexity of the information space related to a work of art, interactive facilities must be provided for the user to gain a quick overview of the information available as well as the means to deepen the inspection rapidly, avoiding needless and cumbersome detours. At the same time, the user must be allowed to inspect the (multimedia) material related to the work of art, such as a collection of images or video recordings.

To accommodate the complex structure of the information space we have developed a 3D representation of a concept graph, a semantic device well-known from artificial intelligence, which allows the user to expand a node representing a particular concept or aspect into a collection of related nodes. Moreover, for the presentation of (multimedia) information we have developed a (generic) presentation gadget, that displays images or video and textual information simultaneously. The actual content of the presentation gadget depends on the node of the concept graph from which it is expanded. The presentation gadget allows for limited browsing following (textual) links, within the material presented. At each moment, the presentation gadget can be collapsed to continue navigation in the concept graph.

Using the concept graph in conjunction with the presentation gadget resolves the duality of information and presentation that is inherent for (multimedia) digital dossiers, and thus promotes what may be called the immersive experience of digital artist's dossiers.


Apart from the presentation gadget, which allows for the simultaneous inspection of multiple media items, in a variety of formats, we explored the use of guided tours as a means to present the information in a story-like way, relieving the user of the often cumbersome task to interact.

Guided tours, in the digital dossier, may take one of the following forms:

  • automated (viewpoint) navigation in virtual space,
  • the (narrative) presentation of a sequence of concept nodes,
  • an animation explaining, for example, the construction of an artwork.
In practice, an actual guided tour may be constructed as a combination of these elements, interweaving, for example, the explanation of concepts, or biographic material of the artist, with the demonstration of the positioning of an artwork in an exhibition space.

The research issues, here, encompass:

  • the selection of items in the guided tour, as well as
  • the presentation of these items in an immersive manner.
Immersion, here, means that we strive for a seamless transition between the presentation of conceptual material and visual material, such as video registrations or 3D models of the artworks.


Our aim is to arrive at a general framework for artist's digital dossiers, that provide intelligent guidance to both the expert user, responsible for the future re-installation of the work(s), and the interested layman, that wishes to get acquainted with a particular work or collection of works. In general, there are two techniques that we can apply to provide such guidance:
  • filtering the information space according to the user's perspective, and
  • intelligent agents, that (pro) actively aid the user in searching the information space.
Filtering the information space may be used to restrict the concept graph that defines the navigation structure, by stating assumptions with respect to the relevance of particular categories from a user's perspective.

Intelligent agents is an approach stemming from artificial intelligence which allows for providing guidance in a variety of ways, possibly even in an embodied form using a face or humanoid figure to give suggestions to the user on what interactions to perform. In  [Hoorn et al. (2004) ] we have investigated the use of embodied agents in a digital dossier for the artist Marinus Boezem. In our current research, however, we will very likely not use embodied agents. Nevertheless, we will investigate to what extent we can use an agent model, possibly with learning capabilities as explored in  [Hildebrand et al. (2003) ], to provide guidance and support interaction.

Our goal is to arrive at an advice function, that offers the user at any navigation point a choice of continuations and/or a selection of guided tours, focussing on a topic of interest.

For selecting the items to be presented in a guided tour, the most obvious way is to pre-define a sequence based on user profiles. Very likely this can be done in a more flexible way in a rule-based manner, applied to a template tour. More interesting, however, is to investigate whether guided tours can be generated dynamically based on tracking actual user interaction of (expert) users, using techniques from prediction theory, as explained in section 6b.

To allow for meaningful interaction with 3D models, allowing to view for example information about the materials used or its installation procedure, we must find a way to connect that information to user actions in a generic way. In other words, there is an information representation problem, namely, how to relate contextual information in a generic fashion to elements of a 3D model representing an artwork. Although such interactions can be realized by embedding invisible) action/event objects in the model, a more generic way of representing such relations is desirable, to avoid the need for the time-consuming hand-crafting for which in practice there may not even be the necessary (human) resources.

6b) Method of research

The method of research will primarily be explorative develepment of (digital dossier) applications, in cooperation with ICN and possibly other institutes of cultural heritage. Although by some denounced as a non-scientific method, we believe that the approach of explorative design and development is most fruitful in this area of research. See for a discussion also  [Grau (2003)].

Given the variety of information material related to artworks, and the need to present this material in a visually compelling way, we will apply techniques as used in the development of interactive narrative games, as described in  [Hawkins (2005)].

For the selection of items in guided tours and the generation of interesting sequences, we will explore the use of prediction theory. As explained in  [Cesa-Bianchi and Lucosi (2006)], prediction theory uses a model of prediction based on expert advice. However, instead of the traditional loss function, used in a stochastic approach, prediction theory uses a regret function, which expresses the difference between an actual prediction and the advice of a collection of experts. An expert, in this context, is an abstract entity, that may be either embodied by an algorithm, a random selection, or an actual expert.

We will investigate, for the construction of guided tours, whether it is possible to generate interesting sequences by using a (sequence of) prediction(s) that minimizes the regret function, which respect to the navigation sequence(s) recorded from actual expert users.

In particular, we will strive for implementing the advice function, in a generic way, by means of a learning mechanism that extracts recommended continuations and guided tours from tracking expert user navigation.

6c) Scientific interest

Although digital archives or digital libraries are by no means a new phenomenon, our concept of digital dossiers contains a number of innovative elements.

A digital dossier provides a unified information and presentation space. In this sense it differs significantly from a digital archive with a traditional web interface, where navigation and presentation are distinct. Digital dossiers allow to a much greater extent for an immersive experience of the information related to works of art. As such it is reminiscient to explorations in virtual archeology.

As concerns intelligent guidance, again, although agent technology itself is by no means new, we firmly believe that our use of agent technology offers exciting new ways in which intelligent guidance may be realized.

Generating guided tours based on (expert) user tracking is no doubt an interesting issue from a scientific point of view, eventhough we may not be able to guarantee that such tours will be interesting for the target audience.

6d) Relation to other research

In contrast with the majority of research (proposals) in the CATCH-framework, which are concerned with making a great mass of artworks accesible to a wide audience using (fairly) traditional techniques of presentation, our research addresses the more narrow area of contemporary artworks, which require the use of more advanced presentation technologies.

Our work has started as an extension to the INCCA project, which provides a digital archive of meta-information indicating sources of information on (the conservation of) contemporary art. However, as also indicated in the Culture 2000 project proposal, more advanced practices and techniques are needed to adequately capture the requirements for the re-installation of contemporary art works.

Our research aims at developing the (information) technologies needed to implement such practices and techniques, and more in particular to develop a framework for realizing digital dossiers with intelligent guidance, adapted to the particular perspective of its users, which may be conservation experts or intelligent laymen. The framework will be built on open standards and may eventually be donated to the (traditional) cultural heritage community.

6e) Relation to current research

The background of our research is provided by our previous research on intelligent multimedia, as described in  [IMVU]. Our intelligent multimedia platform supports interactive 3D graphics, multi-user virtual environments, as well as embodied agents. See  [Eliens et al. (2002) ],  [Community] and  [Huang et al. (2003) ]. It uses distributed logic programming,  [ Eliens (1992) ], and is built on object-oriented programming technologies,  [Eliens (2000) ]. It has been applied in student projects, done in cooperation with ICN, for developing digital dossiers for respectively Marinus Boezem, Marina Abramovic and Jeffrey Shaw.

As a complement to the theoretical and technical research we also plan to continue our empirical human-computer interaction studies, as reported in  [Hoorn et al. (2004) ], to assess the viability of our approach.

The contribution of the cultural heritage institutes, in particular ICN, will be to provide the material, assist in finding relevant categorizations and, inevitably, function as the touchstone against which our efforts will be evaluated.

Our current interest focusses on the use of game technology for immersive (serious) applications. This research provides the necessary background for the presentation of multimedia material as well as the construction of naaratives. See  [Eliens and Bhikharie (2006)],  [Eliens (2006b) ].

7) Work Programme

A brief summary of the issues that will be tackled and the deliverables that we expect to produce during the four years of the project looks as follows.

We distinguish, over the four years, between a theoretical/study track and a practical/application track.

theorectial track

  • year 1: identification of major issues -- study of prediction theory
  • year 2: adaptive guided tour -- study of personalisation and user profile issues
  • year 3: guided tour based on user tracking -- study of agent technology and narrative theory
  • year 4: writing of the thesis
In the first stage of the research, the work to be done by the proposed candidate, Chris van Riel, will be a follow-up on his master thesis, which describes a first exploration in the development of guided tours for digital dossiers.

target application(s)

  • year 1: extension of abraomiv dossier with user navigation tracking
  • year 2: prototype realization of guded tour based on user tracking
  • year 3: implementation of an advice function offering continuations and (limited) guided tours
  • year 4: development of a generic digital dossier construction kit

The candidate researcher will work in close cooperation with the other members of the intelligent multimedia research group, in particular dr A. Eliëns. to explore the topics mentioned and to apply the intelligent multimedia technology that is being developed to digital dossiers.

education track

The advanced education of the candidate researcher will mainly be taken care of by the standard courses offered by the SIKS research school. (See
In addition, the candidate is supposed to become familiar with the literature in multimedia and other projects in the domain of cultural heritage, in particular CATCH, with which there will be close cooperation.

In the education track of the candidate, there will be specific focus on the mathematical background of decision and prediction theory, personalization and user-profile issues, and the application of agent technology for learning and the realization of the advice function.

8) Literature

Our papers, mentioned below, are avaliable online at: As`our five most important papers, in the area of this research, we would like to mention:  [Ballegooij and Eliens (2001)],  [Eliens et al. (2002) ],  [Eliens et al. (2006)],  [Hildebrand et al. (2003) ] and  [Hoorn et al. (2004) ].
[Remediation] Bolter J.D and Grusin R. (2000),
Remediation -- Understanding New Media, MIT Press
[Query] Ballegooij A. van and Eliens A. (2001),
Navigation by Query in Virtual Worlds, Web3D 2001 Conference, Paderborn, Germany, 19-22 Feb 2001
[Prediction] Cesa-Bianchi N. and Lucosi G. (2006),
Prediction, Learning, and Games, Cambridge University Press
[DLP] Eliens A. (1992),
DLP -- A language for Distributed Logic Programming, Wiley
[OO] Eliens A. (2000),
Principles of Object-Oriented Software Development, Addison-Wesley Longman, 2nd edn.
[Platform] Eliens A., Huang Z., and Visser C. (2002),
A platform for Embodied Conversational Agents based on Distributed Logic Programming, AAMAS Workshop -- Embodied conversational agents - let's specify and evaluate them!, Bologna 17/7/2002
[Navigate] Eliens A., van Riel C., Wang Y. (2006),
Navigating media-rich information spaces using concept graphs -- the abramovic dossier, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain
[VULife] Eliens A. and Bhikharie S.V. (2006),
game @ VU -- developing a masterclass for high-school students using the Half-life 2 SDK, accepted for: GAME'ON-NA'2006, September 19-21, 2006 - Naval Postgraduate School, Monterey, USA
[Odyssee] Eliens A. (2006b),
Odyssee -- explorations in mixed reality theatre, accepted for: GAME'ON-NA'2006, September 19-21, 2006 - Naval Postgraduate School, Monterey, USA
[VirtualArt] Grau O. (2003),
Virtual Art -- From Illusion to Immersion, MIT Press
[Cinema] Hawkins B. (2005),
Real-time cinematography for games, Charles River Media
[Interactive] Hildebrand M., Eliens A., Huang Z. and Visser C. (2003),
Interactive Agents Learning their Environment, Proc. Intelligent Virtual Agents 2003, Irsee, September 15-17, 2003 J.G. Carbonell and J.Siekmann (eds.), LNAI 2792, Springer, pp. 13-17
[Empathic] Hoorn J., Eliens A., Huang Z., van Vugt H.C., Konijn E.A., Visser C.T.,
Agents with character: Evaluation of empathic agents in digital dossiers, Emphatic Agents, AAMAS 2004 New York 19 July - 23 July, 2004
[Communities] Huang Z., Eliens A., Visser C. (2002),
3D Agent-based Virtual Communities, In: Proc. Int. Web3D Symposium, Wagner W. and Beitler M.( eds), ACM Press, pp. 137-144
[STEP] Huang, Z., Eliens, A., and Visser, C.,
STEP: a Scripting Language for Embodied Agents, in: Helmut Prendinger and Mitsuru Ishizuka (eds.), Life-like Characters, Tools, Affective Functions and Applications, Springer-Verlag, 2003.
[HalfReal] Juul J. (2005),
Half Real -- Video Games between Real Rules and Fictional Worlds, MIT Press
[Guide] Riel C. van, Eliens A. and Wang Y (2006),
Exploration and guidance in media-rich information spaces: the implementation and realization of guided tours in digital dossiers, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain
[ConceptMaps] Riel C. van, Wang Y. and Eliens A. (2006b) ,
Concept map as visual interface in 3D Digital Dossiers: implementation and realization of nthe Music Dossier, CMC2006, Costa Rica, Sept 5-8 2006
[Present] Wang Y., Eliens A. and van Riel C. (2006),
Content-oriented presentation and personalized interface of cultural heritage in digital dossiers, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain

9) Requested Budget

OIO4 year172.371

As indicated, the amount of euro 177.373 includes the 5000 euro benchfee for travel expenses and other support.

(C) Æliens 04/09/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.