topical media & game development

PDF

# A theoretical foundation for the aesthetics of interaction and awareness -- making sense of the senses

Anton Eliëns, Dhaval Vyas

resource(s)

#### abstract

To clarify the notion of aesthetics in the context of interactive systems and to arrive at a foundation for interaction aesthetics, we start by looking at the history of thought from a philosophical stance. We distinguish between two complementary aspects of aesthetics, namely awareness and judgement. We identify space, time and dynamics as the dimensions constituting aesthetic awareness, and tentatively propose a model based on interactive game playing to clarify the meaning of interaction. From semiotic theory we derive a grammar of visual design that may aid our understanding of how interactive systems are perceived and what affordance they offer. Combined, these perspectives clarify the dynamics of the aesthetic experience of interactive systems, as an interplay between awareness and judgement, informed by art, architecture and gaming. Here we provide an outline of the dialectics of aesthetic awareness, that aids our understanding of the adoption and use of the new category of fun or awareness-oriented interactive systems, which we will illustrate by discussing our own work, in particular PANORAMA, a system supporting social awareness.
keywords: aesthetics, interaction, semiotics, affordance, emotive dialogs, augmented reality, digital dossier, social awareness

# Introduction

When we think of media as an extension of our senses,  [Zielinski (2006)], we may reformulate the question of interaction aesthetics as the problem of clarifying the aesthetics of media rich interactive applications.

However, what do we mean exactly by the phrase aesthetics? The Internet Enceclopedia of Philosophy discusses under the heading of aesthetics topics such as

Aesthetics

• intentions -- motives of the artist
• expression -- where form takes over
• representation -- the relation of art to reality
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('aest-ececlop'); if (!slidemode) document.write('.html');document.write('>');
These topics obviously do not cover what we want, so we took the call for contributions to the aesthetics of interaction as a good chance to dust of our old books, and rekindle our interest in this long forgotten branch of philosophy, aesthetics.

It may come as a shock to realize how many perspectives apply to the notion of aesthetics. First of all, we may take an analytical approach, as we do in section 2, to see in what ways the phrase aesthetics is used, and derive its meaning from its usage in a variety of contexts. However, we find it more worthwhile to delve into the history of thought and clarify the meaning of aesthetics from an epistemological point of view, following  [Kant (1781)], as an abstract a priori form of awareness, which is in later phenomenological thinking augmented with a notion of self-consciousness. In this line of thinking we also encounter the distinction between aesthetic awareness and aesthetic judgement, the dialectic relationship of which becomes evident in for example the occurrence of aestheticism in avant-garde art,  [Burger (1981)].

When writing this paper, we came along a report of how the Belgium curator Jan Hoet organized the Documenta IX, a famous yearly art event in Germany, and we were struck by the phrase art and the public sharing accomodation,  [Hoet (1992)], which in a way that we have yet to clarify expresses some of our intuition we have with respect to the role the new interactive systems may play in our lives.

What can we hope to achieve when taking a more philosophical look at interaction aesthetics? Not so much, at first sight. According to  [Körner (1973)], aesthetic theory ... will not be able to provide aesthetic guidance even to the extent to which moral theory can give moral guidance. The reason is that aesthetic experience and creation defy conceptualization, or in other words they defy the identification, classification and evaluation of aesthetic objects by means of non-aesthetic attributes. However, as  [Körner (1973)] observes, in a paradoxical way aesthetic experience not only defies but also invites conceptualization, and therefore it seems worthwhile to gain a better understanding in what underlies the experience and creation of (aesthetic) interactive systems.

In a mere analytical sense, our paper may be regarded as giving a clarification of some of the terminology used, and in a synthetic sense it may be regarded as another contribution to the rethorics of interaction aesthetics. However, our intention is to arrive at a deeper understanding of the notion of aesthetics that allows us to formulate a model that provides us with the concepts that we need not only to make sense of what we experience using interactive applications, but also make sense when we approach the problem of designing such systems. Afterall, what is the real question? Ultimately to design artefacts that embody the computing technology of the future.

In this paper we will take a little intellectual detour, after a brief analytical exercise in section 2, by sketching how the notion of aesthetics evolved in the history of thought in section 3, to arrive at a classification of the dimensions of awareness in section 4, followed by a discussion of the dialectics of awareness in section 5. In section 6 we propose a tentative model for interaction dynamics, based on interactive game playing, which allows for degrees of interactivity, that is a certain indirectness of interaction. In section 7 we explore a grammar of design by discussing the meaning of composition from the perspective of semiotic theory. In section 8 we discuss the various notions in the context of our own work, and in section 9 we draw our conclusions and give indications for future exploration and research.

# An analytical approach

Let us take a more close look at in what phrases the term aesthetics is used. The list of phrases below is collected from papers discussed in a workshop on the Aesthetics of Computing,  [Hallnäss et al. (2006)]. Taking a philosophical stance, a mere analytical approach shows rightout confusion and a blurred understanding of notions central to any theory of aesthetics. Here are some samples:

usage of aesthetics

1. design aesthetics is what relates to notions of form and expression in design practice.
2. design practice where aesthetic qualities are emphasied.
3. the aesthetics of an interactive artefact evolves in the relationship with the user.
4. experimental design aesthetics differ very little from art.
5. when pro-active technology goes home, pragmatic aesthetics is needed.
6. there is a need for a specific basic interaction design course and knowledge about the aesthetics of interactions.
7. recent trends call for a stronger focus on the aesthetics of user experiences.
8. this intention is aesthetics, and aesthetics for its own sake, and even goes beyond an interest of meaning.
9. to use software is to perceive, to grasp and to apply. We are right in the middle of aesthetics!
10. by turning to aesthetics the critical approach to computing thus includes an emancipatory aspect.
11. very few of these actually hold any artistic or aesthetic quality, which is not surprising at all with such a new media.
12. aesthetics have their root in philosophy, which defines aesthetics as the (perceived) sense of beauty.
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('aest-ana'); if (!slidemode) document.write('.html');document.write('>');
There obviously is a bewildering variety of ways in which the term aesthetics is used. When we make a first selection we may reduce the list above to phrases such as aesthetic qualities of objects, aesthetics of design process, aesthetics of user experiences, aesthetics as critical judgemente, and aesthetics as sense of beauty

Still, there is confusion, for example is sense in sense of beauty meant as appearance or as faculty of perception or both? So, finally, in our most drastic reduction we arrive at the following forms of usage:

usage of aesthetics (3)

• aesthetics of experience -- for subject (audience)
• aesthetics of appearance -- for object/artefact
• aesthetics of use -- in interaction
• aesthetics of design -- in creative process
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('aesth-mea-2'); if (!slidemode) document.write('.html');document.write('>');
Let us be clear, although we definitely do cherish the insights expressed in the phrases and sentences above, we refuse to accept the terminology used without further ado, and propose instead to look for definitions and explanations in some of the classics of philosophy.

# The notion of aesthetics in the history of thought

Philosophy is not a very popular subject, and some seem to easily do away with philosophical abstractions and apparently tedious theory, even though these same philosophical abstractions may provide better understanding of the forces that are at work.

In this section, we will briefly trace the evolution of the notion of aesthetics to our current day understanding, starting with the idealist transcendental conception of aesthetics as the abstract a priori form of experience, ending with semiotic theory that emphasizes the social determinants of aesthetic experience. Our discussion, in this section, is based on our studies in the past,  [Eliens (1979)], and the outline given below includes the references to the material we originally studied. However, for reference, links to relevant online material are also included.

perspective(s)

1. transcendental -- abstract form of experience,  [Kant (1781)]
2. speculative -- criteria for beauty,  [Kant (1781)]
3. phenomenological -- self-conscious subjectivity,  [Hegel (1807)]
4. psychoanalytical -- sub-conscious meaning,  [Freud (1958)]
5. pragmatical -- art as experience,  [Dewey (1931)]
6. hermeneutical -- understanding of the senses,  [Gadamer (1977)]
7. semiotics -- social construction of meaning,  [Kress and van Leeuwen (1996)]
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('aest-persp'); if (!slidemode) document.write('.html');document.write('>');
To our mind, the epistemological understanding of aesthetics as the pure form of sensuousness, as expressed in  [Kant (1781)], is most fundamental in understanding the notion of aesthetics in the context of interactive systems, since it allows us to characterize the dimensions of sensuous awareness delimiting our experience of art, architecture and interactive systems. The epistemological or transcendental characterization of aesthetics describes, in other words, the a priori principles of sensuousness, that determine our perception of reality, by imposing organisation and form on the chaotic multitude of appearances. As phrased in  [Kant (1781)], appearances consist of material, which is a posteriori given, and form, determined by the a priori nature of our mind.

As dimensions of pure sensuousness, or aesthetic awareness, Kant distinguishes between space and time. In  [Kant (1781)], the notion of aesthetic judgement is introduced. Our ability for aesthetic awareness allows us to recognize and appreciate beauty, however Kant emphasizes that any attempt to conceptualize the judgement of beauty is doomed to fail, or may at best be determined empirically, in an ad hoc manner.

Later thinkers in the idealist school took over Kants conception of aesthetic awareness as the receptive side of our mind, in search for knowledge, and emphasized the relation between truth and beauty,  [Schiller (1977)]. In particular  [Hegel (1807)] characterized beauty as the sensuous presence of Idea, and he identifies our need for truth and beauty with the intrinsic movement of self-consciousness. In other words, aesthetic awareness in not a dis-interested a priori ability that allows us to organise our perceptions and to recognize and appreciate pure form, rather it is intentional and through self-reflection subject to recurrent improvement and change, continuously looking for truth and beauty, that is meaning. We may note here that psychoanalytic theory has contributed to understanding the hidden dimensions of meaning,  [Freud (1958)].

Hegels conception of aesthetic awareness is surprisingly close to the idea of pragmatic aesthetics as expressed by  [Dewey (1931)], a representative of the anglo-saxon school of empiricist philosophy which is in many ways irreconcilable with the German idealist/phenomenologist school of thinking. Essential in Dewey's thinking is the notion of qualitative immediacy and the unification of awareness and judgement in the experience of art, where Dewey stresses the re-creating role of the subject/recipient in experiencing art. In this way, the experience of art is instrumental, according to Dewey, to reconcile the individual with his environment.

A similar concern with the existential role of the experience of art, and consequently aesthetic awareness, may be found in hermeneutic thinking of the 20th century, where for example  [Gadamer (1977)] speaks of beauty bridging the gap between the ideal and reality. However, by that time art is no longer pure but must as aesthetic art be appreciated with a certain degree of distance, that is its judgement is no longer direct, governed by pure sensuousness, but regulated by reflection and to a certain extent disciplined appreciation. This position may, however, be attributed to the role of the arts in the 19th and 20th century, and even, as argumented by  [Grau (2003)], be seen as an opposition to the mass media of the 19th century, which strived for direct sensuous immersion, for example in life-like panoramas.

The influence of convention and social context has been studied in semiotic theory,  [Kress and van Leeuwen (1996)], and in our time, where we are concerned with the influence of the old and new media, and media literacy is (again) one of the urgent topics on our political agenda, the relation between sensuousness and reflection is again of interest. We believe that the semiotic perspective is of particular importance for the design of interactive systems. Nevertheless, to summarize this section, for our epistemological understanding of aesthetics the original notion of sensuousness as the receptive side of our faculty of knowledge still seems to provide a good starting point. However, both an analytic view of aesthetic awareness, which for example forces us to think about the difference between aesthetic experience and a drug-induced state of mind,  [Saw (1971)], and a recognition of the moral dimension of beauty,  [Cheng (2006)], may serve us in establishing the value of aesthetics for the design and appreciation of interactive systems.

# Dimensions of aesthetic awareness

In  [Hallnäss and Redström (2002)] it is observed that the aesthetic potential of the narrative space centered on the consumer product has received surprisingly little attention. The authors then argue that, motivated by insights from phenomenology, there should be a shift of attention from use to presence, where presence does not merely mean appearance but a more complex dialectic process of appearance and gradual disappearance dependent on the role the object plays in the life of the user/subject. The notion of expressional is then introduced, to convey the expressive meaning of objects, and in particular interactive objects, in our surroundings. For the design of presence, aesthetics is then considered as a logic of expressions, in which expressions act as the presentation of a structure in a given space of design variables. However appealing the notion of expressional, in the light of our discussion in section 3, where we distinguish between aesthetic awareness as a given, or a priori, sensibility and aesthetic judgement as being of a more empirical nature, we would prefer to consider aesthetics as a logic of sensibility, which includes a dimension of self-reflection in the sense of its being aware of its own history. Put differently, to characterize the contextual aspect of aesthetics, as it certainly applies to art, we may speak of aesthetic literacy, that is aesthetic awareness that is self-reflective by nature.

Assuming a notion of aesthetics as a logic of sensibility, we may distinguish between three dimensions of form, extending Kant's original proposal, as indicated below:

dimensions of aesthetic awareness

• spatial -- topological relations, layout of image
• temporal -- order, rhythm, structure
• dynamic -- interaction, reflection, involvement
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('aest-aware'); if (!slidemode) document.write('.html');document.write('>');
The dimension of dynamics clearly is the great unknown, and more in particular it is the dimension we have to explore in the context of interactive systems, not in isolation but in relation to the other dimensions, not so much to establish definite criteria, but to understand the forces at work, or in other words the relevant parameters of design.  [Sartre (1936)] gives an existential foundation for the dimension of dynamics, by observing that the human body is instrumental in gaining awareness, as the centre of both obscurity and reflection from which consciousness emerges, through selection and action.

It is in the existential dimension of aesthetic awareness that we come most close to the experience of the new digital artefacts, since it concerns both involvement and human action. Interestingly, and in apparent contradiction with  [Hallnäss and Redström (2002)], cited previously, to establish a foundation for the aesthetics of interactive systems  [Graves-Petersen et al. (2004)] seek refuge with pragmatist aesthetics as it promotes aesthetics of use rather than aesthetics of appearance. Again, although we agree with the gist of  [Graves-Petersen et al. (2004)], we wish to emphasize that the contribution of pragmatist aesthetics is not its focus on use, but the role of experience in understanding and appreciating aesthetic artefacts, that is the active role of the subject in becoming aware of its meaning.

For objects that are not designed for usability in the functional sense the notion of use is too strict and is, using a dialectic argument, subject to the dialectics of presence, as argued in  [Hallnäss and Redström (2002)]. Conversely, using a similar dialectic argument, for new categories of objects, presence requires use, or getting used to, in other words a process in which the user becomes interested and familiar with the object. We may even speak of aesthetic affordance, with the realization that the notion of affordance, explaining an ecology of behavior, originally stems from the late-idealist phenomenology expounded in  [Heidegger (1927)].

For the design and appreciation of the new category of digital systems we may, looking back at our discussion of the history of thought, well take pragmatist aesthetics as a common ground, since it does justice to the existential dimension of aesthetic awareness, and allows for a process of aesthetic literacy, that is becoming sensible to aesthetic awareness and reflection.

# The dialectics of awareness

In the course of our field study for the PANORAMA system,  [Vyas et al. (2007) ], that we will discuss in more detail in section 8, we tried to establish what relation users would have to the system, not only in the way they interact with it, but also in terms of what role the system plays in their lives, and when and how they would be aware of the system.

Due to the intrinsic properties of the PANORAMA system, which is a system meant to support social awareness in a work environment, we could not assume the direct focussed attention that characterized the applications we had in mind when formulating the design framework in  [Vyas and van der Veer (2006)], discussed in section 8. Instead, we must adapt the model to take the various forms of awareness or attention into account.

Our thoughts in this direction were triggered by a lecture of Linda Stone (former vice-president of Microsoft) at the Crossmedia Week September 2006 in Amsterdam, entitled Attention -- the Real Aphrodisiac. In that lecture Linda Stone made a distinction between applications popular before 1985, applications which were in general meant for self-improvement, for example language-learning, applications that were popular between 1985 and 2005, applications that she characterized as supporting continuous partial awareness, such as email and news-feeds, and applications of the period thereafter, from now into the future, which may be characterized as applications that allow the user to be creative, take part in a community, and are in other words more focussed and less dependent on the external environment.

Admittedly, it takes a few more steps to formulate a theory of the dialectics of awareness. However, with the function of the PANORAMA system in mind, we may make, following  [Benjamin (1936)], some interesting distinctions between the experience of art and architecture. Where art is usually experienced in a delimited time span, and is similarly delimited in space, that is the position of the observer, architecture is everywhere and always there. As a consequence, art receives focussed attention and may be appreciated with reflective distance, whereas architecture is often not perceived consciously, but merely present and subject to an almost sub-conscious sensibility, which is only brought to the focus of attention when it is either aesthetisized, for example when taking photographs, or when something surprising is sensed, for example in the change of skyline in New York.

As argued in  [Hallnäss and Redström (2002)], many of the new interactive systems, whether in the category of ambient media, ubiquitous computing or calm technology, will fall somewhere inbetween the spectrum spanned by art and architecture, or more likely even alternate between the forms of awareness associated with respectively art and architecture.

In designing the new interactive systems, we need to be explicitly concerned with the actual phases of awareness that occur, simply because it is not clear what role these systems play in our life. When introducing a new system or artefact, we may distinguish between the following phases:

phases of awareness

• initiation -- appeal to curiosity
• promotion -- raising interest
• progression -- prolonged involvement
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('aesth-aware'); if (!slidemode) document.write('.html');document.write('>');
As designers we must ask ourselves the following questions. How do we appeal to the users' curiosity, so that our system is noticed? How do we get a more sustained interest? How de we get the user to interact with or contribute to the system? And, how do we obtain prolonged involvement, and avoid boredom? These questions are not simple to answer, and require also an understanding of the actual context in which the system is deployed as well as an understanding of the level of (aesthetic) literacy of the user(s).

So far we have abstained from discussing aesthetic judgement, in accordance with our (philosophical) observation that aesthetic quality can not be codified in definite concepts. Moreover, we even distrust aesthetic judgement, since often judgements are based on the wrong 'metrics', as for example in art the place where a work is exhibited. Awareness we see as more important, also as a parameter of design. Nevertheless, when we consider the context in which the experience of the new digital artefacts takes place, we may assume a dialectic relation between awareness and judgement, similar as in the experience of art, where gradually our sensibility is becoming more informed, where in other words our awareess becomes educated and our judgement(s) more appropriate and refined. And, similar as in our experience of art, we gradually become involved with the conventions and style issues, in other words the rules of the game.

# A model of interaction dynamics

Although nowadays art may also be recognized to offer interaction in various degrees, for a model of interaction we prefer, following a suggestion in  [Grau (2003)], to look for inspiration at game playing, and in particular we propose the game model introduced in  [Juul (2005)] as a first explanation of the dynamics of interaction. Later, in section 7, we will further explore the differences between games, art, architecture and interactive applications, and discuss possible refinements to the model.

Following  [Juul (2005)], we may characterize a game as a system, or a formal set of rules. In addition, we can identify a relation between the player and a game, a relation that can be of a rather affectionate or involved nature, and we may consider the context of playing, which is in a broad sense a negotiable relation with the real world, which may go as far as becoming rich or famous in the real world.

In a more formal way, still following  [Juul (2005)], we can define a classic game model by considering the following aspects or elements:

classic game model

• rules -- formal system
• outcome -- variable and quantifiable
• value -- different valorisation assignments
• effort -- in order to influence the outcome
• attachment -- emotionally attached to outcome
• consequences -- optional and negotiable
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('gm-model'); if (!slidemode) document.write('.html');document.write('>');
Where an arbitrary interactive system may differ from a game played for entertainment is obviously the actual outcome, the value attributed to that in the real world, and probably the effort required and the possible consequences. You would not like to run the risk to die a virtual death when answering your email, would you? However, when interactive systems replace task-bound functionality with fun, the difference becomes less clear.

One element not sufficiently captured by the classic game model is the narrative aspect of the game play. To quote  [Juul (2005)]:

rules vs fiction

Game fiction is ambiguous, optional and imagined by the player in uncontrollable and unpredictable ways, but the emphasis on fictional worlds may be the strongest innovation of the video game.

##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('gm-etc'); if (!slidemode) document.write('.html');document.write('>');
We may observe that many games already have a strong relation to reality in what narrative context they supply, or else in the realities of the media industry, in particular Hollywood. For serious interactive systems, we may assume an even stronger and in some sense more straightforward relation with reality, by the use of media content that is relevant for the life of the individual.

All these aspects of playing games are clearly relevant for the new interactive systems, which appeal more to play than task-oriented behavior. For example rules may be used to describe the visual characteristics of a system (e.g. the display of images as a flow in a particle system, as we will describe in section 8), outcome may be regarded as the benefits of the system (e.g. social awareness), value may include the risks of the system (e.g. a transgression of privacy), efforts is important when asking for contributions from the user (e.g. as image material to be displayed in the system), attachment may result when the system is installed (e.g. when people look forward to find new information), and finally consequences must be considered when a system is installed and used (e.g. interaction between people may actually change when they get to know eachother, for better or worse).

#### interaction markers

Given the large variety of games, including first person shooters, role-playing games, strategy games and decision-making simulation games, we can distinguish between a range of degrees of interaction, direct interaction, on the one hand, as for example in first person shooter and indirect interaction, on the other hand, as for example in simulation games, or role-playing games where the individual actions may contribute to a plot such that the effects will become visible at a later time. Where in game playing the variety of interaction modes seems to be well understood within each community of game players, for the development of more general interactive systems we will have to think seriously whether the target user will be able to learn the various modes of interaction, either by explicit instruction or during play. And as designers we must be concerned with the rules of interaction as well as issues of visualisation and interaction mappings, that is in other words which affordances the application offers for a particular group of users.

To tackle this design problem, we introduce the notion of interaction markers, that in a similar way to veracity markers indicating believability in communication, cover the potential for interaction with a vocabulary of visual and other sensory cues, which may partly be derived from interaction conventions common in game communities.

# The meaning of composition

Having an understanding of the dimensions of aesthetic awareness, can we isolate the relevant design parameters and formulate rules of composition that may help us in developing interactive applications? According to our philosophical credo, no! However, the history of art clearly shows the impact of discoveries, such as the discovery of perspective, as well as conventions in the interpretation of art, as for example in the iconic representation of narrative context in 17th century Dutch painting. Moreover, the analysis of the visual culture of mass media may also give us better understanding of the implied meaning of compositional structures.

The notion of perspective, described in  [Alberti (1435)], is an interesting notion in itself, since it describes both the organisation of the image as well as the optimal point of view of the viewer. The normal perspective as we know it is the central perspective. However, there are variants of perspective that force the viewer in an abnormal point of view, as for example with anamorphisms.

Perspective had an enormous impact on (western) art and visual culture. It defines our notion of naturalist realism, and allowed for the dvelopment of the panorama as a mass medium of the 19th century,  [Grau (2003)]. Art that deviated from central perspective, such as cubism or art from other cultures, was often considered naive. Photography and its pre-cursors had a great impact on the perfection of perspectivist naturalism, and what is called photorealism became the touchstone of perfection for early computer graphics,  [Bolter and Grusin (2000)].

Apart from perspective, other conventions regulate the composition of the 2D image, in particular, following  [Kress and van Leeuwen (1996)], the information value related to where an object is placed in the image, and the salience of the object, determined by its relative size, being foreground or background, and visual contrast. Also framing is used to emphasize meaning, as for example in the close-up in a movie shot. In analysing a large collection of image material,  [Kress and van Leeuwen (1996)], somewhat surprisingly found that lef/right positioning usually meant given versus new, top/bottom positioning ideal versus real, and centre/margin positioning important versus marginal. It is doubtful whether these meaning relationships hold in all cultures, but as a visual convention it is apparently well-rooted in western visual culture.

For 2D images,  [Kress and van Leeuwen (1996)] further identify narrative elements, that is relations between objects in the image that suggest a story, such as a diagonal line from a person to a door, or a relation of an object to the viewer, such as a gaze towards the viewer, a technique that has been used only since late renaissance painting.

More than paintings or 2D images, film is the medium for conveying narrative structures. The art of storytelling in film has been perfected in such a way that Hollywood films may seem more real than life. However, as emphasized in  [Bolter and Grusin (2000)], this is not due to any inherent form of naturalism, but to the fact that we have got accustomed to the conventions applied, that is the techniques of cutting, montage, camera movements, close-ups, etcetera. In a highly recommended book,  [Arnheim (1957)], Rudolf Arnheim gives an extensive analysis of the principles of montage and film technique, and he explains why film is such an effective medium:

frames of reference

It is one of the most important formal qualities of film that every object that is reproduced appears simultaneously in two entirely different frames of reference, namely the two-dimensional and the three-dimensional, and that as one identical object it fulfills two different functions in the two contexts.

##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('12-2-23D'); if (!slidemode) document.write('.html');document.write('>');
Due to the subtle play between these two frames of reference film may be considered an art form, and as such perhaps the dominant art form of the 20th century. As a mass medium, film may be characterized by what Arnheim, following Benjamin, called the aesthetics of shock, replacing reflective distance with immersive thrill. As an art form, however, it is the dominant paradigm for aesthetic awareness, lacking however still one dimension, interactive dynamics.

As observed in  [Bolter and Grusin (2000)], interaction is what distinguishes video games from film. Current day technology allows for high-resolution photorealist graphics, that make video games or virtual applications almost indistinguishable from film. Virtual reality technology as applied in video games adds arbitrary choice of perspective, as exemplified in first-person shooters or fly-overs, as well as an arbitrary mix of the imaginary and real, as in CG movies, in an interactive fashion.

Now, should we take the aesthetics of interactive video games as the standard for interactive applications? Not necessarily, since the naturalism strived for in most games may at best be characterized as naive realism, mostly photorealism. As observed in  [Kress and van Leeuwen (1996)], realism is a social construct, and hence the program for developing an aesthetics for interactive applications should perhaps include the development of appropriate realisms. Again with an eye to the history of art, where we have for example impressionism, cubism, expressionism, as a guideline in the design of interactive systems, it might be even better to look for appropriate interaction-isms, styles of developing interactive systems from a particular perspective.

# Applications to design

Much of our own work may be best characterized as explorative development using virtual reality and game technology for (more or less) serious applications. In  [Eliens et al. (2003)], we describe a framework for mixed media, based on agent technology,  [Eliens et al. (2002)], supporting emotive dialogs by humanoid agents, in rich media virtual environments.

In  [Eliens et al. (2006)], we describe an immersive digital dossier that allows navigating the concept space surrounding the work of the artist/performer Marina Abramovic, and to look at her work, including video recordings of performances and 3D models of art work installations. Immersive, in this context, means that there is no disruptive break between navigation and presentation, for example by pop-up windows.

In  [Eliens (2006)], we explored the combination of live video with 3D technology to achieve visual effects in a transparent manner, that we called see-through aesthetics, giving emotive distortions and images affecting the interpretation of the perceived scene.

In  [Eliens and Bhikharie (2006)], we created a realistic multi-user game, featuring our facul;ty building, offering a simple puzzle for the player to obtain a hidden treasure, in effect the power to use weapons.

Although each of these projects had their implied aesthetics, that is a particular perspective on reality, as reflected in the system, our approach to design in these cases has been rather intuitive. In our explorative efforts, we were primarily led by what Brancusi called the rethorics of the material.

#### PANORAMA

In developing the PANORAMA system we could, however, not follow this naive approach, since it is rather more complex in what we hope to achieve with it,  [Vyas et al. (2007) ]. The PANORAMA system is meant to support social awareness, in non-work related ways, using a large screen display in a public room in our faculty. To achieve social awareness, we ask the staff to contribute items of self-reflection, such as holiday postcards or birth announcements. In order to reflect the liveliness of the workplace, we monitor places where occasional encounters may take place, for example during a break at the coffee machine or in the printer room, waiting for the printer queue. Encounters in such places are often of an informal, personal nature, but may be mixed with work-related interests. As an experimental feature, we consider to allow for direct interaction using the system, for example, to play a game, possibly with a mobile phone as an input device. In summary, the PANORAMA system is determined by the following contributions of its users, contributions that are not necessarily direct or even do require explicit activity.

PANORAMA

• self-reflection(s) -- e.g. picture/postcard(s)
• casual encounter(s) -- at coffemachine or printer
• occasional battle(s) -- optional direct interaction
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('p-aest-pano'); if (!slidemode) document.write('.html');document.write('>');
For a deeper understanding of what role the system would play in the (working) life of the staff, we engaged in several field studies (in progress) and used cultural probes to determine what could be valuable contributions to ask for and how to display these on the PANORAMA screen.

In this stage, PANORAMA is still in a design phase. We have developed, however, a first prototype implementation using ViP technology, based on the system described in  [Eliens (2006)]. In this realization, we deploy a moving virtual gallery, containing video and image feeds. The gallery acts like a moving scroll, displaying information in a continuous manner, in a panorama-like way. The images in the gallery are fed by channels, containing information that is either due to explicit contributions (self-reflections) or ongoing activity in the work place (casual encounters or occuring events), monitored by cameras or other sensors.

Obviously, the PANORAMA system itself will be subject to a dialectic of awareness, that is it will be present, but the staff will only occasionally pay atention to it, dependent on their interests and also on what visual cues and effects the system presents to draw attention to ongoing activity. Although we would like the system to be autonomous in the decision how to present information, we cannot hope to do this by computational means only,  [Eliens (1988)], and hence we need to provide interaction markers to invite the users to contribute actively to the system, or influence the way information is displayed according to their preference.

For the display of information, we provide a rich context of material, including videos showing the faculty and its surroundings, fragments from video clips, and of course the material resulting from the occassional encounters and self reflections. In PANORAMA we use particle systems displaying the information in a pictorial way by images flowing according to the rules of the particle system chosen to represent that particular type of information. To organize this material we took the observations about the conventions governing our interpretation of 2D displays as a guideline in designing the flow of particle systems. Identifying bottom with plain, top with ideal, left with given and right with new, we arrived at the following identifications. bottom $->$ top/right

semiotic rules

• self reflections: plain $==>$ ideal/new
• casual encounters: plain $==>$ ideal/given
• contextual stories: ideal/given $==>$ plain/new
• personell faces: ideal/new $==>$ plain/given
• occurring events: ideal $==>$ plain
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('sl-pano-semi'); if (!slidemode) document.write('.html');document.write('>');
For example one may remark that people's faces become more familar in time, and that in the process of getting to know them we see more of the plain reality of people. Naturally, different interpretations and different designs are possible.

Apart from the spatial characteristics of these flows of information we also used the speed with which the images move accross the screen as a paramter of design. For eample events and occurrences move very fast, while both casual encounters and self reflections move slowly. Faces come across the screen with intermediate speed. To give self reflections more visual salience, the images are displayed in a non-transparent way, whereas all other flows of images merge with the background due to transparency. Although it is debatable whether the interpretations given above hold, we found the heuristics given by semiotic theory extremely helpful in deciding how to represent the information as flows of images in space/time.

#### experience as meaning

In  [Vyas and van der Veer (2006)], we introduced a framework for the design of interactive systems, based on the observation that the user's experience of an artefact or application may be considered to consist of the meaning s/he constructs. These observations concerned both the appreciation of musical scores and movies, and actual interactive applications. Our framework is captured in the following postulates, which entail that an interactive system is determined by function, interaction and appearance:

framework

• experience occurs during the interaction between the user(s) and the interactive system(s) in the lived environment
• designers convey meaning (consciously or unconsciously) through the appearance, interaction and function of the system
• user(s) construct a coherent whole that is a combination of sensual, cognitive, emotional and practical forms of experience
##### document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('paper-aesthetics.html#slide-'); } document.write('r-12-1-framew'); if (!slidemode) document.write('.html');document.write('>');
To validate the framework we carried out several design experiments, in which we instructed the designers to enact the envisaged experience of the potential user. In later work we deployed cultural probes to gain deeper understanding of the world of experience of potential users and their expectations with regard to the system.

In a critical review of our work, from the perspective of formulating a foundation for the aesthetics of interaction, our framework may be considered lacking in not considering the dialectics of awareness, as discussed in section 5. In particular, we must augment the framework by giving explicit attention to the phases of initiation, promotion and progression, to account for the aspects determining the process in which the user gets familiar with the design artefacts. A fortiori, this holds for our design efforts for the PANORAMA system.

# Conclusions

In this paper, we have sketched a foundation for the aesthetics of interaction by, in summary, making a distinction between aesthetic awareness and aesthetic judgement. From an epistemological perspective, we characterized aesthetic awareness as the pure sensuousness of mind, imposing form on our experience and perceptions, motivated by a desire for meaning and understanding. As dimensions of aesthetic awareness we distinguished between space, time and dynamics, and we provided a tentative model for interaction dynamics based on a model of interactive game playing. Looking at semiotic theory, we explored whether we could identify any rules of composition, that could guide us in the process of design. Finally, we briefly discussed our own work in the area of media-rich interactive systems.

Our paper is based on a relatively extensive study of philosophical literature, including works of modern analytical philosophy. In discussing our work and speculating about possible guidelines for design, we may well have fallen victim to our own analytical sword. However, given our interest in the the forces at work in media and aesthetic literacy, we feel justified not only in presenting our philosophical findings, but also in our transgression of philosophical rigor, to clarify how we make sense of our senses, and how to apply this understanding to the design of interactive systems.

As our main contribution, we see that we have clarified the inherent dialectics of awareness that governs the process in which we gain (aesthetic) literacy with the new category of systems to which the aesthetics of interaction apply.

# References

[Perspective] Alberti L.B. (1435),
On painting and on sculpture, Phaidon, edited by C. Grayson, 1972
[Film] Arnheim R. (1957),
Film as Art, University of California Press
[Reproduction] Benjamin W. (1936),
The Work of Art in the Age of Mechanical Reproduction, Online Archive -- http://www.marxists.org
[Remediation] Bolter J.D and Grusin R. (2000),
Remediation -- Understanding New Media, MIT Press
[Avantgarde] Burger P. (1981),
Theorie de Avantgarde, Edition Suhrkamp
[Meditations] Cheng F. (2006),
Cinq meditations sur la beaute, Editions Albin Michel
[Pragmatics] Dewey J. (1931),
Art as Experience, Ref ???
[Creativity] Eliens A. (1979),
Creativity: reflection and involvement, Ninth Int Conf of Aesthetics, Dubrovnic, August 1979
[Computational] Eliens A. (1988),
Computational Art, First Int Symposium on Electronic Art, Leonardo Supplementary Issue, Pergamon Press 1988, pp. 21-26
[Platform] Eliens A., Huang Z., and Visser C. (2002),
A platform for Embodied Conversational Agents based on Distributed Logic Programming, AAMAS Workshop -- Embodied conversational agents - let\'s specify and evaluate them!, Bologna 17/7/2002
[Media] Eliens A., Dormann C., Huang Z., Visser C. (2003),
A framework for mixed media -- emotive dialogs, rich media and virtual environments, Proc. TIDSE03, 1st Int. Conf. on Technologies for Interactive Digital Storytelling and Entertainment, Göobel S. Braun N.,n Spierling U., Dechau J. and Diener H. (eds>), Fraunhofer IRB Verlag, Darmstadt Germany, March 24-26, 2003
[Topical] Eliens A. (2005),
Topical media & game development, Syllabus Introduction Multimedia, Vrije Universiteit, Amsterdam, http://www.cs.vu.nl/~eliens/media
[VULife] Eliens A. and S.V. Bhikharie (2006),
Game @ VU -- developing a masterclass for high-school students using the Half-life 2 SDK, In Proc. GAME'ON-NA 2006, P. McDowell ed., Eurosis-ETI, pp. 49-53, Sept 19-21, Monterey, USA
[Odyssee] Eliens A. (2006),
Odyssee -- explorations in mixed reality theatre, In Proc. GAME'ON-NA 2006, P. McDowell ed., Eurosis-ETI, pp. 62-64, Sept 19-21, Monterey, USA
[Dossier] Eliens A., van Riel C., Wang Y. (2006),
Navigating media-rich information spaces using concept graphs -- the abramovic dossier, Proc. InSciT2006, 25-28 Oct. 2006, Merida, Spain
[Witz] Freud S. (1958),
Der Witz und seine Beziehung zum Unbewussten, Fischer Verlag
Die Aktualität des Schönen, Reclam, Stuttgart
[VirtualArt] Grau O. (2003),
Virtual Art -- From Illusion to Immersion, MIT Press
[Interaction] Graves-Petersen M., Iversen O.S., Gall Krogh P.,
Ludvigsen M. (2004), Aesthetic Interaction -- A Pragmatist's Aesthetics of Interactive Systems, Proc. DIS2004, Cambridge Mass. USA, pp. 269-276
[Presence] Hallnäss L. and Redström J. (2002),
From Use to Presence: On the Expressions and Aesthetics of Everyday Computational Things, ACM Trans. on Computer-Human Interactions, Vol. 9, No 2, pp. 106-124
[Workshop] Hallnäss L., Graves Petersen M., Gall Kroch P.,
and Pold S. (2006), Aesthetics as Critical Computing -- A Workshop on the Role of Aesthetics within the field of Information Technology, The Fourth Decennial Aarhus Conference Aarhus, Denmark 20-24 August 2005
[Phenomenology] Hegel G.W.F. (1807),
Phänomenologie des Geistes, Felix Meiner Verlag, ed. 1952
[Sein] Heidegger M. (1927),
Sein und Zeit, Niemeyer Tubingen, 1993
[Documenta] Hoet J. (1992),
Jan Hoet -- On the way to Documenta IX, Edition Cantz
[HalfReal] Juul J. (2005),
Half Real -- Video Games between Real Rules and Fictional Worlds, MIT Press
[Kritik] Kant E. (1781),
Kritik der reinen Vernunft, Felix Meiner Verlag, ed. 1976
[Urteil] Kant E. (1781),
Kritik der Urteilskraft, Reclam, Stuttgart, 1976
[Fundamental] Körner S. (1973),
Fundamental Questions of Philosophy, Penguin Books
[Semiotics] Kress G. and van Leeuwen T. (1996),
Reading Images: The Grammar of Visual Design, Routledge
[Imagination] Sartre J.P. (1936),
L'imagination, Presse Universitaire de France, ed. 8, 1971
[Aesthetics] Saw R.L. (1971),
Aesthetics -- An Introduction, Anchor Books, New York
[Erziehung] Schiller F. (1977),
Uber die ästhetische erziehung des Menschen, Reclam, Stuttgart
[Meaning] Vyas, D., van der Veer, G. C. (2006),
Experience as Meaning: Some Underlying Concepts and Implications for Design, Proc. of 13th European Conf. on Cognitive Ergonomics. ACM Press: NY, (2006), 81-91.
[Affordance] Vyas, D., Chisalita, C., van der Veer, G. C. (2006),
Affordance in Interaction, Proc. of 13th European Conference on Cognitive Ergonomics. ACM Press: NY, (2006), 92-99.
[Panorama] Vyas D., van de Watering M., Eliens A.,
van der Veer G. (2007), Engineering Social Awareness in Work Environments, accepted for HCI International 2007, 22-27 July, Beijing, China
[DeepTime] Zielinski S. (2006),
Deep Time of the Media -- Towards an archaeology of Hearing and Seeing by Technical Means, MIT Press

(C) Æliens 27/08/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.