Research Projects

I am currently involved in the following research projects (updated January 2018):

  • Virtually Bad
  • Cultura
  • Virtual Robbery Awareness Training
  • Playing It By Ear

  • STRESS: Simulation-based Training of Resilience in Emergencies and Stressful Situations

    In crisis situations, policemen and other emergency workers often face difficult decision making problems. Even though these people usually have clear instructions about how to act, they often have difficulties in making appropriate decisions, due to a combination of factors, including time pressure and heavy emotions. The main goal of the current project is to develop an intelligent system that is able to analyse human decision making processes in stressful circumstances, and analyse the causes of incorrect decisions. In addition, the system will be incorporated in an electronic training environment for emergency workers. In this environment, trainees will be placed in a virtual emergency scenario, in which they have to make difficult decisions, while negative emotions are induced. During the scenario, various existing techniques will be applied to measure their mental state (i.e., stress level and anxiety). This information will then be used as input for the intelligent system, to determine why they made certain incorrect decisions and teach them how to improve this.

    Funded by the national NWO Brain and Cognition Programme. In collaboration with NSCR.

    Virtually Bad: A Study on Virtual Agents that Physically Threaten Human Beings

    Intelligent Virtual Agents (IVAs), interactive human-like characters, become widely used for numerous applications, varying from healthcare decision support to communication training. In such applications, IVAs play various roles in which they interact with users, for instance as an instructor or teammate. Interestingly, in the vast majority of these cases, IVAs are friendly and supportive. Instead, the area of IVAs with a 'negative' attitude towards users (i.e., 'virtual bad guys') has been heavily under-researched. However, 'virtual bad guys' are a highly interesting topic of study for at least two reasons: 1) Several prominent people recently expressed their concern that autonomous systems might evolve to a point where they threaten human beings. Controlled studies can provide a better understanding of how humans would react to such threatening AI systems. 2) The concept of virtual bad guys opens up a range of useful applications, including virtual training of aggression de-escalation skills (e.g., for security personnel), Virtual Reality exposure therapy, and anti-bullying education. However, believability, a common problem in the design of IVAs, poses a particular challenge for 'virtual bad guys'. Many effective applications require that users feel indeed seriously threatened or stressed by the IVA. However, as a general problem, IVAs are non-consequential, i.e., are unable to apply serious sanctions to users. Accordingly, users also perceive and categorize IVAs are virtual beings. These factors plausibly shape -and skew- how users respond to IVAs. So how to build virtual bad guys that are taken seriously? The present project tackles this question by designing and experimentally examining the effects of a technologically advanced IVA that is able to seriously threaten users. The study will test effects on stress and anxiety, and assess the role of believability. Ethical boundaries and implications will represent an important part of this project, too.

    Funded by the Network Institute. In collaboration with the VU Department of Communication Science.


    Cultural Awareness refers to the capacity of human beings to adapt to a different cultural environment and context. In the military domain, Cultural Awareness is increasingly being recognized as a crucial skill, for which adequate training is required. The aim of the Cultura project is to develop and validate a simulation-based Cultural Awareness training system for military staff. The system will be based on high fidelity virtual environments, and dialogue-based interaction with believable virtual characters. Skills that will be addressed include aggression de-escalation, information acquisistion, and negotation. The effectiveness of the system will be studied in an evaluation experiment with miltary students.

    Funded by the Ministry of Defence. Coordinated by IC3D Media.

    Virtual Robbery Awareness Training

    Supermarkets, jewelry stores, and gas stations are prototypical examples of enterprises that are regularly confronted with robberies. During a robbery, the behaviour of the employees that are being robbed has an important impact on the outcome of the incident. Hence, employees of such enterprises need dedicated training to prepare them for robbery incidents. To this end, the current project focuses on the development of a simulation-based Robbery Awareness training system. The system will be based on high fidelity virtual environments, and dialogue-based interaction with believable virtual characters. As the decisions made by the trainee will have a direct impact on the behaviour of the virtual robbers, the system provides run-time feedback, thereby enhancing the learning effectiveness. The system will be validated based on user studies with employees from small enterprises.

    Funded by the Ministry of Justice. In collaboration with NSCR, IC3D Media, and G4S.

    IMPACT: Impact of Cultural Aspects in the Management of Emergencies in Public Transport

    The objective of the IMPACT project is to investigate the essential role played by cultural factors in managing safety- and security-issues related to emergencies in public transport systems. It is aimed at analysing the different cultural behaviours for the prevention of emergencies, the management of emergency events and the post-events with particular emphasis on: crowd management; management of first responders to care for different cultural groups; information to passengers. From the above analyses IMPACT will produce a cultural risk assessment methodology and the associated mitigation actions for the public transport sector also developing simulators and models; identify innovative solutions that can support public transport operators in improving the communication with passengers through tailored messages to the different cultural communities (via mobile phones and social networks) and other solutions to enhance the management of emergencies considering cultural aspects; develop best practices, dedicated training material and procedures for both public transport operators and first responders; develop policy recommendations for policy makers, regulators, municipalities and public transport operators.

    Funded by the EU under the Horizon 2020 programme. Coordinated by Deep Blue.

    Playing It By Ear: Serious Gaming for Better Hearing

    To improve the hearing quality of hearing impaired individuals, regular training is important. Unfortunately, patients (especially children) are quickly bored by traditional training programs. Additionally, their social and communicative skills, including cognitive control of emotion and motivation, are often less developed. The aim of this project is to explore the potential benefit of gamification to make (computer-based) training applications for hearing impaired children more appealing, and therefore more effective. Mechanisms that will be investigated include fantasy (e.g., using role play), rewards (e.g., the possibility to obtain points or items), and challenge (e.g., controlling the difficulty level).

    Funded by the Network Institute. In collaboration with the VU Department of Language, Literature and Communication.