Kinecting with Orang-utans

Project overview

Digital technology presents significant opportunities to support Zoos in their goals of animal welfare and enrichment. Researchers from the Microsoft Research Centre for Social Natural User Interfaces (SocialNUI) at The University of Melbourne are excited to be collaborating with Zoos Victoria to explore how natural user interface technologies (NUIs) can improve animal welfare.

Orang-utans are highly intelligent species critically endangered due to deforestation of their natural habitat for palm oil plantations. ¬†Through our collaborative project we are studying how digital enrichment can improve the welfare of Melbourne Zoo’s 6 orang-utans and building a better understanding of the way orang-utans learn and interact with technology. By studying the design and use of digital enrichment we are seeking to better understand the zoo as a context for human-computer interaction (HCI) research, and develop methodologies for designing for non-human users.

The research team are applying ‘user-centered design’ methods to approach the development of enrichment for orang-utans. This involves observing orangutans and keepers, interviews and workshops with animal welfare experts, keepers and other personnel, to identify orang-utans’ motivations and interests and define the opportunities for NUIs in this space.

Early on in this process, the SocialNUI researchers studied how Melbourne Zoo was utilising technology with the orang-utans. As well as projecting movies into the enclosure, the keepers have explored the use of tablet computers to provide enrichment in the form of puzzles, games, musical instruments and painting. While this shows the potential for digital enrichment, it is limited as the orang-utans (who would readily break fragile tablet computers) can only interact with their fingertips through metal bars while a Zoo Keeper holds the tablet outside the enclosure. Consequently, this project utilises NUI technologies to give the orang-utans a wider range of ways to interact with technology by offering them more active, full body movement.

Orang-utan lying on its back with red light shining on its face.

The proof of concept system was tested in February 2016 at Melbourne Zoo. This solution uses a Microsoft Kinect and a projector placed outside the enclosure, which avoids the need to give the orang-utans physical devices that could be broken. The system projects moving images on to the floor of the orang-utan enclosure. Using custom software, the Microsoft Kinect sensor tracks the orangutans’ movements and detects their touches and interactions with this area of the floor, so that the projection becomes a large touch-enabled surface.

The orang-utans were given complete choice over whether to interact with the system or not, and all trials were overseen by the Zoo’s animal welfare expert.

orang-utan leaning on her knuckles and looking down

The team developed a variety of games and simple applications for this system. The applications aim to explore how orangutans prefer to interact with the system, and demonstrate opportunities for enrichment. The games are designed to encourage orang-utans to play, explore and learn about what they can do with the system, building on their natural problem-solving abilities.

The games were developed in close collaboration with the zoo keepers, drawing on their knowledge of the orang-utans different personalities, likes and dislikes. For example, researchers discovered that Kiani (Suma), Melbourne Zoo’s 37 year old female orang-utan, loves looking at photos of herself. The team developed an application that showed a gallery of photographs that the orang-utans could browse, select and interact with. The orangutans have experience of painting on canvas and seem to find this enriching, so a painting application was also developed.

Orang-utan sitting in straw and concentrating on a checkboard of coloured lights on the ground.

Orang-utans’ willingness and ability to use this technology without training or encouragement indicates that it can provide a foundation for diverse cognitive challenges which respond to their individual needs and interests, to keep their minds and bodies active.

Orang-utan and keeper collaborating on game

As part of the trial the orang-utans were offered the opportunity to interact with humans through the technology. As orang-utans seem to find humans interesting, the team created a shared digital space where orang-utans could choose to interact safely with keepers and even visitors. In the first trial of a game for humans and orang-utans, the team observed with delight that they chose to play even with people they had not met before, creating a powerful sense of connection for the human player.

Technology at the Zoo Workshop

On the 6th and 7th May 2016 the project’s researchers hosted a workshop on technology at the zoo at the ACM SIGCHI Conference, the leading academic conference on human-computer interaction (HCI). Titled ‘HCI goes to the Zoo’, the workshop included delegates from leading zoos and institutions in North America,¬† Europe and Australasia, to discuss the opportunities and challenges for digital enrichment as part of zoo animal enrichment, zoo keeping and the visitor experience.


This project is a collaboration between The University of Melbourne’s Microsoft Research Centre for Social Natural User Interfaces (SocialNUI) and Zoos Victoria and proudly supported by Microsoft.

Project team

  • Sarah Webber, PhD Candidate, Microsoft Research Centre for SocialNUI, University of Melbourne
  • Zaher Joukhadar, Lead Software Engineer, Microsoft Research Centre for SocialNUI, University of Melbourne
  • Marcus Carter, Department of Media and Communications, University of Sydney
  • Sally Sherwen, Animal Welfare Specialist, Wildlife Conservation & Science, Zoos Victoria

Contact details


Pons, P., Carter, M., & Jaen, J. (In Press). Sound to your Objects: A Novel Design Approach to Evaluate Orangutans’ Interest in Sound-based Stimuli. In Proceedings of The Third International Congress on Animal Computer Interaction. ACM. [PDF]

Webber, S., Carter, M., Smith, W., & Vetere, F. (In Press). Interactive Technology and Human-Animal Encounters at the Zoo. International Journal of Human-Computer Studies [DOI] [PDF]

Carter, M., Webber, S., & Sherwen, S. (2015). Naturalism and ACI: Augmenting Zoo Enclosures with Digital Technology. In Proceedings of The Second International Congress on Animal Computer Interaction. ACM. [PDF]

Rault, J.-L., Webber, S., & Carter, M. (2015). Cross-Disciplinary Perspectives on Animal Welfare Science and Animal-Computer Interaction. In Proceedings of the Second International Congress on Animal Computer Interaction. ACM. [PDF]


The Age Newspaper (Print), 2 February 2016

Orang-utans get smart with technology