Tele-consultation: NUI Interactions Between Clinicians and Patients

A woman holding her neck while speaking with a doctor via video call on a laptop.

Project overview

Teleconsultation refers to the electronic communication that happens between a clinician and patient for the purpose of diagnostic or therapeutic advice. They are particularly useful to provide healthcare services in situations where face-to-face consultation may not be easy. To date Tele-consultations session have been primarily supported by videoconferencing systems (such as Skype) via web cams to video and audio communication. While these systems are advantageous for teleconsultations they do not fully support diagnosing and treating physical injuries that require touch and movement. Current tele-consultation systems often require the support of a third person to act as a proxy for the clinician to conduct the physical examination.

Recent developments in natural user interface (NUI) technologies provide various opportunities to sense and track the human body in order to enhance the quality of tele-consultations. For example, depth cameras like the Microsoft Kinect can analyse the body’s form and sense movement to aid diagnosis and to track progress in physical rehabilitation after an injury or surgery. Mobile phone cameras offer support for detailed scans of eyes to detect anaemia or moles to detect skin cancer. The combination of depth cameras with mobile phones creates new opportunities to freely zoom in on different parts of the body, e.g. to assess wounds or joint movements.

NUI technologies hold great potential to enhance current tele-consultation systems and to improve health outcomes. However, it is unclear how these technologies will shape the experience of tele-consultations, both for the patient and the clinician. How will they affect current routines and practices? How will they affect mutual understanding, trust and empathy between patient and clinician? Will they provide a sense of enhanced presence between both sides more akin to face-to-face consultations, or will they create entirely new experiences and relationships between patients and their clinicians?

The aim of this project is to understand and improve the experience of a tele-consultation between a patient and their clinician through technology to support bodily interactions so teleconsultations move closer to the experience of a face-to-face consultation. Bodily communication can entail nonverbal communication through facial expression, gaze, and posture, complaints like pain, stiffness, inflammations, exhaustion, as well as physical exercises communicated over a distance to remediate such concerns. This project will develop new technology and tangible interface to support lower limb pain communication that happens in a teleconsultation with a physiotherapist.


This project is a collaboration between the Royal Children’s Hospital via Mark Bradford in the Chronic Pain Management CPMS and the Microsoft Research Centre for Social Natural User Interfaces (SocialNUI) at the University of Melbourne.

Project team

  • Deepti Aggarwal, PhD Candidate, Microsoft Research Centre for SocialNUI, University of Melbourne
  • Frank Vetere
    Frank Vetere, Professor & Director, Microsoft Research Centre for SocialNUI, University of Melbourne
  • Bernd Ploderer, Honorary Fellow, University of Melbourne and Lecturer, Queensland University of Technology
  • Thuong Hoang
    Thuong Hoang, Research Fellow, Microsoft Research Centre for SocialNUI, University of Melbourne
  • Steven Baker
    Steven Baker, Research Fellow, Microsoft Research Centre for SocialNUI, University of Melbourne
    Mark Bradford, Physical Therapist, Royal Children’s Hospital

Contact details


Aggarwal, D., Ploderer, B., V., Vetere, F., Bradford, M., & Hoang, T. (2016). Doctor, can you see my squats? Understanding bodily communication in video consultations for physiotherapy. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS 2016). New York: ACM Press, pp.1197–1208 [PDF] [DOI]

Aggarwal, D. (2016) Supporting Bodily Communication in Video Based Clinical Consultations. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI 2016). New York: ACM Press, pp. 188–192 [PDF] [DOI]

Aggarwal, D., Ploderer, B., & Vetere, F. (2015) Addressing research gaps in teleconsultation settings. In CHI 2015 Workshop on Everyday Telepresence: Emerging Practices and Future Research Directions. Seoul, Korea. [PDF]