Health Times, 4 August 2017
Physiotherapy is currently taught via multiple modalities, such as lectures, tutorials, demonstrations and hands-on training in practical classes. However, physiotherapy students often struggle to translate anatomical knowledge from textbooks to dynamic understanding of the mechanics of body movements in real life patients. Moreover, one of the critical skills in physiotherapy, and other clinical domains, is clinical reasoning. Students must learn to negotiate the complexity of multiple factors, starting with the mastery of strong domain knowledge, such as human anatomy, to forming hypotheses through patient inquiry and testing the hypotheses through examination. Clinical reasoning is particularly difficult to teach because it requires students to critically review clinical decisions which are often ephemeral and tacit. The best practice for learning such skills typically involves a class of students observing an expert therapist conducting a consultation with a real patient (or a surrogate patient). However, such settings do not make the therapist’s clinical decision making explicitly available to students. One of the major difficulties for students in learning clinical reasoning is the ability to identify medical cues to develop diagnostic hypotheses.
This project aims to enhance student learning in physiotherapy via an augmented learning environment titled the ‘Augmented Studio’ developed using projectors and Microsoft Kinect sensors to enable projections of an anatomy model onto a moving body for practical physiotherapy classes. Augmented Studio enables augmentation through projection mapping to turn a human body into a display surface, showing anatomical information such as muscles and skeleton in real time. The moving body of a volunteer student becomes a live canvas for the teacher to illustrate clinical reasoning. Using a tablet interface with pen input, the instructor can draw annotations directly on the virtual body, which are rendered in real time on the patient’s body.
Currently the instructor uses still photos, figures, and diagrams which do not capture the real time and complex dynamics of human movements. With the Augmented Studio, the researchers explore the capability of creating virtual annotations, to identify certain anatomical structures or to illustrate the joints or muscle connections to teach students. The annotations enhance teacher and student communication with the purpose of transferring the real life knowledge and experience of the teacher to a class of students.
Augmented reality based learning and teaching presents an innovative pedagogical practice delivering benefits to both the staff and the students. The student learning experience is enhanced in multiple ways with the usage of the augmented reality learning environment. Augmented kinaesthetic information and instructor annotations support interactive observation for better clinical understanding of human anatomy and musculoskeletal structures. The ability to track a full body movement in 3 dimensions and examine the changes throughout the range of movement will enhance appreciation by students of the dynamic change in the anatomical configuration of the body.
The project aims to deliver positive learning outcomes for the students by enabling experiential learning and increased connection with the learning materials, lessons and the instructor, catering for both tactile and visual learning, and encouraging cognitive and clinical reasoning.
This project is a collaboration between the Microsoft Research Centre for Social Natural User Interfaces (SocialNUI) in the School of Computing and Information Systems and the Department of Physiotherapy at the University of Melbourne.
The project is supported by an ARC Discovery Project (DP130102142) and a University of Melbourne Learning and Teaching Initiative (LTI) Grant 2016.
Thuong Hoang, Research Fellow, Microsoft Research Centre for SocialNUI, University of Melbourne
Frank Vetere, Professor & Director, Microsoft Research Centre for SocialNUI, University of Melbourne
Zaher Joukhadar, Lead Software Engineer, Microsoft Research Centre for SocialNUI, University of Melbourne
Martin Reinoso, PhD Candidate, Microsoft Research Centre for SocialNUI, University of Melbourne
Louisa Remedios, Associate Professor, Department of Physiotherapy, University of Melbourne
David Kelly, Lecturer, Department of Physiotherapy, University of Melbourne
Egemen Tanin, Associate Professor, School of Computing and Information Systems, University of Melbourne
Hoang, T, Reinoso, M, Joukhadar, Z, Vetere, F & Kelly, D (2017) Augmented Studio: Projection Mapping on Moving Body for Physiotherapy Education. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, USA. [DOI]
Hoang, T., Reinoso, M., Vetere, F., & Tanin, E. (2016) Onebody: Remote Posture Guidance System using First Person View in Virtual Environment. In Proceedings of Nordic Conference on Human-Computer Interaction, 2016, Gothenburg, Sweden
Channel 10, 22 July 2017
Scope, Season 4, Episode 5
The Daily Mail Australia, 25 May 2017
The incredible 'zombie projector' that can virtually peel back the layers of a person to show their skeleton and muscles working
Futurity, 23 May 2017
Augmented reality turns live volunteers into anatomy lessons
Augmented Reality Pursuit article re-published on Futurity.
Pursuit, 12 April 2017