TutAR: Semi-Automatic Generation of Augmented Reality Tutorials

With Augmented Reality (AR) on Optical-See-Through-Head-Mounted Displays, users can observe the real world and computer graphics simultaneously. Creating AR applications is hard. In addition to master the technical components of an AR system, the author needs to have an in-depth knowledge of 3D modeling tools. Thus, creating AR applications are inaccessible to the casual user. On the other hand, recording and authoring videos are accessible to the masses with the high adoption rate of smartphone and their ease of use.

With the recent success of video sharing portals, homemade tutorial videos of every aspect of life are available for everyone. Video tutorials convey complex movement information. However, following instructions in a video and applying them to real world requires mentally complex hand-eye coordination. AR tutorials are proven to reduce the cognitive load significantly.

In my thesis, I present the design and implementation of TutAR, a pipeline that creates semi-automatically AR tutorials of 2D RGB videos. TutAR extracts relevant 3D hand motion from the input video. The acquired motion will be displayed as an animated 3D hand relative to the human body and plays synchronously with the movement in the video on an Optical-See-Through Head-Mounted Display. This approach can be applied to many video tutorials. In this work, we concentrate on a class of tutorials which uses the human hand as the primary tool. In a user study with 16 participants, we compared the performance of chest compression for cardiopulmonary resuscitation (CPR) of subjects who used an AR animation that has been created by TutAR vs. subjects who just saw the original video for training. Both subject groups used an Optical-See-Through Head-Mounted Display. We could not find a significant improvement of CPR performance with the use of an AR animation created by TutAR in comparison to a video. This may be due to the small number of participants who could be recruited as well as the type of visualization of TutAR.