Cognitive and Perceptual Effects of Cross-Modal Illusions in Augmented Reality

My Ph.D. research aims to explore to which extent Augmented Reality (AR) experiences can evoke measurable physiological, psychological, and neurological responses in the user’s body. During the first year of my Ph.D. studies, I have developed an AR experience that enables users to see and hear their own hands burning while looking through a Video See-Through Head-Mounted Display (VST-HMD).

A series of three experiments are presented and future experiments are being proposed. The first experiment examined the prevalence of heat illusions and physiological and psychological stress responses induced by your AR experience. Half of the participants reported a heat-sensation on the affected left hand. All participants experienced a significant increase in skin conductance during the experiment. Moreover, participants who experienced a heat sensation had a higher skin conductance response.

The second experiment investigated whether a heat illusion would lead to thermoregulatory responses. Using a Laser Doppler Flowmeter, I demonstrated that skin blood flow in the affected hand changed significantly in some participants. In a follow-up study, I showed that this experience also affected hand microcirculatory blood flow in some participants.

Experiment three examined whether this AR experience could manipulate participants' perception of pain. The virtual flames on participants' hands led to a decrease in the temperature at which they perceived heat-related pain. These results suggest that AR-induced stimuli may be an effective method for top-down modulation of the pain experience.

The main conclusions of the first three experiments can be summarized as follows: (1) AR can reliably evoke cross-modal illusions in subjects through visual and auditory stimuli. (2) Involuntary heat illusions in AR can lead to some thermoregulatory responses, such as an increase in blood flow in the skin. (3) AR may strongly affect pain perception.

My research will continue with several planned follow-up experiments to further explore how AR experiences can have measurable effects on the user’s body and mental state. This will provide deeper insights into the perceptual and cognitive effects unique to AR experiences. Findings from our experiments could be relevant in a neuroscience or medical context.

Daniel Eckhoff
Daniel Eckhoff
PhD Researcher