Pop’Balloons, the first serious mixed reality game for autistic children

Demand for mental health assessment and intervention is soaring; available services are stretched to the limit, centralized and difficult for many to access. This is particularly true for children with autism spectrum disorder (ASD).

“Serious games” that can help meet the pressing need are now starting to appear. One of the first is Pop’Balloons, a game designed to help autistic kids develop their motor skills and that is based on years of fundamental research.

The project was led by Guillaume Dumas, a professor of computational psychiatry in Université de Montréal’s Faculty of Medicine and director of the Precision Psychiatry and Social Physiology Laboratory at the CHU Sainte-Justine Research Centre. Dumas also holds the IVADO Chair in AI and Mental Health and is an associate academic member of Mila, the Quebec Artificial Intelligence Institute.

It is possible to record a people’s brain activity by placing them in a scanner or attaching electrodes to their scalp and performing an encephalogram, but these technologies were designed for examining only one person at a time, and so had drawbacks.

Typically, to measure brain activity during social interaction, researchers would expose people to recorded or simulated social stimuli. So, paradoxically, neuroscientists who studied social interactions did so by observing the brains of isolated individuals in labs.

Luckily, the eventual development of “hyperscanning” made it possible to simultaneously record the activity of several brains and thus study real-life social interactions.

People’s brains in sync

In 2010, while using hyperscanning to study spontaneous imitation in his native France, Dumas discovered that people’s brains synchronize with each other during social interaction – proving that people can literally be on the same wavelength.

Four years later, Dumas left France to do a post-doc in the U.S., where he continued his research. “We developed a system to record a human being in interaction with a virtual avatar,” Dumas recalled. “The avatar’s behaviour wasn’t tightly scripted; it reacted to the human in real time, in milliseconds.”

This virtual partner passed the Turing test, meaning it could not be distinguished from a human being. Dumas called the cognitive-neuroscience-based human-machine interaction system he had created the Human Dynamic Clamp. The avatar was able to adjust to human behaviours and also to prompt new ones through its interactions with users. Initially, it was tested on neurotypical humans.

Studies have suggested that sensory-motor differences in individuals diagnosed with ASD may be the cause of their social interaction difficulties.

The human-machine interface developed by Dumas supported the extraction of motor coordination and interpersonal synchronization indices, so he exploited these capabilities to create a new machine-assisted method of neuropsychological evaluation of ASD, which has since been clinically tested on more than 155 subjects and peer-validated.

As the machine had demonstrated its ability to teach new behaviours, Dumas’ team also worked on a version that could help people with autism develop their motor skills.

‘Your video games sucks!’

Dumas tested this version on autistic children at the Hôpital universitaire Robert-Debré, in Paris. “Children are blunt and they told me straight out: ‘Your video game sucks!’ It’s true that it was a research protocol and not visually appealing.”

Working with students at the École Centrale de Paris, he went on to develop a prototype of a more engaging video game. The company Act’image then collaborated with the Pasteur Institute to develop a 3D version of the game. A partnership with Microsoft’s HoloLens goggles yielded a mixed reality version of the game. Unlike virtual reality, which takes users into an imaginary immersive experience, mixed reality superimposes holograms onto the real world.

The result was Pop’Balloons, a game that lets young people with ASD safely explore their real environment. Holograms of balloons are virtually inserted into a room and the object of the game is to burst them. The faster the balloons are popped, the higher the score. Childrenwith autism can investigate the room at their own pace and repeat the experience as many times as necessary.

An open-access version was then developed with financial support from the Orange Foundation, which enabled Dumas to continue working on the game in Montreal when he was hired as a professor at UdeM.

Towards precision medicine

A collaboration with the Canadian video game studio Eidos helped enhance the game. And connecting with Mila, the Quebec Artificial Intelligence Institute, made it possible to develop more advanced machine learning techniques. IVADO’s dedicated collaborative research team facilitated linkages between UdeM and the video game industry, which has the expertise to develop more engaging serious games.

“Starting from the video game, we were able to go as far as clinical stratification—dividing study participants into sub-groups—for precision medicine trials in mental health,” said Dumas. “That is what we do at my Precision Psychiatry and Social Physiology Laboratory: try to understand physiology and the bases of sociality in order to create psychiatric approaches adapted to the individual.”

A paper reporting the preliminary results of clinical stratification based on video game data has just been accepted by NeurIPS, the most prestigious and competitive international conference on machine learning and computational neuroscience, being held this year in New Orleans.

A game accessible to all

Can a game be created that’s accessible to all, including neurodiverse players? Could it be given to all autistic children without some ripping off their headsets? “The first step is to validate the accessibility parameters,” saod Romain Trachel, a machine-learning specialist at Eidos-Montréal, one of the few video game studios with an accessibility department. “We’re talking about a population that often has more developed perception than other people.”

Added Mariem Hafsia, a master’s student at UdeM: “The first step is to make sure the game is comfortable for children with autism. Will they be at ease with the device? Will they have difficulty grasping the concept of the game? Will they find it engaging? Maybe playing with a balloon doesn’t mean the same thing to children with atypical development.”

The hope is that this first game could serve as a model for developing more inclusive video games. As well, Pop’Balloons could potentially be used in the future to perform neuropsychological assessments in a non-medical setting.

“The game wouldn’t necessarily be used to make diagnoses that require a psychologist or physician,” said Dumas. “It could support the health care system, which is very congested and has long wait times for evaluations. The game could save time for health care professionals; they wouldn’t have to spend hours performing a clinical evaluation.”

Who knows? Someday the game could also be used in personalized telemedicine.

“It could have a dashboard on which a clinician could track the patient and see their performance history,” Dumas suggested. “The patient wouldn’t have to come back to the clinic every time. The game itself could direct them to the most appropriate levels of play to help them improve. It would be an adaptive video game that could serve as a psycho-educational intervention.”