Teaching robots how to interact with children with autism

People with autism see, hear and feel the world differently from other people, which affects how they interact with others. This makes communication-centred activities quite challenging for children with autism spectrum conditions (ASCs). Therapists therefore find it difficult to engage them in these activities during educational therapy.

To address this challenge, therapists recently began to use humanoid robots in therapy sessions. However, existing robots lack the ability to autonomously engage with children, which is vital for improving the therapy. And the fact that people with ASCs have atypical and diverse styles of expressing their thoughts and feelings makes the use of such robots even more challenging.

Researchers working on the EU-funded project EngageME have now created a personalised machine learning framework for robots used during autism therapy. As they describe in their paper published in ‘Science Robotics’, this framework helps robots automatically perceive the affect – facial, vocal and gestural behaviour – and engagement of children as they interact with them.

A personalised approach

To achieve this exciting advance, project partners had realised that in the case of children with ASCs, one size doesn’t fit all. As a result, they personalised their framework to each child using demographic data, behavioural assessment scores and other characteristics unique to that child. The novel framework enabled the robots to automatically adapt their interpretations of children’s responses by taking into account cultural and individual differences between them.

“The challenge of creating machine learning and AI [artificial intelligence] that works in autism is particularly vexing, because the usual AI methods require a lot of data that are similar for each category that is learned. In autism where heterogeneity reigns, the normal AI approaches fail,” explained co-author Prof. Rosalind Picard in an article posted on ‘MIT News’.

Robot-assisted therapy

The researchers tested their model on 35 children from Japan and Serbia. Aged 3 to 13, the children interacted with the robots in 35-minute sessions. The humanoid robots conveyed different emotions – anger, fear, happiness and sadness – by changing the colour of their eyes, the tone of their voice and the position of their limbs.

As it interacted with a child, the robot would capture video of their facial expressions, movements and head pose, as well as audio recordings of their tone of voice and vocalisations. A monitor on each child’s wrist also provided the robot with data on their body temperature, heart rate and skin sweat response. The data was used to extract the child’s various behavioural cues and was then fed into the robot’s perception module.

Using deep learning models, the robot then estimated the child’s affect and engagement based on the extracted behavioural cues. The results were used to modulate the child-robot interaction in subsequent therapy sessions.

Audiovisual recordings of the therapy sessions were also observed by human experts. Their assessments of the children’s responses showed a 60 % correlation with the robots’ perceptions. This was a higher agreement level than achieved between human experts. The study’s results suggest that trained robots could play an important role in autism therapy in the future.

EngageME (Automated Measurement of Engagement Level of Children with Autism Spectrum Conditions during Human-robot Interaction) is working to augment robots with key information that will help therapists personalise therapies and make human-robot interaction more engaging and natural.


For more information, please see:
CORDIS web page