Dr. Connie YUEN Using AI-based Mobile App to Support Children with Speech Sound Disorders

Title of Project: Development and Pilot Test for an AI-based Mobile App to act as Speech Language Pathology Robot for Long-term Treatment of Cantonese-speaking Children with Developmental Speech Sound Disorders

Principal Investigator: Dr. Connie YUEN (Head of Department of Applied
Data Science, Assistant Professor)

Funding Source: Competitive Research Funding Schemes for the Local
Self-financing Degree Sector 2021/22 of the Research Grants Council (RGC) – Faculty Development Scheme (FDS)

Amount of Grant: HK$1,412,117

Project Duration: 24 months

The most common form of communication between people is verbal language. As children grow up, they learn from the pronunciation of others and gradually acquire the correct speech sounds, but if there is a disorder in speech sound, communication with others will be affected. Developmental speech sound disorder (SSD) is a condition in which a child, without a medical condition or physical defect, past the age at which he or she should acquire a certain phonetics, but still has a problem with misconstruing speech sounds and is unable to produce the correct phonetics. Speech sounds refers to the process of producing phonetics through speech and sound organs such as lips and tongue.

In an interview with Shue Yan Newsletter, Dr. Connie YUEN, the Head of Department of Applied Data Science said, developmental speech sound disorder is a common communication disorder among young children and highly affects early childhood development and even their future educational, professional, and social success. Based on the literature review, 24.6% of young children in the world were estimated to have speech delay or speech sound disorder. Among these children, 48.1% of 3- to 10-year-old children and 24.4% of 11- to 17-year-old children had speech sound problems only. These children can improve their speech intelligibility after regular training. In Hong Kong, however, there are certain difficulties in treating and training such children. One of the reasons for this is because of the characteristics of the Cantonese language itself. Since Cantonese is primarily a spoken language, and written Cantonese is not acceptable in standard written Chinese, it increases the difficulty for young children in Hong Kong to learn and understand spoken Cantonese, especially for Cantonese-speaking children with SSD. The other reason is that there are fewer training materials for Cantonese-speaking children with SSD compared with other languages.

Dr. YUEN, who has always been concerned about the growth and development of children, thought of using a mobile app to support the treatment of these children.

Dr. YUEN said, once children with SSD are identified, speech-language pathologists (SLPs) select initial therapy programs for children, and then the program is carried out by SLPs and parents together. The success of therapy highly relies on the effectiveness of home training. Parents need training tools to accompany their children during speech therapy training at home, and beyond-clinic measurements are needed to provide SLPs with better insight into the training progress of children with SSD. Furthermore, due to the insufficient number of SLPs in many places, more resources to support the work of SLPs are required, for example, helping SLPs to collect and understand the correct conditions of patients quickly during short in-clinical treatment periods. At the same time, children might feel bored during long-term regular training at home and would therefore require motivation and encouragement from their parents.

Using an AI mobile app to support treatment

“To address the above problems, I propose an AI mobile app to act as a speech language pathology robot to provide training to children with SSD at home. The mobile app has a cartoon face, collects visual data from a camera and voice data from a microphone, and provides visual instructions through a screen and audio instructions through speakers. Parents simply download the mobile app, which can be accessed easily in a home setting. The mobile app can automatically monitor the training progress by collecting video data and voice data, and by analyzing this data in an intelligent cloud-based platform. The analyzed results in the cloud can be used by SLPs for in-clinic consultation sessions, thus SLPs can understand the training progress of the children easily, quickly, and accurately, and provide better therapy for the children,” said Dr. YUEN.

Dr. Connie YUEN’s project is conducted in 5 phases:

(1) Collect children’s speech data and establish a database, including children with normal phonetics and SSD.

(2) Organize and analyze data.

(3) Develop AI mobile application.

(4) Test the mobile app.

(5) Evaluate the mobile app – To collect opinions on the use of mobile apps by children with SSD and related persons, to examine how helpful the app is to home training for children with SSD, and to further examine the effectiveness of the mobile app.

Dr. YUEN added that under the ongoing COVID-19 pandemic, parents are less likely to take a child with SSD out for treatment. She believes that after the mobile app is launched, it can play a better role in support and treatment.