#idea Meet JAC - Jason's Aid to Communications

A computer system with speech recognition and speech output that helps develop the oral language skills of Jason.

The Vision and scenery

Jason is autistic and has poor oral communications skills but he has very good visual communications skills.

The goal is to increase Jason vocabulary and quality of speech diction
The computer system is called JAC and is a  patient and smart speech therapist.
Jason's current tutors can spend hours on small improvement
This is often very repetitive and frustrating for a human tutor.
I want a device that engages Jason in conversation and listens to his response and rates the conversation. The conversations will be based on reflecting about the day's activities.
JAC will also prompt Jason with picture symbols and a verbal announcement using a SONOS speaker system. JAC will request Jason to reply. The known and reference speech diction is compare with Jason's response and this is rated. Like a language tutor JAC requests that Jason repeat his reply for a poor response. Jason at the moment is echolalia  - that he often echoes more than once and we want to control this and there is an obvious conflict of interests here that still needs to be resolved. JAC will measure Jason's performance in this area.

What is based on?

  • A Chatbot
The conversational part of JAC will be based on a Chatbot. The big difference is that JAC leads the conversation instead of just replying as most chat bots do.
JAC will emulate some of the features of Harlie - A chatbot app on Android for autistic people. It was develop in Queensland Australia by the CSIRO and University of Queensland.
  • Jason's visual experiences - this requires JAC to have photos of what Jason did that day for the basis of conversation. Initially a human will have to translate today's photos to simple textual descriptions but latter an image recognition system may be able to do this automatically
  • node-red user interface on Chrome. This will provide the human tutor with Jason's performance report for the day
List materials and tools both hardware and software

  • A computer system for developing the software on an IDE - a typical desktop PC or Mac will do
  • A microcomputer system with speech input and output
LinkIt may be a good candidate
  • A microphone
  • A quality speaker system such as SONOS
  • A speech recognition system
consider IBM Watson node-red modules
  • A text to speech system such as IBM Watson
  • SONOS node-red modules from the node-red web library
  • Software IDE
consider using node-red

Method to ensure its technical feasibility

  • Testing each step

Check that the sub-system speech to text for Jason's existing language style works 

Check the text to speech is understood by Jason

Verify that the analysis is able to count echos in speech

The chatbot should engage Jason in at least a 5 minute conversation

Check that LinkIt has adequate IO and CPU performance if not off load some of the tasks esp the analysis

  • Has the Goal being Achieved - provide a measurable performance

I will be happy if Jason improves his oral vocab by 50% in the next year. Jason will improve the speech diction so that a human tutor understands 90% his conversations, requests and replies


This is a difficult and challenging project. All speech to text technologies rely on speakers with good speech diction but here the speech will often be fuzzy and difficult to understand. Some advanced signal processing algorithms may have to be investigated and developed if the current speech to text software is inappropriate

dsrc12Jul 20,2016 15:04 PMAdd Comment


Popular Topics