BLI’s spoken dialog and character animation technologies are used in programs that:
To understand how BLI’s human language and character animation technologies support graceful spoken dialogs between children and a virtual tutor, we present a high level overview of how these systems work together during tutorial dialogs.
When Marni asks a question, her behaviors are controlled by the BLI Avatar system, which controls her speech and facial expressions when she talks. The input to the BLI Avatar system is provided by the Phoenix spoken dialog system, which provide the avatar system with a speech waveform and time-aligned phonetic transcription of the utterance to be spoken. When the student responds to Marni’s questions, the Bavieca speech recognition system recognizes the child’s speech, and passes the string of hypothesized words to the Phoenix Spoken Dialog system. Phoenix uses natural language processing to interpret the student’s speech; it determines what the student knows—the points the student has already made in pursuit of a complete and accurate explanation, and what points the student has yet to explain. The dialog system then decides what action to take next, such as asking a follow-up question, presenting a new animation, and takes immediate feedback actions, such as generating a follow-on question or moving on to a new point, and whether to present new media. The language generation system constructs the next prompt Marni will produce, and passes the text string and speech waveform, recorded by a human tutor, to the BLI avatar system, which synchronizes the pre-recorded utterances with Marni’s mouth, head and face movements. Because all speech produced by Marni is recorded by an experienced and sensitive human tutor, who understands the dialog context of each utterance produced, and produces them with appropriate prosody, Marni takes on the personality portrayed by the tutor’s speech. For this reason, the majority of students who have interacted with Marni have reported that she acts like a real teacher and that they enjoy working with her. More detailed information about MyST’s technology systems and how the work together during tutorial dialogs with Marni is provided in Ward et al., 2011; 2013.
In MindStars Books, Marni teaches students science by narrating multimedia science presentations, assessing their understanding of the presentations through thoughtful multiple choice questions with challenging answer choices, and facilitates learning by providing immediate feedback to students on their answer choices. Correct answers are reinforced and followed by a brief expansion of the answer, while incorrect answers are followed by a hint designed to scaffold learning and stimulate the student to reason about the science before making a second choice. Science learning in the MindStars Books emphasizes children’s development of listening comprehension strategies. Research has shown that listening comprehension is optimal when students can listen to explanations of science presented in media, enabling learners to construct rich multimodal mental models. Research has also shown that presenting the text of the narration is actually distracting to the learner, and leads to decreased learning. However, once the learner has completed listening comprehension activities, they practice reading the science texts, and engage in activities in which the Bavieca speech recognition system is used to help them achieve oral reading fluency, leading to improved reading comprehension.