1     Functions.

2     Reference Architecture

3     I/O data of AI Workflow

4     Functions of AI Modules

5     I/O Data of AI Modules

6     AIW, AIMs, and JSON Metadata

1      Functions

Speech carries information not only about its lexical content, but also about several other aspects including age, gender, identity, and emotional state of the speaker. Speech synthesis is evolving towards support of these aspects.

In many use cases, emotional force can usefully be added to speech which by default would be neutral or emotionless, possibly with grades of a particular emotion. For instance, in a human-machine dialogue, messages conveyed by the machine can be more effective if they carry emotions appropriately related to the emotions detected in the human speaker.

Emotion-Enhanced Speech (EES):

  1. Enables a user to indicate a model utterance or an Emotion to obtain an emotionally charged version of a given utterance.
  2. Converts an individual emotionless speech segment to a segment that has a specified emotion. Both input and output speech segments are contained in files. The desired emotion is expressed either as a tag belonging to a standard list of emotions or derived by extracting features from a model utterance. EES produces an output speech segment with emotion.

CAE-EES implementations can be used to create virtual agents communicating as naturally as possible, and thus improve the quality of human-machine interaction by bringing it closer to human-human interchange.

2      Reference Architecture

The Emotion-Enhanced Speech Reference Model depicted in Figure 2 supports two Modes or pathways enabling addition of emotional charge to an emotionless or neutral input utterance (Emotion-less Speech).

Figure 2 – Emotion-Enhanced Speech Reference Model

3      I/O data of AI Workflow

Table 2 gives the input and output data of Emotion-Enhanced Speech.

Table 2I/O data of Emotion-Enhanced Speech

Input data Comments
Emotionless Speech An Audio File containing speech without music and other sounds, and in which little or no identifiable emotion is perceptible by native listeners.
Emotion A Data Type representing the internal status of a human or avatar resulting from their interaction with the context or subsets of it, such as “Angry”, and “Sad”.
Model Utterance An Audio Segment used as a model or demonstration of the Emotion to be added to Emotionless Speech in order to produce Speech with Emotion.
Output data Comments
Speech with Emotion An Audio File containing speech with emotional features.

4      Functions of AI Modules

The AI Modules of  Figure 2 perform the functions described in Table 3.

Table 3AI Modules of Emotion-Enhanced Speech

AIM Function
Speech Feature Analysis 1 Extracts Prosodic Speech Features of a model emotional Utterance and transfers them to Emotion Insertion1.
Speech Feature Analysis 2 Extracts Emotionless Speech Features of an emotionless input utterance, passing these to the Emotion Feature Production.
Emotion Feature Production Receives the Emotionless Speech Features produced by Speech Feature Analysis2 plus a list of Emotions to be added and Produces Neural Speech Features. (If the Degree of an Emotion is not specified, the Medium value is used.)
Emotion Insertion 1 Integrates the (emotional)Prosodic Speech Features with those of the Emotionless Speech input, yielding and delivering an emotionally modified utterance.
Emotion Insertion 2 Integrates the (emotional) Neural Speech Features with those of the Emotionless Speech input, yielding and delivering an emotionally modified utterance.

5      I/O Data of AI Modules

Table 4 – CAE-EES AIMs and their data

AIM Input Data Output Data
Speech Feature Analysis 1 Model Utterance Prosodic Speech Descriptors
Speech Feature Analysis 2 Emotionless Speech Emotionless Speech Descriptors
Emotion Feature Production Emotionless Speech Descriptors
Emotion List
Language Identifier
Neural Speech Descriptors
Emotion Insertion 1 Emotionless Speech
Prosodic Speech Descriptors
Speech with Emotion
Emotion Insertion 2 Emotionless Speech
Neural Speech Descriptors
Speech with Emotion

6      AIW, AIMs, and JSON Metadata

Table 5 – AIW, AIMs, and JSON Metadata

CAE-EES Emotion Enhanced Speech File
CAE-SF1 Speech Feature Analysis 1 File
CAE-SF2 Speech Feature Analysis 2 File
CAE-EFP Emotion Feature Production File
CAE-EI1 Emotion Insertion 1 File
CAE-EI2 Emotion Insertion 2 File