1     Functions

2     Reference Model

3     I/O data of AI Workflow

4     Functions of AI Modules

5     I/O Data of AI Modules

6     AIW, AIMs, and JSON Metadata

1      Functions

Speech carries information not only about its lexical content, but also about several other aspects including age, gender, identity, and emotional state of the speaker. Speech synthesis is evolving towards support of these aspects.

In many use cases, emotional force can usefully be added to speech which by default would be neutral or emotionless, possibly with grades of a particular emotion. For instance, in a human-machine dialogue, messages conveyed by the machine can be more effective if they carry emotions appropriately related to the emotions detected in the human speaker.

Emotion-Enhanced Speech (EES):

  1. Enables a user to indicate a model utterance or an Emotion to obtain an emotionally charged version of a given utterance.
  2. Converts an individual emotionless speech segment to a segment that has a specified emotion. Both input and output speech segments are contained in files. The desired emotion is expressed either as a tag belonging to a standard list of emotions or derived by extracting features from a model utterance. EES produces an output speech segment with emotion.

CAE-EES implementations can be used to create virtual agents communicating as naturally as possible, and thus improve the quality of human-machine interaction by bringing it closer to human-human interchange.

2      Reference Model

The Emotion-Enhanced Speech Reference Model depicted in Figure 2 supports two Modes or pathways enabling addition of emotional charge to an emotionless or neutral input utterance (Emotion-less Speech).

Figure 2 – Emotion-Enhanced Speech Reference Model

3      I/O data of AI Workflow

Table 2 gives the input and output data of Emotion-Enhanced Speech.

Table 2I/O data of Emotion-Enhanced Speech

Input data Comments
Emotionless Speech See definition in Table 1.
Emotion See definition in Table 1.
Model Utterance See definition in Table 1.
Output data Comments
Speech with Emotion See definition in Table 1.

4      Functions of AI Modules

The AI Modules perform the functions described in Table 3.

Table 3AI Modules of Emotion-Enhanced Speech

AIM Function
Speech Feature Analysis 1 Extracts Neural Speech Features of a model emotional utterance and transfers them to the Prosodic Emotion Insertion AIM.
Speech Feature Analysis 2 Extracts Emotionless Speech Features of an emotionless input utterance, passing these to the Emotion Feature Production AIM.
Emotion Feature Production Receives the Emotionless Speech Features produced by Speech Feature Analysis2 plus a list of Emotions to be added. (If the Degree of an Emotion is not specified, the Medium value is used.)
Prosodic Emotion Insertion Integrates the (emotional) Prosodic Speech Features with those of the Emotionless Speech input, yielding and delivering an emotionally modified utterance.
Neural Emotion Insertion Integrates the (emotional) Neural Speech Features with those of the Emotionless Speech input, yielding and delivering an emotionally modified utterance.

5      I/O Data of AI Modules

Table 4 – CAE-EES AIMs and their data

AIM Input Data Output Data
Speech Features Analysis1 Model Utterance Prosodic Speech Features
Speech Features Analysis2 Emotionless Speech Emotionless Speech Features
Emotion Feature Production Emotionless Speech Features

Emotion List

Language

Neural Speech Features
Prosodic Emotion Insertion Emotionless Speech

Prosodic Speech Features

Speech with Emotion
Neural Emotion Insertion Emotionless Speech

Neural Speech Features

Speech with Emotion

1.1.6      AIW, AIMs, and JSON Metadata

Table 5 – AIW, AIMs, and JSON Metadata

AIW AIMs Name JSON
CAE-EES Emotion Enhanced Speech File
CAE-SF1 Speech Feature Analysis 1 File
CAE-SF2 Speech Feature Analysis 2 File
CAE-EFP Emotion Feature Production File
CAE-PEI Prosodic Emotion Insertion File
CAE-NEI Neural Emotion Insertion File