Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance
1 Functions
Speech Restoration System restores a Damaged Segment, a Speech Object containing only speech from a single speaker. The damage may affect the entire segment, or only part of it.
Restoration will not involve filtering or signal processing. Instead, replacements for the damaged vocal elements will be synthesised using a speech model. The latter is a component or set of components, normally including one or more neural networks, which accepts text and possibly other specifications, and delivers audible speech in a specified format – here, the speech of the required replacement or replacements. If the damage affects the entire segment, an entirely new segment is synthesized; if only parts are affected, corresponding segments will be synthesized individually to enable later integration into the undamaged parts of the Damaged Segment, with reference to appropriate Time Labels.
The speech segments necessary for the creation of the speech model can be flexibly resourced from undamaged parts of the input segment or from other recording sources that are consistent with the original segment’s acoustic environment.
Restoration is carried out by synthesising replacements for the damaged vocal elements as follows:
The Speech Segments for Modelling – Speech Objects necessary for the creation of the Neural Network Speech Model – may be obtained from any undamaged parts of the input speech segment; however, other Speech Objects consistent with the original segment’s sound environment can also be used.
The Speech Restoration System (CAE‑SRS) AIM:
| Receives | Speech Segments for Modelling | A set of Speech Objects used to train the Neural Network Speech Model. The Speech Qualifier of each states whether its data is a speech file or samples. |
| Text List | List of texts to be converted into speech by the Speech Synthesis for Restoration AIM. | |
| Damaged List | A list of strings of Texts corresponding to the Damaged Segments (if any) requiring replacement with synthetic segments. | |
| Damaged Segment | A Speech Object containing only speech (and not containing music or other sounds) which is either damaged in its entirety or contains one or more Damaged Sections specified in the Damaged List. | |
| Produces | Restored Speech Segment | A Speech Object in which the entire segment has been replaced by a synthetic speech segment, or in which each Damaged Segment has been replaced by a synthetic speech segment. |
2 Reference Model
Figure 1 depicts the Reference Model of the Speech Restoration System (CAE‑SRS) AIM.
Figure 1 gives the Reference Model of the Speech Restoration System.
Figure 1 – Speech Restoration System (SRS) Reference Model
3 I/O Data
Table 1 specifies the Input and Output Data of the Speech Restoration System (CAE‑SRS) AIM.
| Input | Description |
|---|---|
| Speech Segments for Modelling | A set of Speech Objects used to train the Neural Network Speech Model. The Speech Qualifier of each states whether its data is a speech file or samples. |
| Text List | List of texts to be converted into speech by the Speech Synthesis for Restoration AIM. |
| Damaged List | A list of strings of Texts corresponding to the Damaged Segments (if any) requiring replacement with synthetic segments. |
| Damaged Segment | A Speech Object containing only speech (and not containing music or other sounds) which is either damaged in its entirety or contains one or more Damaged Sections specified in the Damaged List. |
| Output | Description |
| Restored Speech Segment | A Speech Object in which the entire segment has been replaced by a synthetic speech segment, or in which each Damaged Segment has been replaced by a synthetic speech segment. |
4 SubAIMs
4.1 Reference Model
The structure of the Speech Restoration System (CAE‑SRS) Composite AIM is depicted in Figure 1.
4.2 Operation
In the SRS use case, the entire Damaged Segment can be replaced by a synthesised segment, or parts within it can be synthesized to enable integration of the replaced segments.
4.3 Functions of SubAIMs
Table 2 specifies the functions of the Speech Restoration System (CAE‑SRS) SubAIMs.
| SubAIM | Function |
|---|---|
| Speech Model Creation | 1. Receives the Speech Segments for Modelling, adequate for model creation, each as a Speech Object. 2. Creates the current Neural Network Speech Model. 3. Sends that Neural Network Speech Model to the Speech Synthesis for Restoration. |
| Text To Speech | 1. Receives 1.1. The current Neural Network Speech Model. 1.2. Damaged List as a data structure: 1.2.1. Containing one element if Damaged Segment is damaged throughout or 1.2.2. Representing a list in which each element specifies via Time Labels the start and end of a damaged section within Damaged Segment. 2. Synthesizes each Damaged Section in Damaged List. 3. Sends the newly synthesised segments to the Speech Restoration Assembly as an ordered list. |
| Speech Restoration Assembly | 1. Receives the Damaged Segment. 2. Receives the ordered list of synthetic segments. 3. Receives Damaged List Time Labels, indicating where the synthesized segments should be inserted in left-to-right order. In case Damaged Segment as a whole was damaged, the list contains one entry. 4. Assembles the final version of the Restored Segment. |
4.4 I/O Data of SubAIMs
Table 3 gives the Input and Output Data of the Speech Restoration System (CAE‑SRS) SubAIMs.
| SubAIM | Input | Output |
|---|---|---|
| Speech Model Creation | Speech Segments for Modelling | Neural Network Speech Model |
| Text To Speech | Text List Neural Network Speech Model |
Synthesised Speech |
| Speech Restoration Assembly | Damaged Segments Damaged List |
Restored Segment |
4.5 AIMs and JSON Metadata
Table 4 provides the links to the AIM specifications and JSON schemas. AIM1 indicates the Composite AIM and AIM2 its SubAIMs.
| AIM1 | AIM2 | Name | JSON |
|---|---|---|---|
| CAE‑SRS | Speech Restoration System | X | |
| CAE‑SMC | Speech Model Creation | X | |
| MMC‑TTS | Text To Speech | ||
| CAE‑SRA | Speech Restoration Assembly | X |
4.6 Use of the Basic API
The SubAIMs of the Speech Restoration System (CAE‑SRS) conform to the Basic API. This section states what each SubAIM asks of the Controller and what it gives back through the functions called by AIMs (MPAI_AIFM_). How a SubAIM processes its data is for the Implementer.
Speech Model Creation
- Receives the Speech Segments for Modelling at its input Port. Each Speech Object carries its data by value or by reference: where by reference, its data entry states DataLength and DataURI and the SubAIM reads it with MPAI_AIFM_Payload_Get. The Speech Qualifier of the Speech Object says whether the data is a speech file or samples.
- Obtains the resources that creating the model requires with the Resource functions (MPAI_AIFM_Resource_GetGlobal and MPAI_AIFM_Resource_GetMODULE).
- Creates the Neural Network Speech Model. The Basic API has no function to train a model: how the model is created is for the Implementer. While the SubAIM creates the model, the Controller may pause, resume or stop it.
- Gives the Neural Network Speech Model, an ML Model Object, at its output Port. The data of the model is carried by reference: the SubAIM places it with MPAI_AIFM_Payload_Put and the data entry of the ML Model Object states DataLength and DataURI.
- If the Speech Segments for Modelling do not allow it to create a model of the intended quality, it says so with MPAI_AIFM_AIM_Report and gives the model it could create. A report is not an error.
Text To Speech
- Receives the Text List and the Neural Network Speech Model at its input Ports, and reads the data of the model with MPAI_AIFM_Payload_Get.
- Runs when it has both.
- May take an updated Neural Network Speech Model with MPAI_AIFM_Model_Update. How the update is effected without impacting the operation of the SubAIM is for the Implementer.
- Gives the Synthesised Speech at its output Port.
Speech Restoration Assembly
- Receives the Damaged Segment, the Damaged List and the Synthesised Speech at its input Ports.
- Runs when it has all three, and gives the Restored Segment at its output Port.
Speech Restoration System
- Speech Model Creation runs once for each set of Speech Segments for Modelling it receives. Text To Speech uses the Neural Network Speech Model for every Damaged Section of the Damaged List.
- The Module may monitor the Neural Network Speech Model with MPAI_AIFM_Model_Drift. If the Damaged Segment differs significantly from the speech the model was created from, new Speech Segments for Modelling may be given to Speech Model Creation, so that a new Neural Network Speech Model is created.
5 JSON Metadata
https://schemas.mpai.community/CAE1/V2.5/AIMs/SpeechRestorationSystem.json
6 Profiles
No Profiles.
7 Reference Software
8 Conformance Testing
Table 5 provides the Conformance Testing Method for the Speech Restoration System (CAE‑SRS) AIM. Conformance Testing of the individual SubAIMs is given by the individual AIM specifications.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.
| Receives | Speech Segments for Modelling | Shall validate against the Speech Object Schema. The Qualifier shall validate against the Speech Qualifier schema. |
| Text List | Shall validate against the TextObject Schema. The Qualifier shall validate against the TextQualifier schema. |
|
| Damaged List | Shall validate against the Damaged List Schema. Shall validate against the Time Schema. The Time Qualifier shall validate against the Time Qualifier schema. |
|
| Damaged Segment | Shall validate against the Speech Object Schema. The Qualifier shall validate against the Speech Qualifier schema. |
|
| Produces | Restored Speech Segment | Shall validate against the Speech Object Schema. The Qualifier shall validate against the Speech Qualifier schema. |
