<- API Go to ToC

1 General 2 Components 3 The API as implemented
4 Extensions 5 The Apps 6 AI Modules and models
7 Installation and operation 8 Security and privacy 9 Verification
10 Access to the code

1 General

The MPAI-MAS Reference Software – called Thalia – implements the Service Controller Instance (SCI) and the Server API specified by Technical Specification: MPAI as a Service (MPAI-MAS) V1.0, and Remote Client Applications (RCA) that use it. Through them, a person with a web browser or a Windows desktop can use AI Modules executed in an AI Framework conforming with Technical Specification: AI Framework (MPAI-AIF) V3.0 on a remote server: talk with an avatar, ask a question about a picture, have speech translated, be registered and recognised.

The software is written in C# for .NET 10. The server runs on Windows and Linux; the browser client runs in any current browser; the desktop client runs on Windows.

2 Components

Component Function
MAS Service The Server API and the SCI. It exposes the /MPAI/AIFU routes of the API over HTTP(S), creates the Controller Instances requested by RCAs, and executes the requested Modules in an MPAI-AIF V3.0 Controller. Each Module is built from its Implementation Description (L3), which is checked against its Specification Description (L2) when it is submitted to the MPAI Store. All Controller Instances share one AIF Controller, in which each Module is instantiated once and executed for the clients in turn. The Service is configured by one JSON file.
Browser client An RCA running in the browser (Blazor WebAssembly), served by a host program that also forwards the /MPAI/AIFU requests to the Service, so that the browser talks to one origin. It acquires audio, text and pictures from the microphone, the keyboard and the camera, and presents the answers through a speaking avatar.
Desktop client An RCA for Windows with the same functions, reaching the Service directly.
Apps What the RCA offers the user. An App is a Module (its L3) and a workflow that tells the RCA which data to acquire, which Ports to send them to, and what to do with the answers. The workflow addresses the Module’s Ports only by Data Type and Port Number, never by the names of AI Modules.
MPAI Store (optional) When configured, the Service offers only the Apps whose Module’s L3 the Store has approved, and may fetch L3s, AIM packages and models from it.

3 The API as implemented

All routes are under /MPAI/AIFU. In this implementation {pid} identifies a Port by its Data Type and Port Number, as DataType:PortNumber (Port 1 if the number is omitted).

API operation Request Notes
Initialise the Controller Instance POST /MPAI/AIFU/Controller 201 with {"id": …}. No alternative prefix is returned.
Discover, Upload – Not implemented: the Modules are those the Service is configured with.
Start POST /{cid}/MODULE/Start, body {"module": …} 200 with the Module Instance state.
Status GET /{cid}/MODULE/{mid} controller, id, name, state, change.
Pause, Resume GET /{cid}/MODULE/{mid}/Pause, …/Resume The state is changed; execution is not held.
Send Input Data POST /{cid}/MODULE/{mid}/Input/{pid} The data are checked against the JSON Schema of their Data Type; a mismatch is reported.
Receive Output Data GET /{cid}/MODULE/{mid}/Output/{pid} The first request after new input executes the Module.
Terminate GET /{cid}/MODULE/{mid}/Stop
Delete Controller Instance DELETE /MPAI/AIFU/Controller/{cid}

Failures are reported with the standard HTTP status codes 400, 401, 404, 415 and 500.

4 Extensions

The Reference Software adds the following to what MPAI-MAS V1.0 specifies:

  • An MPAI-Client header, a random identifier made by each RCA when it starts, which lets the Service know which Controller Instances belong to which client.
  • POST /MPAI/AIFU/Leave: the RCA is closing; all its Controller Instances are released.
  • GET /MPAI/AIFU/Status: the number of active clients. An RCA calls it every 30 s; a client silent for 90 s is considered gone and its Controller Instances are released.
  • The App catalogue: GET /MPAI/AIFU/Apps (with search by text and category), /Apps/{id} (the workflow), /Apps/{id}/Icon, /Apps/{id}/Descriptor, /Categories and /Collections.

5 The Apps

App What the user does AI Modules
Multimodal Conversation (MAD) Holds a spoken conversation with the avatar. ASR, EDP, RSR
Multimodal Question Answering (AMQ) Shows a picture and asks a question about it; the avatar answers aloud. ASR, TIQ, RSR
Multimodal Translation (MAT) Speaks in one language and hears the avatar say it in another (English, Italian, Spanish, Portuguese, French, German, Japanese, Chinese). ASR, TTT, RSR
Multimodal Affective Dialogue (MPD) Converses with an avatar that reads words, voice and face and answers with feeling. ASR, NLU, PSE, EDP, RSR
Access Registration (ACR) Registers face and voice, so that Access Control can recognise the person. EFD, ESD, RSR
Access Control (MAC) Shows the face and says a sentence: access is granted if the person is registered. FIR, SIR, IDR, RSR

The RCA itself runs a workflow – MPAI as a Service – that presents the Apps and lets the user choose one.

6 AI Modules and models

The AI Modules are those specified by MPAI-MMC V2.5, MPAI-PAF V1.6 and MPAI-OSD V1.5. Their implementations use the following technologies:

  • Automatic Speech Recognition (ASR): whisper.cpp with the Whisper small model.
  • Entity Dialogue Processing (EDP): the Llama 3.2 3B language model, run by Ollama.
  • Text and Image Query (TIQ): BLIP VQA. Text-to-Text Translation (TTT): M2M100.
  • Personal Status Extraction (PSE): wav2vec2 for speech, HSEmotion for the face.
  • Face identity (FIR, EFD): SCRFD and ArcFace. Speaker identity (SIR, ESD): ECAPA-TDNN.
  • Response and Scene Rendering (RSR): Piper text-to-speech, and the avatar’s face animated from the phonemes of the speech (espeak-ng).

The neural models run with ONNX Runtime. All run on the server; no data leave it.

7 Installation and operation

Linux. The server is distributed as a package with an installation script that obtains the external programs and models (whisper.cpp, Piper, Ollama and the language model), and with system services for Ollama, the MAS Service and the browser client’s host. It needs about 11 GB of disk and 16 GB of memory. In the recommended layout the MAS Service listens on the machine’s loopback interface only, and people reach the browser client’s host over HTTPS on port 443; the host forwards their requests to the Service. The Service and the host may also run on different machines.

Windows. A script publishes the MAS Service and the browser client’s host, which are then started with their configuration.

Use. A person opens the server’s address in a browser, presses Start, and chooses an App. The browser asks permission to use the microphone and, for some Apps, the camera; browsers give it only to pages served over HTTPS, so the host needs a certificate issued for the server’s name.

8 Security and privacy

  • The Service refuses to start on an address other than loopback without a bearer token, or with HTTPS there without a certificate. Requests are authorised with Authorization: Bearer; the Basic and Digest schemes are not implemented.
  • What a session registers – the descriptors of a face and a voice, and a name – is deleted when the session ends: when the page is closed, or after 90 s of silence. The avatar of Access Registration tells the person so.
  • The host of the browser client sends the headers of a public web site and allows the camera and the microphone to its own pages only.

9 Verification

The Reference Software is verified by automated tests, among them:

  • Two clients using the Service at the same time, each receiving the answers it would receive alone, and one client stopping a Module while the other uses it.
  • The response time of typed and spoken conversation turns and of questions on pictures, against a recorded baseline.
  • A person registered by Access Registration and recognised by Access Control, and forgotten when the client leaves.
  • The browser client’s host, published for release and run as a public server.

10 Access to the code

Please send an email to the MPAI Secretariat to access the code.

<- API Go to ToC