This is the public page of Mixed-reality Collaborative Spaces (MPAI-MCS), an MPAI standard project supporting scenarios where geographically separated humans represented by avatars collaborate in virtual-reality spaces called Ambients. MCSs require that

  1. A service provider create an Ambient. The Ambient can be a naked space such as the walls of a room, but also a space populated by audio-visual objects.
  2. Multiple independent human participants populate the Ambient with Avatars which may be animated graphic characters or Avatars that are virtual twins of humans represented with as realistic as possible features: especially, face, head, arms, hands and speech. The body is obviously also an element of the virtual twin.

MPAI-MCS seeks to define standard formats for the Ambient and for the Avatar for a service provider to be able to create an MCS populated with avatars with the given features or for a participant client to create its own MCS using data generated by other environments. The standard format can be considered as an interchange format to which any other non-standard formats can be converted.

The standard is intended for an MPAI-specified execution environment called AI Framework (MPAI-AIF) running workflows called AI Workflows composed of processing elements called AI Modules (AIM). More information about the standard can be found here. The standard can be downloaded.

Read An introduction to MPAI’s Mixed-reality Collaborative Spaces project

The Server-based Ambient Use Case is depicted and illustrated in the figures below. It requires

  1. the transmitting client (Figure 1)
  2. the server (Figure 2)
  3. the receiving part of the client (Figure 3).

Clients are used by participants, e.g., to attend a virtual conference, where all participants are represented by an avatar, e.g., sitting at a table. The Avatars faithfully represent the participants with their faces, heads and gestures and made more realistic by the emotion extracted from speech, face and gesture. Each participant can populate the Ambient with the avatars controlled by the participants and can select the viewpoint.

This is not the only possible functionality allocation between client and server. In another configuration, most of the functionalities are supported by the transmitting client and the client architecture is simplified. However, the receiving client need not change.

In this use case the client has only the capability to capture the language preference (the server translate the speech of any participant to the preferred), input text, video the participant’s avatar model. a potential 3D audio-visual object and any actions the participant may like to make on the object.

Audio is captured from an environment that may be noisy. The speech separated from it is forwarded to the server.

Figure 1 – An MCS client (transmitter)
The server is equipped with all the capabilities to required to support the MCS features:

  1. It authenticates the participant by using face and speech information
  2. It translates speech according to language preference
  3. It extracts emotion and meaning from speech, face and gestures generates avatar descriptors
  4. It forwards to all clients the information received from the clients and sends the information generated by the server
Figure 2 – An MCS server
The receiver part of the client executes actions on the AV Object required by the participant, generates the Audio and Visual components of the AV Object acted upon, creates the complete 3D AV scene of the Ambient, including the 3D AV Object and renders the scene in accordance with the selected Viewpoint.
Figure 3 – An MCS client (receiver)

MPAI has developed MPAI-MCS Use Cases and Functional Requirements WD0.5. The goal is to issue a Call for Technologies after the internal process is completed. Anybody is allowed to respond. If a proposed technology is accepted, the proponent will be asked to join MPAI.

MPAI-MCSis at the level of Use Cases and Functional Requirements. If you wish to participate in this work you have the following options

  1. Join MPAI
  2. Participate until the MPAI-MCS Functional Requirements are approved (after that only MPAI members can participate) by sending an email to the MPAI Secretariat.
  3. Keep an eye on this page.