Go to CAV-TEC V1.1 Modules

Function Ref. Model I/O Data SubAIMs JSON MData Profiles Ref. Software Conformance Performance

1     Functions

The Human-CAV Interaction (HCI) Subsystem has the function to recognise the human owner or renter, respond to humans’ commands and queries, converse with humans, manifests itself as a perceptible entity, exchange information with the Autonomous Motion Subsystem in response to humans’ requests, and communicate with HCIs on board other CAVs.

The  Human-CAV Interaction (MMC-HCI) AIM:

Receives Point of View User’s Point of View looking at environment.
Audio-Visual Scene Descriptors Audio-Visual representation of the Environment.
Audio Object From Environment
Text Object From User
Visual Object From Environment.
AMS-HCI Message AMS response to HCI request.
Ego-Remote HCI Message Remote HCI to Ego HCI message.
Produces Text Object HCI’s Text.
Speech Object HCT’s avatar Speech.
Audio Object HCI’s avatar or FED Audio.
Visual Object HCI’s avatar or FED Visual.
AMS-HCI Message HCI request to AMS, e.g., Route or Point of View.
Ego-Remote HCI Message Ego HCI to Remote HCI message.

2      Reference Model

Figure 1 depicts the Reference Model of the (CAV-) AIM.

Figure 1 – Reference Model of the Human-CAV Interaction (MMC-HCI) AIM

3      I/O Data

Table 1 specifies the Input and Output Data of the (CAV-) AIM.

Table 1 – I/O Data of the (CAV-) AIM

Input data Description
Point of View Passenger’s Point of View looking at environment.
Audio-Visual Scene Descriptors Audio-Visual representation of the environment.
Audio Object User authentication, command/interaction with HCI, etc. and environment Audio.
Text Object Text complementing/replacing User input
Visual Object Environment perception, User authentication, command/interaction with HCI, etc. and environment Visual.
AMS-HCI Message AMS response to HCI request.
Ego-Remote HCI Message Remote HCI to Ego HCI.
Output data Description
Text Object HCI’s output Text.
Speech Object HCT’s avatar Speech.
Audio Object HCI’s avatar or FED Audio.
Visual Object HCI’s avatar or FED Visual.
AMS-HCI Message HCI request to AMS, e.g., Route or Point of View.
Ego-Remote HCI Message Ego HCI to Remote HCI.

4      SubAIMs

4.1 Reference Model

Figure 2 depicts the Reference Model of Human-CAV (MMC-HCI) Composite AIM.

Figure 2 – Reference Model of Human-CAV (MMC-HCI) Composite AIM

4.2 Operation

The operation of the HCI subsystem is described by the following scenario where a group of humans is approaching the CAV outside the CAV or is sitting inside the CAV:

  1. Audio-Visual Scene Description (AVS) produces:
    1. Speech Scene Descriptors in the form of Speech Objects corresponding to each speaking human in the Environment (outside or inside the CAV).
    2. Visual Scene Descriptors in the form of Descriptors of Faces and Bodies.
    3. All non-Speech Objects are removed from the Speech Scene or signalled in the Audio Scene.
  2. Automatic Speech Recognition (ASR) recognises the speech of each human and produces Recognised Text supporting multiple Speech Objects as input properly identified by their Spatial Attitudes.
  3. Visual Object Identification (VOI) produces Instance IDs of Visual Objects indicated by humans.
  4. Natural Language Understanding (NLU) produces Refined Text and extracts Meaning from the Recognised Text of each Input Speech using the spatial information of Visual Object Identifiers. Refined Text is either the refined Recognised Text from the Automatic Speech Recognition or the direct Input Text, depending on which one is being used. Meaning is always computed based on the Recognised or Input Text, depending on which is available.
  5. Speaker Identity Recognition (SIR) and Face Identity Recognition (FIR) identify the humans the HCI is interacting with. If FIR provides Face IDs corresponding to the Speaker IDs, Entity Dialogue Processing AIM can correctly associate the Speaker IDs (and the corresponding Text) with the Face IDs.
  6. Personal Status Extraction (PSE) extracts the Personal Status of the humans.
  7. Entity Dialogue Processing (EDP)
    1. Communicates with the Autonomous Motion Subsystem of
      1.  The Ego CAV to request to:
        1. Move the CAV to a destination.
        2. Views the Full Environment Descriptors for the passengers’ benefit.
        3. Be informed about CAV’s situation.
        4. Receive relevant information for passengers.
      2. A Remote CAVs to exchange Environment Descriptors.
    2. Produces the Machine Text and Machine Personal Status.
  8. Personal Status Display (PSD) produces the Machine Portable Avatar conveying Machine Speech, Machine Personal Status, and any other information that may be relevant for the the Audio-Visual Rendering AIM .
  9. Audio-Visual Scene Rendering (AVR) renders Audio, and Visual information using Machine Portable Avatar or the Autonomous Motion Subsystem’s Full Environment Descriptors based on the Point of View provided by the human.
  10. Entity Dialogue Processing (EDP)
    1. Requests the AMS subsystem to provide candidate Routes in response to a human requesting to be taken to a destination.
    2. Responses from AMS are processed by EDP and converted to multimodal messages understandable by the human.
    3. Eventually, the human accepts the Route or further elaborates on the EDP response.
    4. May receive messages from Ego AMS or Remote HCI that are processed and converted to multimodal messages understandable by the human.

The HCI interacts with the humans in the cabin in several ways:

  1. By responding to commands/queries from one or more humans at the same time, e.g.:
    1. Commands to go to a waypoint, park at a place, etc.
    2. Commands with an effect in the cabin, e.g., turn off air conditioning, turn on the radio, call a person, open window or door, search for information etc.
  2. By conversing with and responding to questions from one or more humans at the same time about travel-related issues (in-depth domain-specific conversation), e.g.:
    1. Humans request information, e.g., time to destination, route conditions, weather at destination, etc.
    2. CAV offers alternatives to humans, e.g., long but safe way, short but likely to have interruptions.
    3. Humans ask questions about objects in the cabin.
  3. By following the conversation on travel matters held by humans in the cabin if
    1. The passengers allow the HCI to do so, and
    2. The processing is carried out inside the CAV.

4.3 Functions of AI Modules

Table gives the functions of Human-CAV Interaction’s AI Modules.

Table 2 – Functions of Human-CAV Interaction’s AI Modules

AIM Function
Audio-Visual Scene Description 1. Receives Audio and Visual Objects from the appropriate Devices.
2. Produces Audio-Visual Scene Descriptors.
Automatic Speech Recognition 1. Receives Speech Objects.
2. Produces Recognised Text.
Visual Object Identification 1. Receives Visual Scenes Descriptors.
2. Provides Instance ID of indicated Visual Object.
Natural Language Understanding 1. Receives Recognised Text.
2. Uses context information (e.g., Instance ID of object).
3. Produces Natural Language Understanding Text (using Refined or Input) and Meaning.
Speaker Identity Recognition 1. Receives Speech Object of a human and Speech Scene Geometry.
2. Produces Speaker ID.
Personal Status Extraction 1. Receives Speech Object, Meaning, Face Descriptors and Body Descriptors of a human with a Participant ID.
2. Produces the human’s Personal Status.
Face Identity Recognition 1. Receives Face Object of a human and Visual Scene Geometry.
2. Produces Face ID.
Entity Dialogue Processing 1. Receives Speaker ID, Face ID, AV Scene Descriptors, Meaning, Natural Language Understanding Text , Visual Object ID, and Personal Status. Moreover it receives AMS-HCI Messages and Ego-Remote HCI Messages.
2. Produces Machine (HCI) Text Object and Personal Status. Moreover it produces AMS-HCI Messages and Ego-Remote HCI Messages.
Personal Status Display 1. Receives Machine Text Object and Machine Personal Status.
2. Produces Machine’s Portable Avatar.
Audio-Visual Scene Rendering 1. Receives AV Scene Descriptors, Portable Avatar, and Point of View.
2. Produces Output Speech, Output Audio, and Output Visual.

4.4 I/O Data of AI Modules

Table 3 gives the AI Modules of the Human-CAV Interaction depicted in Figure 3.

Table 3 – AI Modules of Human-CAV Interaction AIW

AIM Input Output
Audio-Visual Scene Description – Input Audio
– Input Visual		 – AV Scene Descriptors
Automatic Speech Recognition – Speech Object – Recognised Text
Visual Object Identification – AV Scene Descriptors
– Visual Objects		 – Visual Object Instance ID
Natural Language Understanding – Recognised Text
– AV Scene Descriptors
– Visual Object Instance ID
– Input Text		 – Natural Language Understanding Text
– Meaning
Speaker Identity Recognition – Speech Object
– Speech Scene Geometry		 – Speaker ID
Personal Status Extraction – Meaning
– Input Speech
– Face Descriptors
– Body Descriptors		 – Personal Status
Face Identity Recognition – Face Object
– Visual Scene Geometry		 – Face ID
Entity Dialogue Processing – Ego-Remote HCI Message
– AMS-HCI Message
– Speaker ID
– Meaning
– Natural Language Understanding Text
– Visual Object Instance ID
– Personal Status
– Face ID		 – Ego-Remote HCI Message
– AMS-HCI Message
– Machine Text
– Machine Personal Status
Personal Status Display – Machine Personal Status
– Machine Text		 – Machine Portable Avatar
Audio-Visual Scene Rendering – AV Scene Descriptors
– Machine Portable Avatar
– Point of View		 – Output Text
– Output Speech
– Output Audio
– Output Visual

4.5 AIMs and JSON Metadata

Table 4 provides the links to the AIW and AIM specifications and to the JSON syntaxes. AIMs/1 indicates that the column contains Composite AIMs and AIMs/2 indicates that the column contains Basic and Composite AIMs. AIMs/3 indicates the the column only contains Basic AIMs.

Table 4 – AIMs and JSON Metadata

AIM1 AIM2 Name JSON
MMC-HCI Human-CAV Interaction X
OSD-AVS Audio-Visual Scene Description X
MMC-ASR Automatic Speech Recognition X
OSD-AVA Audio-Visual Alignment X
OSD-VOI Visual Object Identification X
MMC-NLU Natural Language Understanding X
MMC-SIR Speaker Identity Recognition X
MMC-PSE Personal Status Extraction X
MMC-EDP Entity Dialogue Processing X
PAF-FIR Face Identity Recognition X
PAF-PSD Personal Status Display X
PAF-AVR Response and Scene Rendering X

5     JSON Metadata

https://schemas.mpai.community/MMC/V2.5/AIMs/HumanCAVInteraction.json

6 Profiles

No Profiles

7     Reference Software

8     Conformance Testing

9     Performance Assessment

Go to CAV-TEC V1.1 homepage