The Data Types are organised by the standards that specify them in categories and the definitions in two columns.

Currently, this page provides access to the Data Types specified by MPAI-CAE,  MPAI-CUI, MPAI-MMC, MPAI-OSD, and MPAI-PAF. Version 2.2 AIWs, AIMs, and Data Types are still not officially approved.

AIF CAE CAV CUI HMC MMC MMM NNW OSD PAF
ARC ARC
TEC TEC

Artificial Intelligence Framework(MPAI-AIF)

Data Types 2.1 2.2 AI Modules 2.1 2.2 AI Workflows 2.1 2.2
Neural Network Model X

Context-based Audio Enhancement (MPAI-CAE)

Data Types 2.1 2.2 AI Modules 2.1 2.2 AI Workflows 2.1 2.2
Audio X Audio Analysis for Preservation X Emotion-Enhanced Speech X X
Audio Basic Scene Descriptors X Audio Analysis Transform X X Audio Recording Preservation X X
Audio Basic Scene Geometry X Audio Basic Scene Description X Speech Restoration System X X
Audio Object X X Audio Description Packaging X Enhanced Audioconference Experience X X
Audio Scene Descriptors X X Audio Descriptors Multiplexing X X
Audio Scene Geometry X X Audio Object Identification X X
Damaged List X X Audio Scene Description X X
Editing List X Audio Separation and Enhancement X X
Enhanced Audio X Audio Source Localisation X X
Input Audio X Audio Synthesis Transform X X
Irregularity File X Emotion Feature Production X
Microphone Array Geometry X X Neural Emotion Insertion X
Multichannel Audio X Noise Cancellation Module X
Multichannel Audio Stream X Packaging for Audio Preservation X
Output Audio X Prosodic Emotion Insertion X
Preservation Files X Sound Field Description X
Transform Audio X Speech Detection and Separation X
Transform Enhanced Audio X Speech Feature Analysis 1 X X
Transform Multichannel Audio X Speech Feature Analysis 2 X X
Speech Model Creation X X
Speech Restoration Assembly X X
Speech Synthesis for Restoration X
Tape Audio Restoration X
Tape Irregularity Classification X
Video Analysis for Preservation X

Connected Autonomous Vehicle (MPAI-CAV) – Technologies (CAV-TEC)

Data Types 1.0 AI Modules 1.0 AI Workflows 1.0
Accelerometer Data X AMS Command Interpretation X Autonomous Motion Subsystem X
Alert X AMS Command Issuance X CAV-to-Everything X
AMS-HCI Message X AMS Decision Recording X Environment Sensing Subsystem X
Basic Environment Descriptors X Basic Environment Description X Human-CAV Interaction X
Brake Command X Full Environment Description X Motion Actuation Subsystem X
Brake Response X LiDAR Scene Description X
CAV Identifier X MAS Response Analysis X
Ego-Remote AMS Message X Motion Selection Planning X
Ego-Remote HCI Message X Online Map Scene Description X
Full Environment Descriptors X Path Selection Planning X
Goal X RADAR Scene Description X
HCI-AMS Message X Route Selection Planning X
Input GNSS X Spatial Attitude Generation X
Input LiDAR X Traffic Obstacle Avoidance X
Input RADAR X Traffic Signalisation Description X
Input Ultrasound X Ultrasound Scene Description X
Lidar Scene Descriptors X
MAS-AMS Message X
Motor Command X
Motor Response X
Odometer Data X
Offline Map Data X
Offline Map Scene Descriptors X
Other Environment Descriptors X
Path X
Pose X
RADAR Scene Descriptors X
Road State X
Route X
Speedometer Data X
Steering Command X
Steering Response X
Traffic Signal Descriptors X
Trajectory X
Ultrasound Scene Descriptors X

Compression and Understanding of Industrial Data (MPAI-CUI)

Data Types 1.1 AI Modules 1.1 AI Workflows 1.1
Business Discontinuity Probability X Discontinuity and Default Prediction X Company Performance Prediction X
Default Probability X Financial Data Assessment X
Financial Features X Governance Data Assessment X
Financial Statement X Prediction Result Perturbation X
Governance X Risk Matrix Generation X
Governance Features X
Organisational Model Index X
Prediction Horizon X
Risk Assessment X
Risk Matrix X

Human and Machine Communication (MPAI-HMC)

Data Types 1.0 1.1 AI Modules 1.0 1.1 AI Workflows 1.0 1.1
CEC Profiles X AV Scene Integration and Description X X Communicating Entities in Context X X
Entity and Context Understanding X X

Multimodal Conversation (MPAI-MMC)

Data Types 2.1 2.2 AI Modules 2.1 2.2 AI Workflows 2.1 2.2
Cognitive State X X Answer to Question Module X X Bidirectional Speech Translation X
Emotion X X Audio Segmentation X Conversation About a Scene X
Intention X X Automatic Speech Recognition X X Conversation with Emotion X
Language Identifier X Entity Dialogue Processing X X Conversation with Personal Status X
Meaning X X Entity Speech Description X X Human-CAV Interaction X
Personal Status X X Entity Text Description X X Multimodal Question Answering X X
Recognised Text X Input Speech Description X One-to-Many Speech Translation X
Refined Text X Question Analysis Module X X Text and Speech Translation X
Social Attitude X X Input Text Description X Unidirectional Speech Translation X X
Speech X Multimodal Emotion Fusion X X Virtual Meeting Secretary X X
Speech Descriptors X X Natural Language Understanding X X
Speech Object X Personal Status Extraction X X
Speech Overlap X Personal Status Demultiplexing X
Speech Basic Scene Descriptors X Personal Status Multiplexing X X
Speech Basic Scene Geometry X PS-Speech Interpretation X X
Speech Scene Descriptors X PS-Text Interpretation X X
Speech Scene Geometry X Speech Basic Scene Description X
Summary X X Speech Scene Description X
Text X Speaker Identity Recognition X X
Text Descriptors X  X Summary Creation Module X X
Text Object X Text and Image Query X
Text and Speech Translation X X
Text-To-Speech X X
Text-to-Text Translation X X
Video Lip Animation X X

MPAI Metaverse Model – Technologies

Data Types 1.0
Account X
Activity Data X
Amount X
Asset X
Basic M-Location X
Basic U-Location X
Contract X
Currency X
Discover X
E-Capabilities X
Identifier X
Inform X
Interpret X
M-Capabilities X
M-Environment X
Message X
M-Instance X
M-Location X
P-Capabilities X
Personal Data X
Personal Profile X
Process Action X
Program X
Provenance X
Request-Action X
Response-Action X
Rights X
Rules X
Stream X
Transaction X
U-Environment X
U-Location X
Universe-Metaverse Map X
Value X
Wallet X

Object and Scene Description (MPAI-OSD)

Data Types 1.0 1.1 AI Modules 1.0 1.1 AI Workflows 1.0 1.1
Anchored Direction X Audio-Visual Alignment X X Television Media Analysis X
Audio-Visual Basic Scene Descriptors X X Audio-Visual Event Description X  
Audio-Visual Basic Scene Geometry X Audio-Visual Scene Demultiplexing X X
Audio-Visual Event Descriptors X Audio-Visual Basic Scene Description X
Audio-Visual Object X X Audio-Visual Scene Description X X
Audio-Visual Scene Descriptors X X Audio-Visual Scene Multiplexing X
Audio-Visual Scene Geometry X X Television Splitting X
Bounding Box X Visual Change Detection X
Input Visual X Visual Direction Identification X X
Instance Identifier X X Visual Instance Identification X X
Orientation X Visual Object Extraction X X
Output Visual X Visual Object Identification X X
Point of View X X Visual Basic Scene Description X
Position X Visual Scene Description X X
Selector X X
Space-Time X
Spatial Attitude X X
Time X X
Visual Basic Scene Descriptors X
Visual Basic Scene Geometry X
Visual Object X X
Visual Scene Descriptors X X
Visual Scene Geometry X X

Portable Avatar Format (MPAI-PAF)

Data Types 1.1 1.2 AI Modules 1.1 1.2 AI Workflows 1.1 1.2
Avatar X X Visual Scene Creation X Avatar Videoconference Server X X
Avatar Model X Audio-Visual Scene Rendering X X Videoconference Client Receiver X X
Body Descriptors X X Face Identity Recognition X X Videoconference Client Transmitter X X
Body Object X Entity Body Description X
Face Descriptors X X Entity Face Description X
Face Object X Input Body Description X
Gesture Descriptors X Input Face Description X
Model X PS-Face Interpretation X X
Portable Avatar X X PS-Gesture Interpretation X X
Portable Avatar Demultiplexing X X
Portable Avatar Multiplexing X X
Personal Status Display X X
Service Participant Authentication X
Visual Scene Creation X