Go to CAV-TEC V2.0 AI Modules

Functions
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Environment Sensing Subsystem (CAV-ESS) AIM:

Receives Audio Field What the Audio sensors capture.
Visual Field What the Visual sensors capture.
RADAR Field What the RADAR sensors capture.
LiDAR Field What the LiDAR sensors capture.
Ultrasound Field What the Ultrasound sensors capture.
Basic Offline Map Object From the Autonomous Motion Subsystem.
GNSS Object From the GNSS receiver.
Spatial Attitude From the Motion Actuation Subsystem.
Weather Data From the Motion Actuation Subsystem.
Full Environment Descriptors From the Autonomous Motion Subsystem.
Produces Basic Environment Descriptors To the Autonomous Motion Subsystem.
Alert To the Autonomous Motion Subsystem.

The AIM senses the environment of the CAV with its Audio, Visual, RADAR, LiDAR and Ultrasound sensors, describes what each technology senses as Basic Scene Descriptors on the ego frame given by Spatial Attitude Generation, converts the Offline Map into Basic Scene Descriptors of the same kind, and fuses them all into the Basic Environment Descriptors it gives the Autonomous Motion Subsystem, with the Alerts that call for an immediate reaction.

The reference software implements the visual technology end to end: Visual Sensors, Basic Visual Scene Description, Spatial Attitude Generation and Basic Environment Description.

2 Reference Model

Figure 1 depicts the Reference Model of the Environment Sensing Subsystem (CAV-ESS) AIM.

Figure 1 – The Environment Sensing Subsystem (CAV-ESS) AIM

Figure 1 – The Environment Sensing Subsystem (CAV-ESS) AIM

3 I/O Data

Table 1 specifies the Input and Output Data of the Environment Sensing Subsystem (CAV-ESS) AIM.

Table 1 – I/O Data of the Environment Sensing Subsystem (CAV-ESS) AIM

Input Data Type Description
Audio Field (optional) Basic Audio Object The Audio Field.
Visual Field (optional) Basic Visual Object The Visual Field.
RADAR Field (optional) Basic RADAR Object The RADAR Field.
LiDAR Field (optional) Basic LiDAR Object The LiDAR Field.
Ultrasound Field (optional) Basic Ultrasound Object The Ultrasound Field.
Basic Offline Map Object (optional) Basic Offline Map Object The Offline Map.
GNSS Object (optional) GNSS Object The CAV’s position as GNSS gives it.
Spatial Attitude Spatial Attitude The CAV’s Spatial Attitude by dead reckoning.
Weather Data (optional) Weather Data The weather.
Full Environment Descriptors (optional) Full Environment Descriptors The environment as the AMS has placed it on the map.
Output Data Type Description
Basic Environment Descriptors Basic Environment Descriptors The objects around the CAV, fused from every technology, on the ego frame.
Alert (optional) Alert Something that calls for an immediate reaction.

4 SubAIMs

4.1 Functions of SubAIMs

Table 2 specifies the Functions of the SubAIMs of the Environment Sensing Subsystem (CAV-ESS) AIM.

Table 2 – Functions of the SubAIMs

SubAIM Function
Audio Sensors Captures the Audio Field as Basic Audio Objects, one Acquisition AIM per sensor.
Visual Sensors Captures the Visual Field as Basic Visual Objects, one Acquisition AIM per sensor.
RADAR Sensors Captures the RADAR Field as Basic RADAR Objects, one Acquisition AIM per sensor.
LiDAR Sensors Captures the LiDAR Field as Basic LiDAR Objects, one Acquisition AIM per sensor.
Ultrasound Sensors Captures the Ultrasound Field as Basic Ultrasound Objects, one Acquisition AIM per sensor.
Basic Audio Scene Description Describes what the Audio sensors capture as Basic Audio Scene Descriptors on the ego frame; raises Alerts.
Basic Visual Scene Description Describes what the Visual sensors capture as Basic Visual Scene Descriptors on the ego frame; raises Alerts.
Basic RADAR Scene Description Describes what the RADAR sensors capture as Basic RADAR Scene Descriptors on the ego frame; raises Alerts.
Basic LiDAR Scene Description Describes what the LiDAR sensors capture as Basic LiDAR Scene Descriptors on the ego frame; raises Alerts.
Basic Ultrasound Scene Description Describes what the Ultrasound sensors capture as Basic Ultrasound Scene Descriptors on the ego frame; raises Alerts.
Basic Offline Map Scene Description Converts the Offline Map into Basic Scene Descriptors of the same kind.
Spatial Attitude Generation The ego Spatial Attitude on the frame of the Offline Map, from the MAS’s Spatial Attitude and GNSS.
Basic Environment Description The Basic Scene Descriptors of every technology fused into the Basic Environment Descriptors.

4.2 Operation

Each sensor’s Basic Object goes to the Basic Scene Description of its technology, which describes it on the ego frame – Spatial Attitude Generation gives the ego Spatial Attitude, anchored to the Offline Map by GNSS – with the Basic Environment Descriptors of the previous instant as its prior, and raises Alerts.

Basic Environment Description fuses the Basic Scene Descriptors of every technology and of the Offline Map into the Basic Environment Descriptors, tracking each object from one instant to the next, and gives them to the Autonomous Motion Subsystem; the Alerts go to it directly.

4.3 I/O Data of SubAIMs

Table 3 specifies the Input and Output Data of the SubAIMs.

Table 3 – I/O Data of the SubAIMs

SubAIM Input Output
Audio Sensors Audio Field – Basic Audio Object Basic Audio Object – Basic Audio Object
Visual Sensors Visual Field – Basic Visual Object Basic Visual Object – Basic Visual Object
RADAR Sensors RADAR Field – Basic RADAR Object Basic RADAR Object – Basic RADAR Object
LiDAR Sensors LiDAR Field – Basic LiDAR Object Basic LiDAR Object – Basic LiDAR Object
Ultrasound Sensors Ultrasound Field – Basic Ultrasound Object Basic Ultrasound Object – Basic Ultrasound Object
Basic Audio Scene Description Basic Audio Object – Basic Audio Object
Spatial Attitude – Spatial Attitude
Basic Environment Descriptors – Basic Environment Descriptors		 Basic Audio Scene Descriptors – Basic Audio Scene Descriptors
Alert – Alert
Basic Visual Scene Description Basic Visual Object – Basic Visual Object
Spatial Attitude – Spatial Attitude
Basic Environment Descriptors – Basic Environment Descriptors		 Basic Visual Scene Descriptors – Basic Visual Scene Descriptors
Alert – Alert
Basic RADAR Scene Description Basic RADAR Object – Basic RADAR Object
Spatial Attitude – Spatial Attitude
Basic Environment Descriptors – Basic Environment Descriptors		 Basic RADAR Scene Descriptors – Basic RADAR Scene Descriptors
Alert – Alert
Basic LiDAR Scene Description Basic LiDAR Object – Basic LiDAR Object
Spatial Attitude – Spatial Attitude
Basic Environment Descriptors – Basic Environment Descriptors		 Basic LiDAR Scene Descriptors – Basic LiDAR Scene Descriptors
Alert – Alert
Basic Ultrasound Scene Description Basic Ultrasound Object – Basic Ultrasound Object
Spatial Attitude – Spatial Attitude
Basic Environment Descriptors – Basic Environment Descriptors		 Basic Ultrasound Scene Descriptors – Basic Ultrasound Scene Descriptors
Alert – Alert
Basic Offline Map Scene Description Basic Offline Map Object – Basic Offline Map Object
Spatial Attitude – Spatial Attitude		 Basic Offline Map Scene Descriptors – Basic Offline Map Scene Descriptors
Spatial Attitude Generation GNSS Object – GNSS Object
Spatial Attitude – Spatial Attitude
Basic Offline Map Object – Basic Offline Map Object		 Ego Spatial Attitude – Spatial Attitude
Basic Environment Description Basic Audio Scene Descriptors – Basic Audio Scene Descriptors
Basic Visual Scene Descriptors – Basic Visual Scene Descriptors
Basic RADAR Scene Descriptors – Basic RADAR Scene Descriptors
Basic LiDAR Scene Descriptors – Basic LiDAR Scene Descriptors
Basic Ultrasound Scene Descriptors – Basic Ultrasound Scene Descriptors
Basic Offline Map Scene Descriptors – Basic Offline Map Scene Descriptors
Spatial Attitude – Spatial Attitude
Weather Data – Weather Data
Full Environment Descriptors – Full Environment Descriptors		 Basic Environment Descriptors – Basic Environment Descriptors

4.4 AIMs and JSON Metadata

Table 4 gives the AIMs and their JSON Metadata.

Table 4 – AIMs and JSON Metadata

AIM Name JSON
CAV-ESS Environment Sensing Subsystem EnvironmentSensingSubsystem.json
CAV-AUS Audio Sensors AudioSensors.json
CAV-VIS Visual Sensors VisualSensors.json
CAV-RAS RADAR Sensors RADARSensors.json
CAV-LIS LiDAR Sensors LiDARSensors.json
CAV-ULS Ultrasound Sensors UltrasoundSensors.json
OSD-BAS Basic Audio Scene Description BasicAudioSceneDescription.json
OSD-BVS Basic Visual Scene Description BasicVisualSceneDescription.json
OSD-BRS Basic RADAR Scene Description BasicRADARSceneDescription.json
OSD-BLS Basic LiDAR Scene Description BasicLiDARSceneDescription.json
OSD-BUS Basic Ultrasound Scene Description BasicUltrasoundSceneDescription.json
OSD-BOS Basic Offline Map Scene Description BasicOfflineMapSceneDescription.json
CAV-SAG Spatial Attitude Generation SpatialAttitudeGeneration.json
CAV-BED Basic Environment Description BasicEnvironmentDescription.json

5 JSON Metadata

https://schemas.mpai.community/CAV2/V2.0/AIMs/EnvironmentSensingSubsystem.json

6 Profiles

No Profiles.

7 Reference Software

The reference software implements the visual technology end to end: the camera frames of the simulation, Basic Visual Scene Description with the YOLOX detector, Spatial Attitude Generation and Basic Environment Description, run continuously. The other technologies are not implemented in Stage 1. It is part of the reference software of CAV-TEC V2.0 (see N4195, An introduction to the reference software): C# on .NET 10, released under the BSD 3-Clause licence, obtained from MPAI.

Tested on synthetic drives, the detector included: objects and Alerts against the ground truth.

8 Conformance Testing

Table 5 provides the Conformance Testing Method for the Environment Sensing Subsystem (CAV-ESS) AIM: the data it receives and produces. If a schema references other schemas, conformance of data for the primary schema implies that data referencing a secondary schema shall also validate against that schema, if present, and conform with the Qualifier, if present.

Table 5 – Conformance Testing Method for the Environment Sensing Subsystem (CAV-ESS) AIM

Receives Basic Audio Object Shall validate against the Basic Audio Object schema.
Basic Visual Object Shall validate against the Basic Visual Object schema.
Basic RADAR Object Shall validate against the Basic RADAR Object schema.
Basic LiDAR Object Shall validate against the Basic LiDAR Object schema.
Basic Ultrasound Object Shall validate against the Basic Ultrasound Object schema.
Basic Offline Map Object Shall validate against the Basic Offline Map Object schema.
GNSS Object Shall validate against the GNSS Object schema.
Spatial Attitude Shall validate against the Spatial Attitude schema.
Weather Data Shall validate against the Weather Data schema.
Full Environment Descriptors Shall validate against the Full Environment Descriptors schema.
Produces Basic Environment Descriptors Shall validate against the Basic Environment Descriptors schema.
Alert Shall validate against the Alert schema.

9 Performance Assessment