Go to PGM-AUA V1.0 AI Modules

Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Visual Environment Analysis (PGM‑VEA) AI Module derives a Visual Profile, the MPAI-CVE Data Type that says how the Visual Objects and the scene look, from Visual Scene Descriptors, and infers the possible interactions offered by the Visual Objects the scene contains, by

  • Estimating depth and the spatial layout of the scene (Data Types specified by OSD)
  • Determining occlusion relations and visibility between Visual Objects (Data Types specified by OSD)
  • Characterising illumination and shadowing, which the Visual Profile carries, and the contrast conditions affecting visual interpretation (specified by OSD)
  • Inferring Affordance Tags and Interaction Potential from object properties, scene geometry, and domain knowledge
  • Producing a new Visual Profile representation of the scene

The module is analytical and generative, producing environmental descriptors and interaction annotations rather than modifying existing ones.

Affordances are inferred from the visual modality alone: the possible actions an object offers derive from its observable geometry, material properties, and spatial relations, which the audio modality does not convey. The Acoustic Environment Analysis (PGM‑AEA) AIM, the counterpart of this AIM in Audio Scene Enhancement, therefore characterises the acoustic environment only and has no affordance counterpart.

2 Reference Model

Figure 1 depicts the Reference Model of the Visual Environment Analysis (PGM‑VEA) AIM.

Figure 1 – The Visual Environment Analysis (PGM‑VEA) AIM

The Visual Environment Analysis (PGM‑VEA) AI Module operates as follows:

  1. Receives the Visual Scene Descriptors
  2. Extracts relevant descriptors related to scene structure and visual object characteristics
  3. Performs environmental inference based on:
    • Spatial distribution and geometry of Visual Objects
    • Temporal persistence and interaction patterns
    • Implicit geometric and photometric cues contained in the description
  4. Estimates:
    • Depth and relative distance of Visual Objects (specified by OSD)
    • Occlusion relations and visibility (specified by OSD)
    • Illumination and shadowing (carried by the Visual Profile), and contrast conditions (specified by OSD)
    • Surface and material properties affecting appearance
  5. Infers, for each Visual Object:
    • Affordance Tags, i.e., the actions the object offers on the basis of its physical properties
    • Interaction Potential, i.e., the feasibility of those actions under the conditions of the current scene
  6. Constructs a consistent Visual Profile (MPAI-CVE V1.0)
  7. Outputs the Visual Profile, the Affordance Tags, the Interaction Potential, the Depth and the Occlusion Flag for use by downstream AI Modules. The contrast conditions have no Data Type yet and are to be specified by OSD

The module does not require pre-existing Visual Profiles and produces a scene-level environmental characterisation.

3 I/O Data

Table 1 specifies the Input and Output Data of the Visual Environment Analysis (PGM‑VEA) AIM.

Table 1 – I/O Data of the Visual Environment Analysis (PGM‑VEA) AIM

Input Description
Visual Scene Descriptors Description of the visual scene from which the environmental visual characteristics and the interactions offered by its Visual Objects are inferred.
Visual CXT Request Control directive specifying scope, depth, or policy constraints for visual environment analysis.
Visual Domain Response Domain‑specific knowledge supporting environmental visual inference and affordance determination.
Visual Interaction History Response Prior interaction‑history content supporting temporal consistency across captures.
Output Description
Visual Profile How the Visual Objects and the scene look, as specified by MPAI-CVE: for an Object its colour, metalness, roughness, transparency and emission; for the scene its light sources, ambient light and atmosphere. Depth, relative distance, occlusion, visibility and contrast are not part of the Visual Profile: they are specified by OSD.
Affordance Tags The actions each Visual Object offers on the basis of its physical properties.
Interaction Potential The feasibility of the actions expressed by the Affordance Tags under the conditions of the current scene.
Depth The depth and the relative distance of the Visual Objects, as specified by OSD.
Occlusion Flag The occlusion relations and the visibility of the Visual Objects, as specified by OSD.
Visual CXT Response Response describing the execution and outcome of Visual Environment Analysis processing.
Visual Domain Request Query to Domain Access for domain‑specific knowledge.
Visual Interaction History Request Query, through the Controller, to Private Storage for prior Interaction History content.

4 SubAIMs

No SubAIMs.

5 JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/VisualEnvironmentAnalysis.json

6 Profiles

No Profiles.

7 Reference Software

Not part of this specification.

8 Conformance Testing

Table 2 provides the Conformance Testing Method for the Visual Environment Analysis (PGM‑VEA) AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the Visual Environment Analysis (PGM‑VEA) AIM

Receives Visual Scene Descriptors Shall validate against Visual Scene Descriptors schema.
Visual CXT Request Shall validate against Visual CXT Request schema.
Visual Domain Response Shall validate against Visual Domain schema.
Visual Interaction History Response Shall validate against Interaction History schema.
Produces Visual Profile Shall validate against Visual Profile schema.
Affordance Tags Shall validate against Affordance Tags schema.
Interaction Potential Shall validate against Interaction Potential schema.
Depth Shall validate against the Depth schema of OSD.
Occlusion Flag Shall validate against the Occlusion Flag schema of OSD.
Visual CXT Response Shall validate against Visual CXT Response schema.
Visual Domain Request Shall validate against Visual Domain schema.
Visual Interaction History Request Shall validate against Interaction History schema.

9 Performance Assessment

Not part of this specification.

Go to PGM-AUA V1.0 AI Modules