1 Definition 2 Functional Requirements 3 Syntax 4 Semantics

1 Definition

A Behavioural Signal Qualifier specifies metadata describing a Behavioural Signal Data instance. It does not contain signal samples; instead, it provides information about:

  • the Sub‑Type of the behavioural signal (e.g., gait trajectory, hand gesture sequence, facial action units, touchscreen interaction trace),
  • the Format used to encode the signal (e.g., JSON event stream, CSV time‑series, motion‑capture binary format, interaction log format), and

The Behavioural Signal Qualifier is used in conjunction with Behavioural Signal Data and is aggregated by the Behavioural Signal Object defined in MPAI‑AIH/HSP.

The combination of Behavioural Signal Data and Behavioural Signal Qualifier is called Behavioural Signal Object, specified by AIH-HSP V1.1.

2 Functional Requirements

  • Behavioural Signal Sub‑Type Identification The Behavioural Signal Qualifier shall specify the Sub‑Type of the Behavioural Signal Data.
  • Behavioural Signal Format Identification The Behavioural Signal Qualifier shall specify the Format used to encode the Behavioural Signal Data.
  • Separation from Data The Behavioural Signal Qualifier shall not contain intrinsic behavioural signal samples; it shall only describe the Behavioural Signal Data.
  • Interpretability of Data The Behavioural Signal Qualifier shall provide sufficient information to interpret the Behavioural Signal Data.

Users needing additional entries in the Physiological Signal Qualifier or support of new Qualifiers should make a documented request to the MPAI Secretariat. Requests will be considered by the appropriate MPAI committee.

An instance of the Qualifier states ONE Sub-Type of the data (SubType), from the Sub-Types listed in Semantics, and may state the group it belongs to (SubTypeGroup); and ONE Format (Format), from the Formats listed, and may state its group (FormatGroup). The mapping between Sub-Type groups and Format groups (GroupMap) is part of the specification of the Qualifier and is optional in an instance.

The Qualifier may include Findings: what the data show. Each Finding has an identifier (FindingID), a Code, the Coding System of the Code, chosen among those of this domain, the name of the label set when the Code is from a data set’s own labels (LabelSet), and where in the data the Finding is (Where). Who set a Finding and how sure that source is are not part of the Qualifier: they are in the AIH Data that carries the data (FindingTrust).

3 Syntax

https://schemas.mpai.community/TFA/V1.5/data/BehaviouralSignalQualifier.json

4 Semantics

  1. Sub‑Types
    • Motor Behaviour
      • Gait
      • Upper‑limb movement
      • Whole‑body movement
      • Fine motor actions
    • Gesture Behaviour
      • Hand gestures
      • Arm gestures
      • Body gestures
    • Facial Behaviour
      • Facial action units
      • Facial expressions
      • Eye region movements (non‑EOG)
    • Interaction Behaviour
      • Touchscreen interactions
      • Keyboard interactions
      • Mouse interactions
      • Controller interactions
    • Speech‑Related Behaviour
      • Speech activity (voice activity detection)
      • Prosodic behaviour (pitch, intensity, rhythm)
      • Articulatory movement traces (non‑audio)
  1. Formats
    • Motor Behaviour Formats
      • CSV / TSV (motion time‑series)
      • JSON (motion event stream)
      • BVH (Biovision Hierarchy)
      • C3D (3D motion capture format)
      • HDF5 (structured motion datasets)
    • Gesture Behaviour Formats
      • CSV / TSV
      • JSON
      • BVH
      • C3D
      • HDF5
    • Facial Behaviour Formats
      • CSV / TSV (facial landmark time‑series)
      • JSON (facial action event stream)
      • HDF5
      • C3D (3D facial motion capture)
      • BVH (hierarchical facial motion)
    • Interaction Behaviour Formats
      • CSV / TSV (interaction logs)
      • JSON (event‑based interaction logs)
      • XML (interaction trace format)
      • HDF5
    • Speech‑Related Behaviour Formats
      • CSV / TSV (prosodic features)
      • JSON (speech activity events)
      • HDF5
      • RTTM (Rich Transcription Time Marked – activity only)
      • TextGrid (timing annotations)

 

Findings

Label Description
Findings What the data show, each Finding coded in a vocabulary of this domain: the labels a Neural Network Model learns from and is checked against.
FindingID Identifier of the Finding; the FindingTrust of AIH Data refers to it.
Code The Finding, as the Coding System writes it.
CodingSystem One of: SNOMEDCT, HED, DatasetLabelSet.
LabelSet The name of the label set when the CodingSystem is DatasetLabelSet (a data set’s own labels, e.g. NIH ChestX-ray14).
Where Where in the data the Finding is: Start, End (seconds from the start of the recording).

5 References