Go to MPAI-MMC V2.5 AI Modules
Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance
1 Functions
The Text and Image Query (MMC‑TIQ) AIM receives an input text and an input image and produces an output text that is a response to the inputs:
| Receives | Text Object | Textual part of query. |
| Image Visual Object | Image part of query. | |
| Produces | Text Object | In response to Text and Image provided as input. |
2 Reference Model
Figure 1 depicts the Reference Model of the Text and Image Query (MMC‑TIQ) AIM.
Figure 1 – The Text and Image Query (MMC‑TIQ) AIM
3 I/O Data
Table 1 specifies the Input and Output Data of the Text and Image Query (MMC‑TIQ) AIM.
| Input | Description |
|---|---|
| Text Object | Text asking a question about the Image. |
| Visual Object | Image about which a question is asked. |
| Output | Description |
| Text Object | Response produced by Text and Image Query. |
4 SubAIMs
No SubAIMs.
5 JSON Metadata
https://schemas.mpai.community/MMC/V2.5/AIMs/TextAndImageQuery.json
6 Profiles
No Profiles.
7 Reference Software
7.1 Disclaimers
- This MMC‑TIQ Reference Software Implementation is released with the BSD-3-Clause licence.
- The purpose of this MMC‑TIQ Reference Software is to provide a working Implementation of MMC‑TIQ, not to provide a ready-to-use product.
- MPAI disclaims the suitability of the Software for any other purposes and does not guarantee that it is secure.
- Use of this Reference Software may require acceptance of licences from the respective repositories. Users shall verify that they have the right to use any third-party software required by this Reference Software.
7.2 Guide to the MMC‑TIQ code
Use of this AI Module is for developers who are familiar with Python and downloading models from HuggingFace.
A wrapper for the BLIP NN Module:
- Manages input files and parameters: Text Object, Visual Object.
- Executes the BLIP Module to perform question answering on each individual pair of Text and Visual Object.
- Outputs Text Object as answer.
The MMC‑TIQ Reference Software is found at the MPAI gitlab site. It contains:
- The Python code implementing the AIM.
- Required libraries are: pytorch, transformers (HuggingFace), and Pillow.
7.3 Acknowledgements
This version of the MMC‑TIQ Reference Software has been developed by the MPAI Neural Network Watermarking Development Committee (NNW‑DC).
8 Conformance Testing
Table 2 provides the Conformance Testing Method for the Text and Image Query (MMC‑TIQ) AIM.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.
| Receives | Text Object | Shall validate against Text Object schema. Text Data shall conform with Text Qualifier. |
| Image Visual Object | Shall validate against Visual Object schema. Visual Data shall conform with Visual Qualifier. | |
| Produces | Text Object | Shall validate against Text Object schema. Text Data shall conform with Text Qualifier. |
9 Performance Assessment
Not part of this specification.
