<-Communication Go to ToC Metadata ->

1 Reference Model

Figure 1 depicts the Reference Model of Storage.

Figure 1 - Reference Model of Storage

Figure 1 – Reference Model of Storage

Storage is Communication across time: a datum written by one AIM and read later by another, or by the same AIM, is carried by Storage as a Message is carried by a Channel. Storage is governed as Channels are: the Controller binds every handle, and the writer does not choose what the record says about the write. Table 1 lists the kinds of Storage.

Table 1 – Kinds of Storage

Storage Holds
Private Storage of an AIM Data reachable by that AIM Instance alone.
Private Storage of the Module The data of the Module, reached by its AIMs, by the Controller on its behalf and by the User Agent that drives it, each as the rules of the data allow. It includes the record of the boundary (5).
Shared Storage Data shared by the Modules to which the User Agent gives one location – one Module, or several.

2 Reaching Storage

Storage is direct. The Controller binds each AIM’s Storage handles to the identity of that AIM when it instantiates it, and the AIM then reaches its Storage with no interface to cross. The User Agent, outside the Module, reaches Storage through the Controller API: it initialises a Shared Storage scope, stating its location – one location for several Modules to share it – and naming, where there is one, the Module whose central control governs it. The User Agent says where a scope is; it cannot say what identity a write will carry. The location is the User’s decision, and a Workflow states it with ask Controller to keep storage in.

Each write is stamped with the Module, the AIM within it and the time on the Controller’s time base. A Put is atomic per key; ordering between writers is not guaranteed.

A Shared Storage belongs to one Controller and is shared by the AIMs of its Module, including those that execute apart from the Controller, through handles the Controller binds. AIMs of Modules run by other Controllers do not access it: data move between Controllers through External Ports (Communication). Encryption of stored data is a matter of the Implementation.

The AIM repository of the Controller is not a Storage: AIMs are code and Metadata, not data, and none of the Storages above holds them.

3 Rules

Each datum of the Private Storage of the Module, of Shared Storage and of Access is written under rules (Table 2; for Access, 6.6).

Table 2 – Rules of a datum

Rule Meaning
Category The kind of datum, to which the general rules of the central control, if any, apply.
Readers Who may read it: Modules, AIM Instances, the User Agent.
Time How long it is kept: until the Module instance stops; until the session of the User Agent ends; as long as the scope is kept; or within a duration.
Trace Its writer, its stamp, its category and the rule under which it was written. The Trace is written by the Controller, not by the writer.

Without a central control, the writer sets the rules of its datum. In the Private Storage of the Module, a datum for which no reader is named is read by its writer alone; in Shared Storage, by every Module given the location.

With a central control – the AIM the Module’s Metadata names as StorageControl or, for a Shared Storage, the StorageControl of the Module the User Agent names when it initialises the scope – the general rules of each category are those of the central control: who may write, who may read, the longest time. A writer may restrict them for its datum and may not extend them: a write that tries is refused. A restriction applies at once to what was written. The central control reads everything. The central control is an AIM, never the User Agent.

A call the rules do not allow returns NOT_AUTHORISED.

For example, the A-User Storage of MPAI-PGM Autonomous User Architecture requires exactly this: A-User Control, the central control, authorises writers, categories, persistence and readers, and reads everything.

4 Memories

A Memory with strong semantics of its own – AMS Memory in the Autonomous Motion Subsystem of MPAI-CAV, for instance – is an AIM of its Module. It uses Storage and its rules like any other AIM, and its semantics are specified by the Standard that specifies it, not by MPAI-AIF.

5 The record of the boundary

The Controller records what crosses the boundary of a Module – each Message written to a boundary Input Port, each Message a boundary Output Port receives – with its stamp, in the Private Storage of the Module. Its writer is the Controller; its reader is the User Agent, which cannot write into it.

The Controller records at the User Agent’s request, or from Start to Stop where the Module’s Metadata declares Record Always, whatever the User Agent does: a black box exists because the Module requires it. A record never holds the Module back: what cannot be written in time is counted per Port, and the User Agent is told in the Module’s status and at the end of the record. The form of a record is left to the Implementation.

One mechanism serves in operation – what the Module was given and what it did – and in development: the recorded data on which a Module is tested are played back through the User Agent at their original rate (Workflow Description).

6 Access

Access provides the static or slowly changing data a Module needs – domain knowledge, data models, knowledge bases – through MPAI-specified interfaces.

Access differs from Shared Storage in who writes it. The AIMs of a Module write Shared Storage as they run; they only read Access. Its content is put there by whoever holds it: the Controller, which obtains the data the Metadata of an AIM names from the URI the Metadata gives; a third party that holds the rights to the data and wants to be its only provider; or the User who provides it. MPAI hosts none of this data and the MPAI Store holds only Metadata: the data is published where its owner chooses, the Metadata states where, and whoever deploys the Module downloads it from there and keeps it in Access.

6.1 Sources

Access content is organised in Sources. A Source is a set of items, each identified by a Key, and has a Version that changes whenever its content changes. MPAI does not classify what a Source holds: data is identified by the URI it is obtained from, as it was published, and pinned by its SHA-256 (6.5). Each Source has one Writer: the Controller, for the data it obtains from the Metadata, or the Provider that created it. The Controller lets only the Writer write a Source.

6.2 Reading

The AIMs of a Module read Access through the Controller:

  • MPAI_AIFM_Access_Get(Source, Key) returns an item.
  • MPAI_AIFM_Access_List(Source, Prefix) returns the Keys of a Source that begin with Prefix.
  • MPAI_AIFM_Access_Version(Source) returns the Version of a Source, so that an AIM can tell that its content has changed.
  • MPAI_AIFM_Access_Open(Source, Key) opens an item for reading as a stream, for data – a model, a database – too large to be held in memory.
  • MPAI_AIFM_Access_Path(Source, Key) returns the file where an item is kept, for a program that reads files and cannot be given a stream – a speech engine, a runtime. The file is read, not written; where the Access is not on the same machine as the AIM, it is a copy.
  • MPAI_AIFM_Access_Sources(Name) returns the URIs of the Sources that the AIM declared in its Metadata under that Name (all of them when the Name is empty).

There is no Module API to write Access.

An AIM declares in its Metadata the Access it needs (AccessSources, Metadata): data by URI, with the Keys where it says so. The Source of a file has the URI of the file for its name and the name of the file for its Key (an empty Key gives it); the Source of a set of files has the URI of the set, which ends in a slash, and the paths below it for Keys. Whether the AIM may read a datum is decided by the rules of the datum (6.6): a read they do not allow returns NOT_AUTHORISED.

6.3 Updates by a Provider

When the data belongs to a third party, its updates do not pass through the User Agent: the Provider writes Access directly, through the Provider API.

  • MPAI_AIFP_Access_Create(Source) creates a Source, whose Writer the Provider becomes.
  • MPAI_AIFP_Access_Put(Source, Version, Key, Data, Category, Readers) writes an item, under the rules it states (6.6),, and MPAI_AIFP_Access_Delete(Source, Version, Key) removes one; Version is the new Version of the Source.

The functions are specified in Basic API (4.11.3 and 6). The Controller protects a Source against anyone but its Provider, and the Provider against replay:

  • The Provider opens its link to the Controller with the Trust Protocol over TLS, presenting a credential issued under a Trust Anchor. The Controller admits the link only if the credential is valid and is the one that created the Source.
  • Each item carries Data Exchange Metadata stating the SHA-256 hash of the data, the KeyID of the Provider and its signature. The Controller stores the item only if the KeyID is that of the Writer of the Source, the key is trusted, the hash matches and the signature verifies.
  • Each update states the new Version of the Source. The Controller refuses an update whose Version is not higher than the current one, so that an old item cannot be sent again and a Source cannot be brought back to an earlier state.
  • The Controller records, for each item, who wrote it, when, and with which Version.

AIMs read only items that passed these checks.

6.4 The User Agent

The User Agent has no function on Access. Access is infrastructure that serves the AIMs: an AIM declares in its Metadata the data it reads (6.5), and the Controller obtains it when it reads the Metadata. What the User Agent keeps for itself, it keeps in Shared Storage, by telling the Controller where it is and for how long (MPAI_AIFU_SharedStorage_Init, MPAI_AIFU_SharedStorage_Keep), or in its own storage outside the AIF, from which it gives data to the Controller on the Ports of a Module – the avatar of a person, a choice of avatars – and the Controller hands it to the AIMs.

6.5 Data identified by its URI

The Metadata of an AIM names the data it reads by the URI it is obtained from – an http, https or file URI – and pins it with a SHA-256: of the file, or, for a set of files under a URI that ends in a slash, of the manifest of the set, a file manifest.json under that URI that lists the Keys of the set with the SHA-256 of each file. The Controller obtains the file, or the manifest, keeps it in Access with the Controller as its Writer, and refuses what does not have the SHA-256 the Metadata gives. The items of a set are obtained when they are first read, each checked against the manifest. A Module is not started when data its AIMs need, and do not mark optional, cannot be obtained. The Writer of such a Source is the Controller; a Provider may also write Sources (6.3).

The SHA-256 is the identity of the data and the URI is only where it is found. An entry may give Mirrors, other places holding the same bytes, which the Controller tries in order when the URI cannot be reached; data from any location that does not have the SHA-256 is refused.

Data already in Access is not downloaded again. Before obtaining anything the Controller looks in Access for data with that SHA-256, whatever URI it was obtained from – for another AIM, for another Module, or from a mirror – checks it, and keeps it under the URI the entry gives. This applies to each item of a set as well. Large data is received into a file and checked as it arrives, so that its size is not held in memory, and an AIM reads it as a stream (MPAI_AIFM_Access_Open).

A machine learning model is data of Access like any other: its weights, and what goes with them (a tokeniser, a template), are entries of the AIM, each by the URI where its publisher keeps it and its SHA-256. The program that runs a model – Ollama, a speech engine, a runtime – is not what the AIM reads, but it too may be declared in Access, by the URI of the archive it is published in and the SHA-256 of the archive: the Controller unpacks the archive, once, into a folder (Unpack, Inside) and gives the Implementation the program, which is the program the SHA-256 pins. The program is given the weights from Access, as a stream or, for a program that reads files, as the file where Access keeps them (MPAI_AIFM_Access_Path). An entry may name the Setting of the Implementation that receives that file, and the Controller then gives it to the Implementation; the rules of the datum apply to that as to any read.

An entry may give with ValidWith the SHA-256 of other data of the same AIM that it is only meaningful with – for instance a database of features and the model that computed them. A Module is not started if that data is not declared by the AIM or, when the entry is not optional, cannot be obtained; an optional entry whose partner is missing is not given to the AIM. Terms states the licence under which the data is published, where the terms are stated, and whether the data describes persons: it is information for the one who deploys the Module, and the Controller does not enforce it. In the entries of an AIM a Name is used once and a URI has one SHA-256.

6.6 Who may read

Who may read a datum of Access is decided as it is in Shared Storage (3), by the rules of the datum: its Category, its Readers – AIM Instances, Modules or the User Agent – and its Trace. The Writer of a Source sets them when it writes an item; the Controller, when it writes what it obtains for an AIM, names that AIM among the Readers. Another AIM that declares the same data is added to the Readers, and the data is not obtained again. An item written without Readers is read by every holder. The Writer of an item and the central control read everything; with a central control, the general rules of each category are its, and a writer may restrict them for its datum and not extend them. The Controller binds every handle to the identity of its holder, an AIM Instance or the User Agent, and applies the rules at every read, also for an AIM that executes apart from the Controller. A call the rules do not allow returns NOT_AUTHORISED.

Writing differs from Shared Storage, since only the Writer of a Source writes it and each update states the Version (6.3, 6.4), but the principles are the same. An item of Access has no time: it is kept until its Writer removes it or replaces it with a new Version.

Note. How a Provider makes a Source available to a Module – for instance, under a licence – is outside the scope of this Technical Specification.

7 Security

Every read and write passes through a handle the Controller bound, and Storage checks the Grant of the handle at every operation. The handles of a stopped Module are dead. In the Secure Profile, Secure Storage holds keys and Grants and is reached through the Security API (Zero Trust and Profiles).

8 Requirements

Table 3 – Requirements of Storage

# Requirement
STO-1 An AIM shall reach Storage only through the handles the Controller bound to it.
STO-2 Every write shall be stamped by the Controller with the Module, the AIM and the time; the writer shall not supply the stamp.
STO-3 A datum of the Private Storage of the Module or of Shared Storage shall be written under a category, readers and a time, and the Controller shall keep its Trace.
STO-4 A writer shall not extend the general rules of a central control; a write that tries shall be refused.
STO-5 A call the rules do not allow shall return NOT_AUTHORISED.
STO-6 No party but the Controller shall write into the record of a Module’s boundary.
STO-7 A record shall not delay the Module; what it could not write shall be counted and reported.
STO-8 A Controller shall not download data for Access when data with the SHA-256 that the Metadata gives is already in Access; it shall check that data before giving it to an AIM.
STO-9 A Controller shall refuse data, from the URI or from a mirror, that does not have the SHA-256 that the Metadata gives.
STO-10 A Controller shall not start a Module in which an AIM declares data valid only with other data that the AIM does not declare or, if the entry is not optional, that cannot be obtained.
STO-11 An item of Access shall be written under rules – Category and Readers – and a read that the rules of the item do not allow shall return NOT_AUTHORISED, whether the AIM executes with the Controller or apart from it.

<-Communication Go to ToC Metadata ->