All posts

In the business world, goods are delivered based on technical and commercial specifications. In the standards world, there are good reasons why the goods (the standards) of a Standards Developing Organisation (SDO) are not delivered according to commercial requirements normally accepted in the business world. However, this is not a good reason for an SDO to stay with commercial requirements called “patent declarations” that simply bind the originators to license their SEPs at so-called Fair, Reasonable and Non-Discriminatory (FRAND) terms. This simply would not make sense in business and this is the reason why FRAND has been and continues to be causing problems.

The Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) SDO was established to develop data coding standards mostly using the Artificial Intelligence (AI) technology while offering, for each of its standards, a clear licensing framework to implementers.

This is how MPAI implements its process:

1. A new standard may be proposed by anybody.
2. Anybody may participate in the development of the Use Cases and Functional Requirements of a standard.
3. MPAI Principal Members intending to participate in the development of a standard develop and approve, with 2/3 majority, the Framework Licence (FWL) for that standard. The FWL is a licence without values (dollars, percentages, rates, dates, etc.) containing a declaration that:
   1. The total cost of the licence will be in line with the total cost of the licenses for similar data coding technologies and will consider the market value of the specific standardised technology.
   2. The licence will be issued not after commercial implementations of the standard are made available on the market.
4. During the development of the standard, MPAI members making technical contributions to the committee developing the standard declare they will make available their licences according to the FWL. Non members may participate in the development of the standard by becoming members.
5. After the standard has been approved by the MPAI General Assembly:
   1. MPAI members who believe to be SEP holders express their preference on the patent pool administrator of the standard with a 2/3 majority.
   2. All Members declare they will get a licence for other members’ SEPs, if used, within one year after publication of the licensing terms by SEP holders.

The MPAI process ensures that

1. The use cases and functional requirements of a standard are developed with participation of the eventual users, not just by MPAI members (i.e., the technology developers).
2. Information about the eventual licence of a standard includes time (not after products are on the market) and cost (total cost of the licence in line with the total cost of the licenses for similar technologies).

Sure, this is not the same as a hard delivery date of and a price tag in dollars – standard are a special type of goods that is closely watched by antitrust authorities. But it is a long way from a promise that cost will be “fair”, “reasonable” and “non-discriminatory”, and time at heaven’s will.

You can find more information on the MPAI process and the FWL.

The term watermarking comprises a family of methodological and application tools used to insert data into a content item in a way that is as imperceptible and persistent as possible. Watermarking is used for different purposes such as to enable an entity to claim ownership of a content item or a device to use it.
As a neural network is a type of content – and one that may be quite expensive to develop – does it make sense to apply the watermarking approach to content to neural networks?
MPAI thinks it does and is working to develop requirements for a Neural Network Watermarking (NNW) standard called MPAI-NNW that will enable a watermarking technology provider to validate their products’ claims. The standard will provide the means to measure, for a given size of the watermarking payload, the ability of:

• The watermark inserter to inject a payload without affecting the performance of the neural network. This item requires, for a given application domain:
  • A testing dataset to be used for the watermarked and unwatermarked neural network.
  • An evaluation methodology to assess any change of the performance induced by the watermark.
• The watermark detector to recognise the presence of the inserted watermark when applied to a watermarked network that has been modified (e.g., by transfer learning or pruning) or to any of the inferences of the modified model. This item requires, for a given application domain:
  • A list of potential modification types expected to be applied to the watermarked neural network as well as of their ranges (e.g., random pruning at 25%).
  • Performance criteria for the watermark detector (e.g., relative numbers of missed detections and false alarms).
• The watermark decoder to successfully retrieve the payload when applied to a watermarked network that has been modified (e.g., by transfer learning or pruning) or to any of the inferences of the modified model. This item requires, for a given application domain:
  • A list of potential modification types expected to be applied to the watermarked neural network as well as of their ranges (e.g., random pruning at 25%).
  • ​​Performance criteria for the watermark decoder (e.g., 100% or (100-α)% recovery).
• The watermark inserter to inject a payload at a low computational cost, e.g., execution time on a given processing environment.
• The watermark detector/decoder to detect/decode a payload from a watermarked model or from any of its inferences, at a low computational cost, e.g., execution time on a given processing environment.

You can read the MPAI-NNW Use cases & functional requirements WD 0.2.

The work of developing requirements for the MPAI-NNW standard is ongoing. In this phase of the work, participation is open to non members. Contact the MPAI Secretariat if you wish to join the MPAI-NNW online meetings.

The term watermarking comprises a family of methodological and application tools used to insert data into a content item in a way that is as imperceptible and persistent as possible. Watermarking is used for different purposes such as to enable an entity to claim ownership of a content item or a device to use it.
As a neural network is a type of content – and one that may be quite expensive to develop – does it make sense to apply the watermarking approach to content to neural networks?
MPAI thinks it does and is working to develop requirements for a Neural Network Watermarking (NNW) standard called MPAI-NNW that will enable a watermarking technology provider to validate their products’ claims. The standard will provide the means to measure, for a given size of the watermarking payload, the ability of:

• The watermark inserter to inject a payload without affecting the performance of the neural network. This item requires, for a given application domain:
  • A testing dataset to be used for the watermarked and unwatermarked neural network.
  • An evaluation methodology to assess any change of the performance induced by the watermark.
• The watermark detector to recognise the presence of the inserted watermark when applied to a watermarked network that has been modified (e.g., by transfer learning or pruning) or to any of the inferences of the modified model. This item requires, for a given application domain:
  • A list of potential modification types expected to be applied to the watermarked neural network as well as of their ranges (e.g., random pruning at 25%).
  • Performance criteria for the watermark detector (e.g., relative numbers of missed detections and false alarms).
• The watermark decoder to successfully retrieve the payload when applied to a watermarked network that has been modified (e.g., by transfer learning or pruning) or to any of the inferences of the modified model. This item requires, for a given application domain:
  • A list of potential modification types expected to be applied to the watermarked neural network as well as of their ranges (e.g., random pruning at 25%).
  • ​​Performance criteria for the watermark decoder (e.g., 100% or (100-α)% recovery).
• The watermark inserter to inject a payload at a low computational cost, e.g., execution time on a given processing environment.
• The watermark detector/decoder to detect/decode a payload from a watermarked model or from any of its inferences, at a low computational cost, e.g., execution time on a given processing environment.

You can read the MPAI-NNW Use cases & functional requirements WD 0.2.

The work of developing requirements for the MPAI-NNW standard is ongoing. In this phase of the work, participation is open to non members. Contact the MPAI Secretariat if you wish to join the MPAI-NNW online meetings.

 Geneva, Switzerland – 20 April 2022. Today the international, non-profit, unaffiliated Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) standards developing organisation has concluded its 19^th General Assembly. Among the outcomes is the publication of the working draft of the Use Cases and Functional Requirements of the planned Version 2 of the Multimodal Conversation (MPAI-MMC) standard.

The MPAI process envisages that a standard be developed based on a Call for Technologies referring to two documents Functional Requirements and Framework Licence. While the MPAI-MMC V2 documents are still being finalised, MPAI offers an initial working draft of the Functional Requirements to alert the industry of its intention to initiate the development of the standard. This will happen when the Call for Technologies is published (planned to be the 13^th of July 2022). Responses are expected to be submitted on the 10^th of October 2022 and the standard to be published in the first few months of 2023.

Version 2 will substantially extend the capabilities of Version 1 of the MPAI-MMC standard by supporting three new use cases:

1. Conversation About a Scene: a human holds a conversation with a machine about objects in a scene of which the human is part. While conversing, the human points their fingers to indicare their interest in a particular object.
2. Human-Connected Autonomous Vehicle Interaction: a group of humans has a conversation on a domain-specific suject (travel by car) with a Connected Autonomous Vehicle. The machine understands the utterances, the emotion in the specch and in the faces, and the expression in their gestures. The machine manifests itself as the torso of an avatar whose face and head convey emotions congruent with the the speech it utters.
3. Avatar Videoconference. In this instance of Mixed-reality Collaborative Space (MCS), avatars represent humans participating in a videoconference. Avatars reproduce the movements of the torsoes of human participants with a high degree of accuracy.

MPAI develops data coding standards for applications that have AI as the core enabling technology. Any legal entity supporting the MPAI mission may join MPAI, if able to contribute to the development of standards for the efficient use of data.

So far, MPAI has developed 5 standards (normal font in the list below), is currently engaged in extending two approved standards (underlined) and is developing other 9 standards (italic).

Visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

Most importantly: join MPAI, share the fun, build the future.

Video coding has been one of the first standardisation areas addressed by MPAI (the first was Content-based Enhanced Audio Coding or MPAI-CAE). One MPAI Video Coding area is focused on the use of AI tools to improve the classic block-based hybrid coding framework and offer more efficient video compression solutions. This was motivated by the rough estimate that MPAI did from its survey on published AI-based video coding results leading to a potential improvement of up to 30%.

Figure 1 identifies 10 coding tools that can potentially be improved by AI tools.

Figure 1 – A classic block-based hybrid coding framework

MPAI decided to start from a high-performance “clean-sheet” traditional (i.e., data-processing-based) coding scheme and add AI-enabled improvements to it, rather than starting from a scheme overloaded by IP-laden compression technologies bringing dubious improvements. It selected the MPEG-5 Essential Video Coding (EVC) standard and is modifying it by enhancing/replacing existing video coding tools with AI tools. The EVC Baseline Profile has been selected because it employs 20+ years old technologies and has a compression performance close to that of HEVC.

MPAI calls this project MPAI AI-Enhanced Video Coding (MPAI-EVC). However, MPAI has made the deliberate decision not to initiate a standard project at this time, because it first wants to set up a unified platform and conduct experiments with the goal to ascertain that 25% coding performance improvement can be achieved. Therefore the name of the current activity within the project is “MPAI-EVC Evidence Project”.

Significant results have already been achieved by a growing group of participants that includes both MPAI and non-MPAI members. Note: MPAI has the policy of allowing non-members to participate in its preliminary pre-standardisation activities.

As soon as the Evidence Project will demonstrate that AI tools can improve the MPEG-5 EVC efficiency by at least 25%, MPAI will be in a position to initiate work on the MPAI-EVC standard. The functional requirements have already been developed and only need to be revised. Then, the framework licence will be developed by active principal members and the Call for Technology issued to acquire the technology needed to develop the MPAI-EVC standard..

MPAI-EVC covers the short-to-medium term video coding needs. If you want to know more. visit the MPAI-EVC web page or contact the MPAI Secretariat to participate in MPAI-EVC meetings.

MPAI has a second video coding project motivated by the consensus in the video coding research community that the so-called End-to-End (E2E) video coding schemes can yield significantly higher performance in the longer term.

MPAI is conducting the MPAI End-to-End Video Coding (MPAI-EEV) project in its role as a technical body whose mission is the provision of efficient and usable data coding standards, unconstrained by legacy IP. The notion of Framework Licence is at the basis of this work. The Basic Framework Licence for Collaborative Explorations is the interim MPAI-EEV Framework Licence. This means that contributions submitted to MPAI-EEV (formally, to the Requirements (EEV) group) shall be accompanied by this licence.

MPAI has selected OpenDVC as a starting point and is investigating the addition of novel motion compensation networks.

Figure 2 – The OpenDVC reference model with a motion compensation network

Clearly, MPAI-VVC targets the longoterm term video coding needs. If you want to know more visit the MPAI-EEV web page or contact the MPAI Secretariat to participate in MPAI-EEV meetings.

On the day MPAI Multimodal Conversation (MPAI-MMC) reached its full 6 months since its approval, the IEEE hosted the kick-off meeting of the P3300 working group tasked with the adoption of the MPAI technical specification as an IEEE standard. Earlier, MPAI and IEEE had signed an agreement whereby MPAI grants IEEE the right to publish MPAI-MMC as an IEEE standard.

At its first meeting, the WG has approved the working draft of IEEE 3300 and requested IEEE to ballot the WD. In a couple of months, MPAI-MMC is expected to become IEEE 3300.

The creation of the WG and the development of the IEEE 3300 standard are the natural steps following the issuance of the Call for Patent Pool Administrator by the MPAI-MMC patent holders. The next step will be the development of the Use Cases and Functional Requirements for MPAI-MMC Version 2 that MMC-DC and other groups are busy preparing.

The IEEE 3300 WD is verbatim MPAI-MMC, so this article is a good opportunity to recall the MPAI document and its structure. If you want to follow this description with the actual text, please download it.

Chapter 1 is an informative introduction to MPAI, the AI Framework (MPAI-AIF) approach to AI data coding standards including the notion of AI Modules (AIM) organised in and AI Workflow executed in the AI Framework (AIF), and the governance of the MPAI ecosystem.

Chapter 2 is a functional specification of the 5 use cases:

“Conversation with Emotion” (CWE):  a human is holding an audio-visual conversation with a machine impersonated by a synthetic voice and an animated face. Both the human and the machine express emotion.
“Multimodal Question Answering” (MQA): a human is holding an audio-visual conversation with a machine impersonated by a synthetic voice. The human asks a question about an object held in thei hand.
Three Uses Cases supporting conversational translation applications. In each Use Case, users can specify whether speech or text is used as input and, if it is speech, whether their speech features are preserved in the interpreted speech: – “Unidirectional Speech Translation” (UST). – “Bidirectional Speech Translation” (BST). – “One-to-Many Speech Translation” (MST).

Chapter 3 contains definitions of terms that are specific to MPAI-MMC.

Chapter 4 contains normative and information references.

Chapter 5 contains the specification of the 5 use cases. For each of them, the following is specified:

1. The Scope of the Use Case
2. The syntax and semantics of the data entering and leaving the AIW
3. The Architecture of AIMs composing the AIW implementing the Use Case
4. The functions of the AIMs
5. The JSON Metadata describing the AIW

Chapter 6 contains the specification of all the AIMs of all the Use Cases:

1. A note about the meaning of AIM interoperability
2. The syntax and semantics of the data entering and leaving all the AIMs of the 5 AIWs
3. The formats of all the AIM data

Annex 1 defines the terms not specific to MPAI-MMC

Annex 2 contains notices and disclaimers concerning MPAI standards (informative)

Annex 3 provides a brief introduction to the Governance of the MPAI Ecosystem (informative)

Annex 4 and the following annexes provide the AIW and AIM metadata of all MPAI-MMC Use Cases.

MPAI-MMC is just the initial step. Two more MPAI Technical Specifications have been submitted for adoption: AI Framework (MPAI-AIF) and Context-based Audio Enhancement.

MPAI is looking forward to a mutually beneficial collaboration with IEEE.

A definite answer to such a question is noto going to come anythime soon, and many have attempted to develop algorithms providing an answer.

MPAI could not miss the opportunity to give its own answer and has published the MPAI-CUI standard, or Compression and Understand of Financial Data. This contains one use case: Company Performance Prediction (CPP).

What does CUI-CPP offer? Imagine that you have a company and you would like to know what is the probability that your company defaults in the next, say, 5 years. The future is not written, but it certainly depends on how your company has performed in the last few years and on the solidity of your company’s governance.

Another element that you would like to know is what is the probability that your company suspends its operations because an unexpected event such as a cyber attack or an earthquake has happened. Finally, you should probably also want to know how adequate is the organisation of your company (but many don’t want to be told ;-).

CUI-CPP needs financial statements, governance data, and risk data as input. The first thing the standard does is to compute the financial and governance features from the financial statements and governance data. These features are fed to a neural network that has been trained with many company data and provides the default probability and the organisational model adequacy index (0=inadequate, 1=adequate).

Then CUI-CPP computes a risk matrix based on cyber and seismic risks and uses this to perturb the default probability and obtain the business discontinuity probability.

The full process is described in Figure 1.

Figure 1 – Reference Model of Company Performance Prediction

Many think that once the technical specification is developed, the work is over. Well, that would be like saying that, after you have produced a law, society can develop. That is not the case, because society needs tribunals to assess whether a deed performed by an individual conforms with the law.

Therefore, MPAI standards do not just contain the technical specification, i.e., “the minimum you must know to make an implementation of the standard”, but 4 components in total, of which the technical specification is the first.

The second is the reference software specification, a text describing how to use a software implementation that lets you understand “how the standard works”. Reference software is an ingredient of many standards and many require that the reference software be open source.

MPAI is in an unusual position in the sense that it does not specify the internals of the AI Modules (AIM), but only their function, interfaces, and connections with the other AIMs that make up an AI Workflow (AIW) with a specified function. Therefore, MPAI does not oblige the developers of an MPAI standard to provide the source code of an AIM, the may provide a compiled AIM (of course, they are welcome if they do, more so if the AIM has a high performance).

In the case of CUI-CPP, all AIMs are provided as open-source software, including the neural network called Prediction. Of course, you should expect that the reference software “demonstrates” how the standard works, not that it makes very accurate predictions of a particular company

The third component is conformance testing. Many standards do not distinguish between “how to make a thing” (the technical specification, i.e., the law) from “how to test that a thing is correctly implemented” (the conformance testing specification, i.e., the tribunal). MPAI provides a Conformance Testing Specifcitation that specifies the “Means”, i.e.:

1. The Conformance Testing Datasets and/or the methods to generate them,
2. The Tools, and
3. The Procedures

to verify that the AIMs and/or the AIW of a Use Case of a Technical Specification:

1. Produce data whose semantics and format conform with the Normative clauses of the selected Use Case of the Technical Specification, and
2. Provide a user experience level equal to or greater than the level specified by the Conformance Testing Specification.

In the case of CUI-CPP, MPAI provides test vectors, and the Technical Specification specifies the tolerance of the output vectors produced by an implementation of the AIMs or AIW when fed with the test vectors.

The fourth component is the performance assessment to verify that an implementation is not just “technically correct”, but also “reliable, robust, fair and replicable”. Essentially, this is about assessing that an implementation has been correctly trained, i.e., it is not biased.

The performance assessment specification provides the Means, i.e.:

1. The methods to generate the Performance Testing Datasets,
2. The Tools, and
3. The Procedures

to verify that the training of the Prediction AIM (the only one that it makes sense to implement with a neural network) is not biased against some geographic locations and industry types (service, public, commerce, and manufacturing).

The CUI-CPP performance assessment specification assumes that there are two performance assessment datasets:

1. Dataset #1 not containing geographic location and industry type information.
2. Dataset #2 containing geographic location and industry type information.

The performance of an implementation is assessed by applying the following procedure:

1. For each company compute:
   2. The Default Probabilities of all records in Dataset #1 and in Dataset #2.
   3. The Organisational Model Index of all records in Dataset #1 and in Dataset #2.
2. Verify that the average of the differences of all
   1. Default Probabilities in 1.a is < 2%.
   2. Organisational Model Indices in 1.b is < 2%.

If both 2.a and 2.b are verified, the implementation passes the performance assessment.

Watermarking has been used for a long time. One of its uses in the physical world is paper money where a hard to imitate watermark assures users that a banknote is authentic.

In the digital domain, watermarking can be used to carry information about ownership in a file or stream. The Secure Digital Music Initiative (SDMI) selected a strong (i.e., hard to remove) digital watermark to identify an MP3 soundtrack that had been released “after” and attempted to define a weak (i.e., easy to remove) watermark.

Neural networks are a high-priority topic in MPAI. Is there a reason why MPAI should be concerned with watermarking? The answer is yes, and the reason is that developing neural networks may be a very costly undertaking, e.g., several tens of thousand USD and developers may indeed want to identify that a neural network is theirs.

MPAI has begun to investigate two related but distinct issues: watermarking for neural networks and watermarking for the data produced by a neural network fed with data and generating inference.

By using a specific watermarking technology, the neural network creator can claim that a particular neural network instance:

1. Has been produced by the them.
2. Is a derivative of their network.
3. Has been modified in a particular part of the network.

A related story applies to the inferences. The inference of a neural network can also be watermarked. The purpose is not necessarily that of protecting the creator or a licensee of a neural network. The end user of a neural network may need to be assured that an inference has been produced by the intended network.

So, what is MPAI actually doing in this field? The MPAI Neural Network Watermarking (NNW) project is developing requirements for a future MPAI standard with the goal to measure, for a given size of the watermarking payload:

1. The impact on the performance of the neural network caused by adding a watermark to a neural network.
2. The resistance of the watermark to modifications, e.g., caused by transfer learning, pruning of the weights etc.
3. The cost of watermark injection because a neural network may be very large and adding a watermark costs time and processing.

Read The MPAI Neural Network Watermarking (NNW) project for more details.

If you wish to participate in this work you have the following options:

1. Join MPAI
2. Participate until the MPAI-NNW Functional Requirements are approved (after that only MPAI members may participate) by sending an email to the MPAI Secretariat.