This chapter will deal with two cases:

  • NNs for which the watermark is added after the NNs model was created.
  • NNs for which the watermark is added during the training of the NNs model.

1       Watermark embedding is done after training

The Imperceptibility evaluation specifies the Means that enable a Tester to evaluate the differences in performance of a neural network before and after the watermark embedding process. There are two cases:

  1. The NN has the input and output data format with specified semantics.
  2. The input and output data format of the NN do not have specified semantics.

1.1      Evaluation of an NN whose I/O data format has specified semantics

In this section, two actors are involved: the NN Watermarking provider requesting a Tester to evaluate the Imperceptibility performance of their watermarking technology.

The Tester shall adopt the following procedure:

  1. Define a pair of training and testing datasets with a size with at least an order of magnitude more entries than trainable parameters.
  2. Select:
    1. A set of M unwatermarked NNs trained on the training dataset.
    2. D data payloads corresponding to the pre-established payload size.
  3. Apply the watermarking technology to the M
  4. Process the training dataset and the D data payloads (if needed).
  5. Feed the M unwatermarked NN with the test dataset
  6. Measure the task-dependent quality of the produced inference.
  7. Feed the M x D watermarked NN with the same test dataset
  8. Measure the task-dependent quality of the produced inference, informative examples of quality evaluation are provided in Annex 5 .
  9. Provide the task-dependent quality of the produced inference measured in 6 and 7.

1.2      Evaluation of an NN whose I/O data format has no specified semantics

In this section, two actors are involved: the NN Watermarking provider requesting a Tester to evaluate the Imperceptibility performance of their watermarking technology.

The workflow of the process shall be the following:

  1. Tester connects the NN to other NN until the input and output of the resulting configuration have input / output formats with specified semantics.
  2. Tester applies all the steps in 6.1.1.

2       Watermark embedding is done during training

The Imperceptibility evaluation specifies the Means for evaluating the performance of a watermarked neural network. The workflow of the process shall evaluate the watermarked NN as an NN.