The Robustness evaluation specifies the Means to enable a Tester to evaluate the robustness of the watermark against a set of modifications requested by one of the Actors.
The Tester evaluates the decoder and detector capability of a watermarking technology as specified in the following workflow:
- Select:
- A set of M unwatermarked NNs trained on the training dataset.
- D data payloads corresponding to the pre-established payload size.
- Apply the watermarking technology to the M NNs with the D data payloads
- Produce a set of M x (D + 1) modified NNs (M unwatermarked NNs and M x D watermarked NNs), by applying one of the Modifications in Table 3 to a given Parameter value.
- Evaluate the Robustness of the detector:
- Apply the Watermark detector to any of the M x (D + 1) NNs
- Record the corresponding binary detection results (Yes – the mark is detected or No – the mark is not detected) – see Figure 7.
- Label the Yes/No outputs of the Watermark detector as true positive, true negative, false positive (false alarm) and false negative (missed detection) according to the actual result – see Table 1.
- Count the total number of false positives and the total number of false negatives.
- Evaluate the Robustness of the decoder:
- Apply the Watermark decoder to any of the M x (D + 1) NNs
- Compute a Distance between the outputs of the decoder and their corresponding original data payloads.
- Compute the Symbol Error Rate (SER) for any of the M x (D + 1) NNs, as the ratio of the distance to the size of the corresponding data payload.
- Compute the average SER, as the average over the M x (D + 1) SER values computed in the previous step.
- Provide the average values over the total number of tests:
- The ratio of the number of false positives to M x (D + 1),
- The ratio of the number of false negatives to M x (D + 1).
- The M x D number for tested NNs, and the average SER.
- Repeat steps 3, 4, 5 and 6 for the requested number of Parameters values chosen in the ranges provided by Table 2.
- Repeat steps 3, 4, 5, 6 and 7 for the requested set of Modifications chosen in the ranges provided by Table 2.
Table 2. List of modification with their parameters
| Modification name | Parameter type | Parameter range |
| Modification | Parameter type | Parameter range |
| Gaussian noise addition: adding a zero-mean, S standard deviation Gaussian noise to a layer in the NN model. This noise addition can be simultaneously applied to a sub-set of layers. | – the layers to be modified by Gaussian noise – the ratio of S to standard deviation of the weights in the corresponding layer |
– 1 to total number of layers – 0.1 to 0.3 |
| L1 Pruning: delete the P% of the smallest weights, irrespective of their layers. | – the P percentage of the deleted weights | – 1% to 90% – 1% to 99.99% when aiming one layer |
| Random pruning: delete R% of randomly selected weights, irrespective of their layers. | – the R percentage of the deleted weights | – 1% to 10% |
| Quantizing: reduce to B the number of bits used to represent the weights by 1. reducing the number of bits based on a sequence of three operations: affine mapping from the weights interval to the (0;2B-1) 2. rounding to the closest integer 3. backward affine mapping towards the initial weights interval |
– the layers to be modified by quantization – the value of B
|
– 1 to total number of layers – 32 to 2 |
| Fine tuning / transfer learning: resume the training of the M watermarked NNs submitted to test, for E additional epochs. | – ratio of E to the number of epochs in the initial training | – up to 0.5 time the total number of epochs
|
| Knowledge distillation: train a surrogate network using the inferences of the NN under test as training dataset | – The structure of the architecture – The size of the dataset D – The number of epochs E |
– structures N – 10,000 to 1,000,000 – 1 to 100 |
| Watermark overwriting: successively insert R additional watermarks, with random payloads of the same size as the initial watermark | – R number of watermarks successively inserted | – 2 to 4 |
