The Robustness evaluation specifies the Means to enable a Tester to evaluate the Robustness of the Traceability Data against a set of Modifications requested by one of the Actors.

1       Watermarking

The Tester evaluates the Decoder, and Detector of a NN Watermarking Method as specified in the following workflow:

  1. Select:
    1. A set of M unwatermarked NNs trained on the training dataset.
    2. D data Payloads corresponding to the pre-established Payload size.
  2. Apply the NN Watermarking Method to the M NNs with the D data Payloads
  3. Produce a set of M x (D + 1) modified NNs (M unwatermarked NNs and M x D watermarked NNs), by applying one of the Modifications in Table 3 to a given Parameter value.
  4. Evaluate the Robustness of the Detector:
    1. Find the presence of the watermark in all M x (D + 1) NNs using the Watermark Detector.
    2. Record the corresponding binary detection results (Yes – the mark is detected or No – the mark is not detected).
    3. Label the Yes/No outputs of the Watermark Detector as true positive, true negative, false positive (false alarm) and false negative (missed detection) according to the actual result.
    4. Count the total number of false positives and the total number of false negatives.
  5. Evaluate the Robustness of the Decoder:
    1. Extract the Payload from all M x (D + 1) NNs using the Watermark Decoder.
    2. Count the number of different Symbols between the outputs of the Decoder and their corresponding original data Payloads.
    3. Compute the Symbol Error Rate (SER) for any of the M x (D + 1) NNs, as the ratio of the number of different Symbols to the number of the Symbols in the data Payload.
    4. Compute the average SER, as the average over the M x (D + 1) SER values computed in the previous step.
  6. Provide the average values over the total number of tests:
    1. The ratio of the number of false positives to M x (D + 1),
    2. The ratio of the number of false negatives to M x (D + 1).
    3. The M x D number for tested NNs, and the average SER.
  7. Repeat steps 3, 4, 5, and 6 for the requested set of Intensity Modifications of Table 3.
  8. Repeat steps 3, 4, 5, 6, and 7 for the requested set of Type Modifications of Table 3.

Table 3 – List of modification with their parameters

Modification name Parameter Type Parameter Intensity
Gaussian noise addition: adding a zero-mean, S standard deviation Gaussian noise to a layer in the NN model. This noise addition can be simultaneously applied to a sub-set of layers. – The layers to be modified by Gaussian noise
– The ratio of S to standard deviation of the Weights in the corresponding layer.		 –  1 to total number of layers.
–  0.1 to 0.3.
L1 Pruning: delete the P% of the smallest Weights in a layer for each layer. –  The P percentage of the deleted Weights. –  1% to 90%.

–  1% to 99.99% when aiming one layer.
Random pruning: delete R% of randomly selected Weights, irrespective of their layers. –          The R percentage of the deleted Weights. – 1% to 10%.
Quantizing: reduce to B the number of bits used to represent the Weights by
1.     Reducing the number of bits based on a sequence of three operations: affine mapping from the interval the Weights belong to  interval.
2.     Rounding to the closest integer.
3.     Backward affine mapping towards the initial interval the Weights belong to.		 – The layers to be modified by quantization.
– The value of B.		 –  1 to total number of layers.
–  32 to 2.
Fine tuning / transfer learning: resume the training of the M watermarked NNs submitted to test, for E additional epochs. – Ratio of E to the number of epochs in the initial training. – Up to 0.5 time the total number of epochs.

 
Knowledge distillation: train a surrogate network using the inferences of the NN under test as training dataset – The structure of the architecture.
– The size of the dataset D.
– The number of epochs E.		 – Structures N.
– 10,000 to 1,000,000.
– 1 to 100.
Watermark overwriting: successively insert W additional watermarks, with random Payloads of the same size as the initial watermark –  W number of watermarks successively inserted. – 2 to 4.

2       Fingerprinting

The Tester evaluates the capability of a NN Fingerprinting Method Matcher as specified in the following workflow:

  1. Select a set of Mu NNs trained on the training dataset.
  2. Compute the Mu fingerprints from the unmodified NNs.
  3. Produce a set of Mm modified NNs, by applying one of the Modifications in Table 3 to a given Parameter value.
  4. Evaluate the Robustness of the Matcher:
    1. Compute the fingerprint for any of the Mm
    2. Apply the Matcher to the Mm fingerprints obtained in 4.a and record its output (Yes – the matching found is correct or No – the matching found is not correct).
    3. Label the Yes/No outputs of 4.b as true positive, true negative, false positive (false alarm) and false negative (missed detection).
    4. Count the total number of false positives and the total number of false negatives.
  5. Provide the average values over the total number of tests:
    1. The ratio of the number of false positives to Mm,
    2. The ratio of the number of false negatives to Mm.
    3. The Mm number for tested NNs, and the average BER.
  6. Repeat steps 3, 4, and 5 for the requested set of Intensity Modifications of Table 3.
  7. Repeat steps 3, 4, 5, and 6 for the requested set of Type Modifications of Table 3.