<-Robustness Evaluation     Go to ToC      Imperceptibility Evaluation->

The Computational Cost evaluation specifies the Means that enable a Tester to measure the Computational Cost of:

  • Injecting, in terms of memory footprint, time to process an epoch, and number of epochs necessary to insert the watermark.
  • Detecting, Decoding, or Extracting in terms of memory footprint and time for the Detector or the Decoder or the Extractor to produce the expected result.
  • Matching in terms of memory footprint and time for the Matcher to produce the expected result.

1       Computational Cost of injecting a watermark

The Computational Cost evaluation specifies the Means that enable a Tester to measure the Computational Cost of the injection using NN Watermarking Method under testing.

The following four elements shall be used to measure the injection process:

  1. The memory footprint.
  2. The time to execute the operation required by one epoch normalized according to the number of batches processed in one epoch.
  3. If the injection is done concurrently with the training of the network, the number of epochs required to insert the watermark.
  4. The time for the watermarked NN to compute an inference.

The Tester shall measure the Computational Cost of the injection according to the following workflow:

  1. Define a pair of training and testing datasets with a size with at least an order of magnitude more entries than trainable Weights.
  2. Select:
    1. The training dataset (if needed).
    2. A set of M unwatermarked NNs trained on the training dataset.
    3. D data Payloads corresponding to the pre-established Payload size.
  3. Apply the NN Watermarking Method to the M NNs using the D data Payloads.
  4. Record the corresponding M x D set of Computational Costs.
  5. Provide the average and 95% confidence limits of the Computational Costs divided by M x D for one of the informative Testing Environments of Table 4.

Table 4 – Testing Environments (informative)

  Testing environment
Medium – Single GPU (16GB/6144 CUDA cores)
– 8 cores CPU (2.6GHz)
Large – Double GPU (32GB/12288 CUDA cores)
– 16 cores CPU (3.4GHz)

2       Computational Cost of Detection, Decoding or Extraction

MPAI-NNT specifies the Means that enable a Tester to measure the Computational Cost of the Detection/Decoding/Extraction of a NN Traceability Method.

2.1      Detection and/or Decoding

The Computational Cost of Detection/Decoding is measured by the time and the memory footprint used by the process.

The Tester shall measure the Computational Cost according to the following workflow:

  1. Select a set of M unwatermarked NNs, D data Payloads corresponding to the pre-established Payload size and, if needed, the training dataset.
  2. Apply the NN Watermarking Method to the M NNs with the D data Payloads
  3. Evaluate the Robustness of the Detector:
  4. Apply the Watermark Detector to any of the M x D
  5. Record the corresponding M x D set of values.
  6. Evaluate the Robustness of the Decoder:
  7. Apply the Watermark Decoder to any of the M x D
  8. Record the corresponding M x D set of Computational Costs.
  9. Provide the average and 95% confidence limits of the Computational Costs divided by M x D for one of the informative Testing Environments of Table 4.

2.2      Extraction

The Computational Cost of Extraction is measured by the time and the memory footprint used by the process.

The Tester shall evaluate the Computational Cost according to the following workflow:

  1. Select a set of Mu NNs trained on the training dataset.
  2. Compute the Mu fingerprints from the unmodified NNs.
  3. Evaluate the Robustness of the NN Traceability Method:
  4. Apply the fingerprint Extractor to any of the Mm
  5. Record the corresponding Mm set of Computational Costs.
  6. Provide the average and 95% confidence limits of the Computational Costs divided by Mm for one of the informative Testing Environments of Table 4.

3       Computational Cost of Matching

The Computational Cost of Matching is measured by the time and the memory footprint used by the process.

The Tester shall evaluate the Computational Cost according to the following workflow:

  1. Select a set of Mu NNs trained on the training dataset.
  2. Compute the Mu fingerprints from the unmodified NNs.
  3. Evaluate the Robustness of the NN Traceability Method:
  4. Apply the fingerprint Matcher to any of the Mm
  5. Record the corresponding Mm set of Computational Costs.
  6. Provide the average and 95% confidence limits of the Computational Costs divided by Mm for one of the informative Testing Environments of Table 4.

<-Robustness Evaluation     Go to ToC      Imperceptibility Evaluation->