<-Procedures      Go to ToC      Evaluations ->

This section introduces the NN models and relevant datasets for the evaluation of traceability technologies in specific application domains and the evaluation methods.

1       NN models and datasets

For performance evaluation the following NN-based image processing models, datasets and application fields shall be used:

  1. Image Classification for two NNs architecture: VGG16 [14] and ResNet8 [13] on the CIFAR10 [15] dataset using the Top-1 Accuracy metric.
  2. Generative image task for one NN architecture: Pix2pix [9] on Cityscapes [8] dataset using PSNR and SSIM metrics.
  3. Up-sampling for one NN architecture: RDN [10] on Div2K [11] dataset using PSNR and SSIM metrics.
  4. Semantic segmentation for one NN architecture: DeepLabV3 [12] model on Cityscapes [8] dataset using unweighted mean Intersection over Union (mIoU) metric.

Additional tasks, NN models, performance criteria or datasets can be added in the future.

For each combination of datasets and NN models Table 1 provides the performance measure used for the Imperceptibility evaluation.

Table 1. NN models, datasets and performance measure.

Datasets
CIFAR10 Cityscapes Div2K
NN model VGG16 Top-1 accuracy – –
ResNet8 Top-1 accuracy – –
Pix2pix – PSNR and SSIM –
RDN – – PSNR and SSIM
DeepLab – mIoU –

For Robustness and Computational Cost Evaluation the combination of datasets and NN model remains but the metric depends on the Traceability Technology Evaluation.

1.2       Evaluation types

Active traceability technologies are evaluated in terms of three characteristics:

  • Imperceptibility,
  • Robustness,
  • Computational cost.

Passive traceability technologies are evaluated in terms of two characteristics:

  • Robustness,
  • Computational cost.

Each of these three characteristics is evaluated using the methodology standardised in [2].

 

<-Procedures      Go to ToC      Evaluations ->