1. Introduction

This chapter specifies:

  1. The steps required to implement the pruning method that enables a User to produce a Up-sampling network with reduced complexity starting from a fine-tuned model produced based on the procedure specified in Design Procedure.
  2. The weights reduced-complexity neural networks referenced in UFV Filter Specification.

The Pruning Algorithm specified below in natural language and pseudo-code is applied to the Fine-Tuned or Pre-trained Model (in the following “Model”). The Pruning Algorithm also requires a Pruning Dataset, the one used in the Design Procedure.

2. Pruning Algorithm

2,.1 Natural language specification

Channel:

Dependency Graph

Pruning Groups

Importance: the average of all Parameters of the Channel.

Initial Number of Parameters: the number of parameters of the unpruned Model.

Model Recovery Phase: a training procedure applied after Pruning to recover part of the lost performance in the Pruning Algorithm.

Pruning Target: the percentage of the Model parameters to be remove.

Performance Criterion: the percentage the performance of the unpruned Model considered as acceptable in the final pruned Model.

  1. Set the Performance Criterion that the pruned network should achieve.
  2. Compute the Dependency Graph and Pruning Groups.  
  3. Starting from the non-pruned network, execute a set of Steps numbered from 1 to N (total number of Steps) until the Pruned Model satisfies the Performance Criterion:
    1. At the i-th Step, set the Pruning Target, i.e., the percentage of weightsof the Initial Number of Parameters.
    2. Apply Sparsity Learning to the Model.
    3. Evaluate the Importance of each Channel in each Layer of the Model.
    4. Remove the Channels until the Pruning Target is met starting from the one with the lowest Importance.
    5. Execute the the Model Recovery Phase.
  4. Save the Pruned Model that has reached the Perfarmance Target.

2.2 Pseudo programming specification

# dependecy graph computation

graph = dependecy_graph(MODEL)

# starting pruning process

for i in range(PRUNING_STEP):

    step_pruning_rateo = TARGET_PRUNING_RATEO * ((i+1)/PRUNING_STEP)

       # model pruning

    model = sparsity pruning(model,

    DATASET.train, GROWING_REG_EPOCHS)

 # remove the weights which have lowest Norm2 valu

    model = channel_prune_norm2(model,

     step_pruning_rateo, graph)

    model_tmp = model

    best_error = calc_error(model, DATASET.val) Come e’ calcolato?

# retain the pruned model with best validation error

    for e in range(RETRAIN_EPOCHS):

        model_tmp = train_one_epoch(model, DATASET.train)

        curr_error = calc_error(model_tmp, DATASET.val)

        if best_error < curr_error:

            model = model_tmp

            best_error = curr_error

# Output

model_output = model

The Reference Software of the Pruning Algorithm is available at Git.