Convolutional neural networks with dynamic regularization

Convolutional Neural Networks with Dynamic Regularization: chap1 Problem Background: Dynamic regularization was proposed to deal with two main shortcomings of regularization methods: 1) the regularization strength (or amplitude) is inflexible for different network architectures 2) The regularization strength is constant throughout the training process.

Edited at 2023-01-09 20:02:07

PlotWizard

Recent works View more works>>

Convolutional neural networks with dynamic regularization

PlotWizard

Recent works View more works>>

Recommended to you
Outline

Detailed explanation of the operation process of neural network
- 35
PlotWizard
Convolutional Neural Network (CNN)
- 104
PlotWizard
Common types and applications of neural network models
- 33
- 1
PlotWizard
Neural Networks and Deep Learning Recurrent Neural Networks
- 61
PlotWizard
Neural Networks and Deep Learning Convolutional Neural Networks
- 46
PlotWizard
The foundation of neural networks and deep learning
- 112
PlotWizard
Handwritten digit recognition based on improved Sigmoid convolutional neural network
- 42
PlotWizard
traditional neural network
- 20
PlotWizard
Neural Networks
- 1.0k
- 1
- 1
PlotWizard
Generate model
- 20
PlotWizard

(Outside Q1 2021) Convolutional Neural Networks with Dynamic Regularization

chap1 problem background

Dynamic regularization was proposed to deal with two main shortcomings of regularization methods:

1) Regularization strength (or amplitude) is inflexible for different network architectures 2) Regularization strength is constant throughout the training process

chap2 PROPOSED METHOD

1) Introducing dynamic regularization into the Res structure

2-branch Res block with dynamic regularization, representing backward difference of training loss

① The dynamic regularization unit is embedded in the Res branch of the Res block:

A is the constant amplitude, si is the dynamic factor at the ith iteration, and r is the uniform random noise.

② The regularization amplitude is related to , r∈ [−R, R],

, is the total number of Res blocks

③ In the forward pass, the output of the (l 1)th Res block can be expressed as:

In the backward pass, the noise is related to

2) Update of regularization strength

The proposed updating solution of dynamic regularization strength is achieved through the dynamics of the training loss. In particular, the dynamics of training losses can be modeled as the backward difference between training losses in consecutive iterations:

represents the training loss at the i-th iteration

To eliminate fluctuations in the input of consecutive mini-batches, apply a Gaussian filter to smooth them:

chap3 experimental results

2 Branch structure and dense structure

Experimental results on dense link structures show that our dynamic regularization is stable and reduces the Top-1 error by 1.67%

3 branch structure

The results in Table I and Table II show that the proposed dynamic regularization can be adapted to various network architectures in comparison with the then latest regularization methods Shake-Shake, Shake-Drop, and DropBlok on 2-branch and 3-branch structures. Compared with the baseline, the method proposed in this article can reduce the error by more than 2% on average.

chap4 ablation study and discussion

1) Effectiveness of dynamic regularization

2) List of regularization strengths

3) Random noise

4) Gaussian filter