Compress input feature values ​​in a wide range to between 0 and 1, so that the data amplitude can be maintained without major changes in deep networks

Closest to biological neurons in a physical sense

Depending on its output range, this function is suitable for models that have predicted probabilities as output

When the input is very large or very small, the output is basically constant, that is, the change is very small, which causes the gradient to be close to 0.

Gradients may disappear prematurely, resulting in slower convergence

Exponential operations are relatively time-consuming

The output is not 0-mean, which causes the neurons in the next layer to get the non-0-mean signal output by the previous layer as input. As the network deepens, the distribution trend of the original data will change.

Solve the problem that the output of the above Sigmoid function is not 0 mean

The derivative of the Tanh function ranges from 0 to 1, which is better than the 0 to 0.25 of the sigmoid function, which alleviates the problem of vanishing gradients to a certain extent.

The Tanh function is similar to the y=x function near the origin. When the input activation value is low, matrix operations can be performed directly, and training is relatively easy.

Similar to the Sigmoid function, the vanishing gradient problem still exists

Observe its two forms of expressions, namely 2*sigmoid(2x)-1 and (exp(x)-exp(-x))/(exp(x) exp(-x)). It can be seen that the problem of power operation still exists

Compared with the sigmoid function and the Tanh function, when the input is positive, the Relu function does not have a saturation problem, which solves the gradient vanishing problem and makes the deep network trainable.

The calculation speed is very fast, you only need to determine whether the input is greater than 0 value

The convergence speed is much faster than sigmoid and Tanh functions

Relu output will cause some neurons to have a value of 0, which not only brings network sparsity, but also reduces the correlation between parameters, which alleviates the problem of overfitting to a certain extent;

The output of the Relu function is not a function with 0 as the mean.

There is a Dead Relu Problem, that is, some neurons may never be activated, causing the corresponding parameters to never be updated. The main reasons for this problem include parameter initialization problems and learning rate settings that are too large;

When the input is a positive value and the derivative is 1, in the "chain reaction", the gradient will not disappear, but the strength of the gradient descent depends entirely on the product of the weights, which may lead to the gradient explosion problem

In response to the Dead Relu Problem that exists in the Relu function, the Leaky Relu function gives the input value a very small slope when the input is a negative value. On the basis of solving the 0 gradient problem in the case of negative input, it is also well alleviated. Dead Relu issue

The output of this function is from negative infinity to positive infinity, that is, leaky expands the range of the Relu function, where the value of α is generally set to a smaller value, such as 0.01

Theoretically, this function has better effects than the Relu function, but a large amount of practice has proved that its effect is unstable, so there are not many applications of this function in practice.

Inconsistent results due to different functions applied in different intervals will result in the inability to provide consistent relationship predictions for positive and negative input values.

The first step of softmax is to transform the prediction results of the model into an exponential function, thus ensuring the non-negative nature of the probability.

The method is to divide the converted results by the sum of all converted results, which can be understood as the percentage of the converted results in the total. This gives approximate probabilities.

Cross entropy can be used as a loss function in neural networks (machine learning). p represents the distribution of real labels, and q is the predicted label distribution of the trained model. The cross entropy loss function can measure the similarity between p and q.

Another benefit of cross entropy as a loss function is that using the sigmoid function during gradient descent can avoid the problem of reduced learning rate of the mean square error loss function, because the learning rate can be controlled by the output error.

Consider p(i) as the real probability distribution and q(i) as the predicted probability distribution. If we use cross entropy as the loss function, when we minimize it, we can make q(i) gradually approach p( i), the purpose of fitting is achieved.

Take out certain predicted values ​​individually or assign parameters of different sizes

Multi-objective training tasks, setting reasonable loss combination methods (various operations)

Different neural network losses are combined, and the common loss is used to train and guide the network.

Fixed, that is, fixed learning rate, is the simplest configuration and requires only one parameter.

The learning rate remains unchanged during the entire optimization process. This is a very rarely used strategy, because as it approaches the global optimal point, the learning rate should become smaller and smaller to avoid skipping the optimal point.

adaptive learning rate

It can be seen from the AdaGrad algorithm that as the algorithm continues to iterate, r will become larger and larger, and the overall learning rate will become smaller and smaller. Therefore, generally speaking, the AdaGrad algorithm starts with incentive convergence, and then slowly turns into penalty convergence, and the speed becomes slower and slower.

The RMSProp algorithm does not violently and directly accumulate square gradients like the AdaGrad algorithm, but adds an attenuation coefficient to control how much historical information is obtained.

To put it simply, after setting the global learning rate, for each pass, the global learning rate is divided parameter by parameter by the square root of the square sum of the historical gradients controlled by the attenuation coefficient, so that the learning rate of each parameter is different.

The direction is different and the search is more accurate.

The difference between Dropout and Pooling

During forward propagation, we let the activation value of a certain neuron stop working with a certain probability p, which can make the model more generalizable because it will not rely too much on certain local features.

What effect does Regularization have on the parameter w?

What is weight decay, and how is it related to Regularization?

Most parameters do not need to be updated, and the actual parameters are greatly reduced.

Freeze part of the convolutional layers of the pre-trained model (usually the majority of the convolutional layers close to the input, since these layers retain a lot of underlying information) or even freeze any network layers, and train the remaining convolutional layers (usually the parts close to the output convolutional layer) and fully connected layer.

The principle of fine-tuning is to use the known network structure and known network parameters, modify the output layer to our own layer, and fine-tune the parameters of several layers before the last layer, thus effectively utilizing the powerful generalization capabilities of deep neural networks. fine tuning capabilities, and eliminates the need to design complex models and time-consuming training, so fine tuning is a more suitable choice when the amount of data is insufficient.

significance

migration model