Compared

An input layer, an output layer and multiple hidden layers

Sigmoid

ReLU

Tanh

Leaky ReLU

Normalize the output results

Convert output results into probabilities

Used to measure the difference from the real value (one-hot code) - KL divergence

step

Common door units

gradient disappears

gradient explosion

Solution

not too good

Use Gaussian distribution

The variance of the activation values ​​of neurons in each layer is basically the same.

A good initialization method can prevent information from disappearing during forward propagation and can also solve the problem of gradient disappearance during reverse propagation.

When selecting hyperbolic tangent or Sigmoid as the activation function, it is recommended to use the Xaizer initialization method.

When selecting ReLU or Leakly ReLU as the activation function, it is recommended to use the He initialization method.

Adjust the weight distribution so that the input and output have the same distribution

Adjust the y output after batch training - subtract the mean to remove the variance

Among them, the mean and variance of the data distribution need to be determined independently according to the contribution to classification.

Solve the problem of signal disappearance and gradient disappearance during forward pass

When the model ability decreases on the training set and begins to increase on the validation set, it begins to overfit.

When learning, the selected model contains too many parameters, resulting in good predictions for known data but poor predictions for unknown data.

Usually the training data is memorized rather than the data features learned.

solution

The model description ability is too weak to learn the patterns in the data well.

Usually the model is too simple

is too big

Too big

too small

Moderate

grid search method

Random search method

Generally, random sampling is done in log space.