There is a discrepancy between the output value and the labeled value

Assuming that the model output is the expected value, the probability function of the random variable (labeled value) yi is

Since the samples are independently and identically distributed, the joint probability density function of all labeled values ​​is

log likelihood function

error sum of squares

maximum likelihood solution

Mean square error test formula

The scale of the problem is too large and it is difficult to solve the matrix

Take all samples to calculate the average gradient

average gradient

Recursion formula

Take random samples to calculate the gradient

stochastic gradient

Recursion formula

Take a small batch of samples to calculate the average gradient

average gradient

Recursion formula

The condition number of the matrix is ​​very large and the numerical stability is not good.

The nature of the large condition number of the problem

Solution

Error sum of squares J(w) hyperparameter λ constraining parameter vector w

form

The prior distribution of the weight coefficient vector w is the Bayesian "maximum posterior probability estimate" MAP under the Gaussian distribution

The output is a vector y instead of a scalar y

None of the solution coordinates are 0, and the solution is not sparse.

Most of the solution coordinates are 0, the solutions are sparse, and the processing is relatively easy.

Most of the solution coordinates are 0, the solutions are sparse, and the processing is difficult.

For the problem of minimizing the error sum of squares function, a constraint ||w||1<t is imposed

Zero-mean the data matrix X columns and normalize them to Z

Lasso solution

Cyclic coordinate descent method CCD

Part of the residual value ri(j) replaces yi

parameter estimates

Solve the sparse regression problem under 1-norm constraints

λ=0

The larger λ

Large deviation, small variance

Small deviation, large variance