WayneWhen a model has poor performance, it cannot predict the data accurately. The main cause may be overfitting or underfitting. If it is a case of overfitting, we can use regularization to solve model overfitting.
Table of Contents
Regularization
Regularization is a method used to prevent overfitting. Suppose there is an overfitted model as shown on the lower left. We can make w3 and w4 very small or tend to 0 to reduce the impact of x3 and x4 on the model, so that the model becomes simpilier, as shown on the lower right. This is the basic idea of regularization.
Gradient descent will find the minimum value in the cost function. In the cost function, if we append 
and 
, this will cause and to be small or tend to 0 in the minimum value found by gradient descent. Therefore, by modifying the cost function, we can reduce the impact of and to the model during training.
![min_{\vec{w},b} \left[ \frac{1}{2m} \sum_{i=1}^m (f_{\vec{w},b}(\vec{x}^{(i)})-y^{(i)})^2) + 1000 w_3^2 + 1000 w_4^2 \right]](https://s0.wp.com/latex.php?latex=min_%7B%5Cvec%7Bw%7D%2Cb%7D+%5Cleft%5B+%5Cfrac%7B1%7D%7B2m%7D+%5Csum_%7Bi%3D1%7D%5Em+%28f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29-y%5E%7B%28i%29%7D%29%5E2%29+%2B+1000+w_3%5E2+%2B+1000+w_4%5E2+%5Cright%5D+&bg=ffffff&fg=000&s=1&c=20201002)
The following formula is the cost function with regularization added. The formula added at the end is called regularization term, and λ is called regularization parameter. If λ is set to a very large value, such as 1010, then all W will tend to 0. Therefore, we can reduce W by adjusting λ.
Regularized Linear Regression
Regularized linear regression is the cost function of linear regression with regularization term, as follows.
![min_{\vec{w},b} J(\vec{w},b) = min_{\vec{w},b} \left[ \frac{1}{2m} \sum_{i-1}^m (f_{\vec{w},b}(\vec{x}^{(i)})-y^{(i)})^2 + \frac{\lambda}{2m} \sum_{j=1}^n w_j^2 \right]](https://s0.wp.com/latex.php?latex=min_%7B%5Cvec%7Bw%7D%2Cb%7D+J%28%5Cvec%7Bw%7D%2Cb%29+%3D+min_%7B%5Cvec%7Bw%7D%2Cb%7D+%5Cleft%5B+%5Cfrac%7B1%7D%7B2m%7D+%5Csum_%7Bi-1%7D%5Em+%28f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29-y%5E%7B%28i%29%7D%29%5E2+%2B+%5Cfrac%7B%5Clambda%7D%7B2m%7D+%5Csum_%7Bj%3D1%7D%5En+w_j%5E2++%5Cright%5D+&bg=ffffff&fg=000&s=1&c=20201002)
The Gradient descent algorithm is as follows.
After we expand the derivative part, it becomes the following formula.
![\text{repeat \{} \\\\ \phantom{xxxx} w_j=w_j-\alpha \left[ \frac{1}{m} \sum_{i=1}^m \left[ (f_{\vec{w},b}(\vec{x}^{(i)})-y^{(i)})x_j^{(i)} \right] + \frac{\lambda}{m} w_j \right] \\\\ \phantom{xxxx} b=b-\alpha \frac{1}{m} \sum_{i=1}^m (f_{\vec{w},b}(\vec{x}^{(i)})-y^{(i)}) \\\\ \text{\}}](https://s0.wp.com/latex.php?latex=%5Ctext%7Brepeat+%5C%7B%7D+%5C%5C%5C%5C+%5Cphantom%7Bxxxx%7D+w_j%3Dw_j-%5Calpha+%5Cleft%5B+%5Cfrac%7B1%7D%7Bm%7D+%5Csum_%7Bi%3D1%7D%5Em+%5Cleft%5B+%28f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29-y%5E%7B%28i%29%7D%29x_j%5E%7B%28i%29%7D+%5Cright%5D+%2B+%5Cfrac%7B%5Clambda%7D%7Bm%7D+w_j+%5Cright%5D+%5C%5C%5C%5C+%5Cphantom%7Bxxxx%7D+b%3Db-%5Calpha+%5Cfrac%7B1%7D%7Bm%7D+%5Csum_%7Bi%3D1%7D%5Em+%28f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29-y%5E%7B%28i%29%7D%29+%5C%5C%5C%5C+%5Ctext%7B%5C%7D%7D+&bg=ffffff&fg=000&s=1&c=20201002)
The simplified expression of wj becomes as follows. It can be clearly seen that we can reduce wj by adjusting λ.
Regularized Logistic Regression
The cost function of logistic regression plus the regularization term will become the following formula.
![z=w_1x_1+w_2x_2+w_3x_1^2x_2+w_4x_1^2x_2^2+w_5x_1^2x_2^3+ \cdots +b \\\\ f_{\vec{w},b}(\vec{x})=\frac{1}{1+e^{-z}} \\\\ J(\vec{w},b)=-\frac{1}{m} \sum_{i=1}^m \left[ y^{(i)} log(f_{\vec{w},b}(\vec{x}^{(i)})) + (1-y^{(i)}) logt(1-f_{\vec{w},b}(\vec{x}^{(i)})) \right] \\\\ \phantom{xxxxxxxx} + \frac{\lambda}{2m} \sum_{i=1}^n w_j^2](https://s0.wp.com/latex.php?latex=z%3Dw_1x_1%2Bw_2x_2%2Bw_3x_1%5E2x_2%2Bw_4x_1%5E2x_2%5E2%2Bw_5x_1%5E2x_2%5E3%2B+%5Ccdots+%2Bb+%5C%5C%5C%5C+f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%29%3D%5Cfrac%7B1%7D%7B1%2Be%5E%7B-z%7D%7D+%5C%5C%5C%5C+J%28%5Cvec%7Bw%7D%2Cb%29%3D-%5Cfrac%7B1%7D%7Bm%7D+%5Csum_%7Bi%3D1%7D%5Em+%5Cleft%5B+y%5E%7B%28i%29%7D+log%28f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29%29+%2B+%281-y%5E%7B%28i%29%7D%29+logt%281-f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29%29+%5Cright%5D+%5C%5C%5C%5C+%5Cphantom%7Bxxxxxxxx%7D+%2B+%5Cfrac%7B%5Clambda%7D%7B2m%7D+%5Csum_%7Bi%3D1%7D%5En+w_j%5E2+&bg=ffffff&fg=000&s=1&c=20201002)
The Gradient descent algorithm is as follows.
After expanding the derivative part, it becomes the following formula. It looks exactly the same as regularized linear regression, but it should be noted that f w, b in the formula are logistic regression.
Conclusion
Regularization can reduce the size of parameters to solve overfitting. When the parameters are larger, the penalty will be larger, that is, it will be reduced a lot at a time.
Reference
- Andrew Ng, Machine Learning Specialization, Coursera.
Wayne
