WayneLogistic regression is a data analysis technique used for classification. It is probably the simplest model among classification algorithms. Therefore, it is very suitable as the first classification algorithm for beginners.
Table of Contents
Sigmoid Function
Before introducing logistic regression, let’s first look at the sigmoid function (also known as logistic function). The formula of Sigmoid function is as follows. Its output is between 0 and 1.
As shown in the figure below, the two ends of the sigmoid function will be very close to 0 and 1.
Logistic Regression Model
The model of logistic regression is to first pass x into the model of linear regression to obtain y. Then pass y into the sigmoid function to obtain a value between 0 and 1. The formula of the logistic regression model is as follows.
Therefore, the output value of the logistic regression model is always between 0 and 1. Assume that x is the size of the tumor, when y is 0, it means a non-malignant tumor, and when y is 1, it means a malignant tumor. If, we pass x into a logistic regression model and obtain 0.7, this means there is a 70% chance that y is 1.
Because the probability that y is 0 plus the probability that y is 1 sums to 1, so when there is a 70% probability that y is 1, that means there is a 30% probability that y is 0.
Decision Boundary
As mentioned just now, the output value of logistic regression is between 0 and 1, which represents the probability of y being 1. Therefore, we need to set a threshold. When fw,b is greater than or equal to threshold, ŷ is 1; conversely, when fw,b is less than threshold, ŷ is 0.
Looking carefully, when z is greater than or equal to 0, fw,b, that is, g(z) will be greater than or equal to threshold 0.5. So, the left side of z = 0 is ŷ = 0, and the right side of z = 0 is ŷ = 1. This boundary is called the design boundary.
In the example below, we can calculate the boundary decision by z = 0.
Cost Function and Loss Function
Cost function is used to measure the accuracy of fw,b . With the cost function, we can measure whether the adjusted sum w is b better than the original.
The cost function of Logistic regression is as follows:
Where L is the loss function as follows:
We can combine the two expressions in the loss function into one, as follows.
The following is the cost function after the loss function is expanded.
![J(\vec{w},b)=-\frac{1}{m}\sum^{m}_{i=1} \left[ y^{(i)}log(f_{\vec{w},b}(\vec{x}^{(i)})) + (1-y^{(i)})log(1-f_{\vec{w},b}(\vec{x}^{(i)})) \right]](https://s0.wp.com/latex.php?latex=J%28%5Cvec%7Bw%7D%2Cb%29%3D-%5Cfrac%7B1%7D%7Bm%7D%5Csum%5E%7Bm%7D_%7Bi%3D1%7D+%5Cleft%5B+y%5E%7B%28i%29%7Dlog%28f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29%29+%2B+%281-y%5E%7B%28i%29%7D%29log%281-f_%7B%5Cvec%7Bw%7D%2Cb%7D%28%5Cvec%7Bx%7D%5E%7B%28i%29%7D%29%29+%5Cright%5D+&bg=ffffff&fg=000&s=1&c=20201002)
Gradient Descent
The following is the gradient descent algorithm of logistic regression.
The derivative of the cost function is calculated as follows:
The derivative calculation method of the cost function of logistic regression looks the same, but in fact their fw,b is different.
Conclusion
Logistic regression models are very similar to linear regression. Therefore, before learning logistic regression, you should learn linear regression first.
Reference
- Andrew Ng, Machine Learning Specialization, Coursera.
Wayne
