Hendra Bunyamin

Forgiven sinner and Lecturer at Maranatha Christian University

Building Logistic Regression Model from Linear Regression Model

A model called logistic regression is introduced in Week 3 of Machine Learning course taught by Andrew Ng. This post tries to explain how to obtain logistic regression model from linear regression model which has been explained in Week 1 and Week 2 of the course. Therefore, I recommend all readers for studying Week 1 and Week 2 before reading this post.

Week 1 of the course shows that linear regression model has the capability to solve classification problem with prediction model (hypothesis) as follows:

with is called parameter model and is a test example whose we would like to predict.

The question arises as follows:

How can we derive logistic regression model from linear regression model?

As we already know that $h_\theta(x)$ in logistic regression has a range of values between $0$ and $1$; on the other hand, $h_\theta(x)$ in linear regression has a range of values between $-\infty$ and $\infty$. Therefore, $h_\theta(x)$ which belongs to logistic regression needs to be converted into $-\infty < h_\theta(x) < \infty$. Moreover, after the conversion is done, the new $h_\theta(x)$ shall be substituted into Equation \eqref{eq:linear-regression}.

Let us start by converting $h_\theta(x)$ which belongs to logistic regression and then substitute the new $h_\theta(x)$ into Equation \eqref{eq:linear-regression}.

$\frac{h_\theta(x)}{1-h_\theta(x)}$ in Equation \eqref{eq:odds-ratio} is called odds ratio and $\log \left( \frac{h_\theta(x)}{1-h_\theta(x)} \right)$ in Equation \eqref{eq:logit} is named as logit function (Raschka & Mirjalili, 2017).

Now that the new $h_\theta(x)$ has the range of values between $-\infty < h_\theta(x) < \infty$, this new $ h_\theta(x) $ can be substituted into Equation \eqref{eq:linear-regression} as follows:

Accordingly, logistic regression model that we have derived is

Logistic regression model in Equation \eqref{eq:sigmoid} is popularly also called sigmoid function. Finally, we have successfully derived logistic regression model (Equation \eqref{eq:sigmoid}) from linear regression model (Equation \eqref{eq:linear-regression}).

References

Raschka, S. and Mirjalili, V. (2017) Python Machine Learning Second Edition. Packt Publishing Ltd. Page 59-60.


Written on August 17, 2019