Recap: correlation

Recap: correlation (cont.)

Recap: Terminologies

• Response variable: dependent variable

• Explanatory variables: independent variables, predictors, regressor variables, features (in Machine Learning)

Response variable = Model function + Random Error

• Parameter

• Statistic

• Estimator

• Estimate

Read my blogpost

In-class

In-class

Simple Linear Regression

Simple - single regressor

Linear - has a dual role here.

It may be taken to describe the fact that the relationship between Y$Y$ and X$X$ is linear. The word linear refers to the fact that the regression parameters enter in a linear fashion.

Meaning of Linear Model

| What about this?

Y=β0+β1x1+β2x2+ϵ$$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$$

| Linear or nonlinear?

Y=β0+β1x+β2x2+ϵ$$Y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+ϵ$$ | Linear or nonlinear?

Y=β0eβ1x+ϵ$$Y={\beta }_0{e}^{{\beta }_{1}x}+ϵ$$

What about this?

Y=αXβ1Xγ2Xδ3+ϵ$$Y=\alpha X_1^{\beta }{X}_2^{\gamma }{X}_3^{\delta }+ϵ$$

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

If f$f$ is approximated by a linear function

Y=β0+β1X+ϵ$$Y={\beta }_0+{\beta }_{1}X+ϵ$$

The error terms are normally distributed with mean 0$0$ and variance σ2${\sigma }^2$. Then the mean response, Y$Y$, at any value of the X$X$ is

E(Y|X=xi)=E(β0+β1xi+ϵ)=β0+β1xi$$E(Y|X=x_i)=E({\beta }_0+{\beta }_1{x}_i+ϵ)={\beta }_0+{\beta }_1{x}_i$$

For a single unit (yi,xi)$(y_i,x_i)$

yi=β0+β1xi+ϵi where ϵi∼N(0,σ2)$$y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_i\text{where}{ϵ}_i\sim N(0,{\sigma }^2)$$

We use sample values (yi,xi)$(y_i,x_i)$ where i=1,2,...n$i=1,2,...n$ to estimate β0${\beta }_0$ and β1${\beta }_1$.

The fitted regression model is

^Yi=ˆβ0+ˆβ1xi$$\widehat{Y_i}={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$$

Normal distribution

[1] 5.008403

[1] 5.082935

Normal distribution

Normal distribution

From: https://towardsdatascience.com/do-my-data-follow-a-normal-distribution-fb411ae7d832

Normal distribution

From: https://towardsdatascience.com/do-my-data-follow-a-normal-distribution-fb411ae7d832

Normal distribution

From: https://towardsdatascience.com/do-my-data-follow-a-normal-distribution-fb411ae7d832

In-class

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

In-class

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

Example: Suppose you want to model daughters' height as a function of mothers' height.

Do you think an exact (deterministic) relationship exists between these two variables?

In-class

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

Example: Suppose you want to model daughters' height as a function of mothers' height.

Do you think an exact (deterministic) relationship exists between these two variables?

Why?

In-class

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

Example: Suppose you want to model daughters' height as a function of mothers' height.

Do you think an exact (deterministic) relationship exists between these two variables?

1. Daughters' height may depend on many other variables than Mothers' height.

In-class

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

Example: Suppose you want to model daughters' height as a function of mothers' height.

Do you think an exact (deterministic) relationship exists between these two variables?

1. Daughters' height may depend on many other variables than Mothers' height.

2. Even if many variables are included in the model, it is unlikely that we can predict the daughter's height exactly. Why?

In-class

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

Example: Suppose you want to model daughters' height as a function of mothers' height.

Do you think an exact (deterministic) relationship exists between these two variables?

1. Daughters' height may depend on many other variables than Mothers' height.

2. Even if many variables are included in the model, it is unlikely that we can predict the daughter's height exactly. Why?

There will almost certainly be some variations in the model predictions that cannot be modelled, or explained.

These unexplained variances are assumed to be caused by the unexplainable random phenomena, so they can be referred to as random error.

In-class

In-class: Population Regression Line

True relationship between X and Y in the population

Y=f(X)+ϵ$$Y=f\left(X\right)+ϵ$$

If f$f$ is approximated by a linear function

Y=β0+β1X+ϵ$$Y={\beta }_0+{\beta }_{1}X+ϵ$$

The error terms are normally distributed with mean 0$0$ and variance σ2${\sigma }^2$. Then the mean response, Y$Y$, at any value of the X$X$ is

E(Y|X=xi)=E(β0+β1xi+ϵ)=β0+β1xi$$E(Y|X=x_i)=E({\beta }_0+{\beta }_1{x}_i+ϵ)={\beta }_0+{\beta }_1{x}_i$$

source: http://digfir-published.macmillanusa.com/psbe4e/psbe4e_ch10_2.html

source: https://tex.stackexchange.com/questions/347744/assumptions-for-simple-linear-regression

In-class: Population Regression Line

E(Y|X=xi)=E(β0+β1xi+ϵ)=β0+β1xi$$E(Y|X=x_i)=E({\beta }_0+{\beta }_1{x}_i+ϵ)={\beta }_0+{\beta }_1{x}_i$$

For a single unit (yi,xi)$(y_i,x_i)$

yi=β0+β1xi+ϵi where ϵi∼N(0,σ2)$$y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_i\text{where}{ϵ}_i\sim N(0,{\sigma }^2)$$

Take a sample:

The fitted regression line is

^Yi=ˆβ0+ˆβ1xi$$\widehat{Y_i}={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$$

Our example

Dashboard: https://statisticsmart.shinyapps.io/SimpleLinearRegression/

Our example (0.52, 30.7)

Dashboard: https://statisticsmart.shinyapps.io/SimpleLinearRegression/

Our example (0.582, 28.5)

Dashboard: https://statisticsmart.shinyapps.io/SimpleLinearRegression/

Our example (0.5, 32.5)

Dashboard: https://statisticsmart.shinyapps.io/SimpleLinearRegression/

Which is the best?

Which is the best?

Evaluating your answers: Fitted values

df <- alr3::heights
df$fitted <- 30.7 + (0.52*df$M)
head(df,10)

   Mheight Dheight fitted
1     59.7    55.1 61.744
2     58.2    56.5 60.964
3     60.6    56.0 62.212
4     60.7    56.8 62.264
5     61.8    56.0 62.836
6     55.5    57.9 59.560
7     55.4    57.1 59.508
8     56.8    57.6 60.236
9     57.5    57.2 60.600
10    57.3    57.1 60.496

Evaluating your answers

   Mheight Dheight fitted resid_squared
1     59.7    55.1 61.744     44.142736
2     58.2    56.5 60.964     19.927296
3     60.6    56.0 62.212     38.588944
4     60.7    56.8 62.264     29.855296
5     61.8    56.0 62.836     46.730896
6     55.5    57.9 59.560      2.755600
7     55.4    57.1 59.508      5.798464
8     56.8    57.6 60.236      6.948496
9     57.5    57.2 60.600     11.560000
10    57.3    57.1 60.496     11.532816

[1] 7511.118

Evaluating your answers

Dashboard: https://statisticsmart.shinyapps.io/SimpleLinearRegression/

How to estimate β0${\beta }_0$ and β1${\beta }_1$?

Sum of squares of Residuals

SSR=e21+e22+...+e2n$$SSR=e_1^2+e_2^2+...+e_n^2$$

Observed value

yi$y_i$

Fitted value

^Yi$\widehat{Y_i}$

^Yi=ˆβ0+ˆβ1xi$\widehat{Y_i}={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$

Residual

ei=yi−^Yi$e_i=y_i-\widehat{Y_i}$

The least-squares regression approach chooses coefficients ˆβ0${\hat{\beta }}_0$ and ˆβ1${\hat{\beta }}_1$ to minimize SSR$SSR$.

Least-squares Estimation of the Parameters

yi=β0+β1xi+ϵi, i =1, 2, 3, ...n .$$y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_i\text{, i =1, 2, 3, ...n .}$$

The least squares criterion is

S(β0,β1)=n∑i=1(yi−β0−β1xi)2.$$S({\beta }_0,{\beta }_1)=\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2.$$

Least-squares Estimation of the Parameters (cont.)

The least squares criterion is

S(β0,β1)=n∑i=1(yi−β0−β1xi)2.$$S({\beta }_0,{\beta }_1)=\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2.$$

The least-squares estimators of β0${\beta }_0$ and β1${\beta }_1$, say ^β0$\widehat{{\beta }_0}$ and ^β1,$\widehat{{\beta }_1},$ must satisfy

∂S∂β0|^β0,^β1=−2n∑i=1(yi−^β0−^β1xi)=0$$\frac{\partial S}{\partial {\beta }_0}{|}_{\widehat{{\beta }_0},\widehat{{\beta }_1}}=-2\sum _{i=1}^n(y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{x}_i)=0$$

and

∂S∂β1|^β0,^β1=−2n∑i=1(yi−^β0−^β1xi)xi=0.$$\frac{\partial S}{\partial {\beta }_1}{|}_{\widehat{{\beta }_0},\widehat{{\beta }_1}}=-2\sum _{i=1}^n(y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{x}_i)x_i=0.$$

Least-squares Estimation of the Parameters (cont.)

Simplifying the two equations yields

n^β0+^β1n∑i=1xi=n∑i=1yi,$$n\widehat{{\beta }_0}+\widehat{{\beta }_1}\sum _{i=1}^n{x}_i=\sum _{i=1}^n{y}_i,$$ ^β0n∑i=1xi+^β1n∑i=1x2i=n∑i=1yixi.

These are called least-squares normal equations.

Least-squares Estimation of the Parameters (cont.)

n^β0+^β1n∑i=1xi=n∑i=1yi,

^β0n∑i=1xi+^β1n∑i=1x2i=n∑i=1yixi.

The solution to the normal equation is

^β0=ˉy−^β1ˉx,

and

^β1=∑ni=1yixi−∑ni=1yi∑ni=1xin∑ni=1x2i−(∑ni=1xi)2n.

The fitted simple linear regression model is then

ˆY=^β0+ˆβ1x

Least-squares fit

Try this with R

library(alr3) # to load the dataset
model1 <- lm(Dheight ~ Mheight, data=heights)
model1

Call:
lm(formula = Dheight ~ Mheight, data = heights)
Coefficients:
(Intercept)      Mheight  
    29.9174       0.5417

Least-squares fit and your guesses

fit  <- 0.5417 * df$Mheight + 29.9174
sum((df$Dheight - fit)^2)

[1] 7051.97

Least square fit and your guesses

Try this with R

library(alr3) # to load the dataset
model1 <- lm(Dheight ~ Mheight, data=heights)
model1

summary(model1)

Call:
lm(formula = Dheight ~ Mheight, data = heights)
Residuals:
   Min     1Q Median     3Q    Max 
-7.397 -1.529  0.036  1.492  9.053 
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 29.91744    1.62247   18.44   <2e-16 ***
Mheight      0.54175    0.02596   20.87   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 2.266 on 1373 degrees of freedom
Multiple R-squared:  0.2408,    Adjusted R-squared:  0.2402 
F-statistic: 435.5 on 1 and 1373 DF,  p-value: < 2.2e-16

Visualise the model: Try with R

ggplot(data=heights, aes(x=Mheight, y=Dheight)) + 
  geom_point(alpha=0.5) +
  geom_smooth(method="lm", se=FALSE,
               col="blue", lwd=2) +
  theme(aspect.ratio = 1)

Least squares regression line

summary(alr3::heights)

    Mheight         Dheight     
 Min.   :55.40   Min.   :55.10  
 1st Qu.:60.80   1st Qu.:62.00  
 Median :62.40   Median :63.60  
 Mean   :62.45   Mean   :63.75  
 3rd Qu.:63.90   3rd Qu.:65.60  
 Max.   :70.80   Max.   :73.10

The LSRL passes through the point ( ˉx, ˉy), that is (sample mean of x, sample mean of y)

Least squares regression line

Extrapolation: beyond the scope of the model.

Next Lecture

More work - Simple Linear Regression, Residual Analysis, Predictions