Recap

1. Exploratory data analysis (scatterplot, correlation)

2. Fit a regression model

3. Check the validity of the assumptions/ Residual analysis

4. Check $R_{adj}^2$

5. Hypothesis testing: ANOVA

   • Test the significance of regression

6. Hypothesis testing: t-test

   • Tests on individual regression coefficients

7. Interpret point estimates of coefficients

8. Compute confidence intervals on the regression coefficients and mean response and interpret the results

9. Prediction of new observations

Recap

1. Exploratory data analysis

2. Fit a regression model

3. Check the validity of the assumptions/ Residual analysis

4. Check $R_{adj}^2$

5. Hypothesis testing: ANOVA

   • Test the significance of regression

6. Hypothesis testing: t-test

   • Tests on individual regression coefficients

7. Interpret point estimates of coefficients

8. Compute confidence intervals on the regression coefficients and mean response and interpret the results

9. Prediction of new observations

Confidence Intervals in Multiple Regression

1. Confidence intervals on the regression coefficients

2. Confidence interval estimation of the mean response at a given point

Confidence Intervals on the Regression Coefficients

To construct confidence intervals for regression coefficients ( ${\beta }_j$ , $j=0,1,...p$) we will continue to assume that,

• errors ${ϵ}_i$ are normally and independently distributed with mean zero and variance ${\sigma }^2$.

Hence, before constructing the confidence intervals you need to check the validity of the assumptions.

Confidence Intervals on the Regression Coefficients

Data set

library(tidyverse)
heart.data <- read_csv("heart.data.csv")
heart.data

# A tibble: 498 x 4
      X1 biking smoking heart.disease
   <dbl>  <dbl>   <dbl>         <dbl>
 1     1  30.8    10.9          11.8 
 2     2  65.1     2.22          2.85
 3     3   1.96   17.6          17.2 
 4     4  44.8     2.80          6.82
 5     5  69.4    16.0           4.06
 6     6  54.4    29.3           9.55
 7     7  49.1     9.06          7.62
 8     8   4.78   12.8          15.9 
 9     9  65.7    12.0           3.07
10    10  35.3    23.3          12.1 
# … with 488 more rows

Confidence Intervals on the Regression Coefficients (cont.)

regHeart <-  lm(heart.disease ~ biking+ smoking, data=heart.data)
regHeart

Call:
lm(formula = heart.disease ~ biking + smoking, data = heart.data)
Coefficients:
(Intercept)       biking      smoking  
    14.9847      -0.2001       0.1783

Validity of the assumptions: All satisfied. We discussed in Week 8

Compute 95% confidence intervals for regression coefficients

confint(regHeart, level=0.95)

                 2.5 %     97.5 %
(Intercept) 14.8272075 15.1421084
biking      -0.2028166 -0.1974495
smoking      0.1713800  0.1852878

Compute 95% confidence intervals for regression coefficients

confint(regHeart, level=0.95)

                 2.5 %     97.5 %
(Intercept) 14.8272075 15.1421084
biking      -0.2028166 -0.1974495
smoking      0.1713800  0.1852878

Compute 90% confidence intervals for regression coefficients

confint(regHeart, level=0.90)

                   5 %       95 %
(Intercept) 14.8525973 15.1167186
biking      -0.2023839 -0.1978822
smoking      0.1725014  0.1841665

Interpretation of confidence intervals for regression coefficients

confint(regHeart, level=0.95)

                 2.5 %     97.5 %
(Intercept) 14.8272075 15.1421084
biking      -0.2028166 -0.1974495
smoking      0.1713800  0.1852878

Interpretation of confidence intervals for regression coefficients

                 2.5 %     97.5 %
(Intercept) 14.8272075 15.1421084
biking      -0.2028166 -0.1974495
smoking      0.1713800  0.1852878

Intercept: 95% Confidence Interval [14.82, 15.14]

This means that if $X_1$ (biking) and $X_2$ (smoking) remain at zero, we are 95% confidence that the mean percentage of people with heart disease is between 14.82% and 15.14%.

${\beta }_1$: 95% Confidence Interval [-0.20, -0.19]

This means that if $X_2$ (smoking) remains fixed, we are 95% confidence that an one percent increase in biking is associated with a decrease in the mean percentage of people with heart disease at least 0.19 percent and not more than 0.20 percent.

${\beta }_2$ : 95% Confidence Interval [0.17, 0.19]

This means that if $X_1$ (biking) remains fixed, we are 95% confidence that an one percent increase in smoking is associated with an increase in the mean percentage of people with heart disease at least 0.17 percent and not more than 0.18 percent.

Confidence Interval Estimation of the Mean Response at a Given Point

$$Y={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+ϵ$$

where,

$Y$ - percentage of people with heart disease,

$X_1$ - percentage of people in each town who bike to work,

$X_2$ - percentage of people in each town who smoke

Fitted regression model

regHeart

Call:
lm(formula = heart.disease ~ biking + smoking, data = heart.data)
Coefficients:
(Intercept)       biking      smoking  
    14.9847      -0.2001       0.1783

$\hat{Y}=14.9847-0.2001X_1+0.1783X_2$, where $\hat{Y}$ - Fitted values.

Confidence Interval Estimation of the Mean Response at a Given Point

Suppose we have an observation $X_1=30.8$ and $X_2=10.9$ and we would like to find a 95% confidence interval on the percentage of people with heart disease

The fitted value at this point is:

$\hat{Y}=14.9847-0.2001X_1+0.1783X_2$

$\hat{Y}=14.9847-0.2001\left(30.8\right)+0.1783\left(10.9\right)=10.764$

A 95% confidence interval on the mean percentage of people with heart disease at this point is:

predict(regHeart, list(biking = 30.8, smoking = 10.9),
interval='confidence', level=0.95)

      fit      lwr      upr
1 10.7644 10.69625 10.83255

Interpretation:

We can be 95% confident that the mean percentage of people with heart disease of all towns at $X_1\left(biking\right)=30.8$ and $X_2\left(smoking\right)=10.9$ is between 10.69 and 10.83 percent.

Prediction of New Observation

Suppose we want to predict construct 95% prediction interval on the percentage people with heart disease at $X_1=60$ and $X_2=20$.

predict(regHeart, list(biking = 60, smoking = 20),
interval='predict', level=0.95)

       fit      lwr      upr
1 6.543353 5.255293 7.831413

We can be 95% confident that the percentage of people with heart disease at a town at $X_1=60$ and $X_2=20$ will be between 5.25 and 7.83 percent.

Prediction of the mean response

A 95% confidence interval on the mean percentage of people with heart disease at the point at $X_1=30.8$ and $X_2=10.9$.

predict(regHeart, list(biking = 30.8, smoking = 10.9),
interval='confidence', level=0.95)

      fit      lwr      upr
1 10.7644 10.69625 10.83255

Prediction of a future value

Suppose we want to predict construct 95% prediction interval on the percentage people with heart disease at $X_1=60$ and $X_2=20$.

predict(regHeart, list(biking = 60, smoking = 20),
interval='predict', level=0.95)

       fit      lwr      upr
1 6.543353 5.255293 7.831413

Prediction of the mean response

what would be the average (mean) response with characteristics $X_1=30.8$ and $X_2=10.9$ ?

predict(regHeart, list(biking = 30.8, smoking = 10.9),
interval='confidence', level=0.95)

      fit      lwr      upr
1 10.7644 10.69625 10.83255

We predict the mean value of $Y$ with characteristics $X_1=30.8$ and $X_2=10.9$.

Prediction of a future value

what is the predicted value of $Y$ with characteristics $X_1=60$ and $X_2=20$?

predict(regHeart, list(biking = 60, smoking = 20),
interval='predict', level=0.95)

       fit      lwr      upr
1 6.543353 5.255293 7.831413

We predict $Y$ for a specific new case that comes from the population with characteristics $X_1=60$ and $X_2=20$.

Prediction interval for a new response.

Interpretations

Prediction of the mean response

      fit      lwr      upr
1 10.7644 10.69625 10.83255

We can be 95% confident that the mean percentage of people with heart disease of all towns at $X_1\left(biking\right)=30.8$ and $X_2\left(smoking\right)=10.9$ is between 10.69 and 10.83 percent.

Prediction of a future value

       fit      lwr      upr
1 6.543353 5.255293 7.831413

We can be 95% confident that the percentage of people with heart disease at a town at $X_1=60$ and $X_2=20$ will be between 5.25 and 7.83 percent.

Prediction of Set of New Observations

newheartdata <- data.frame(biking = c(30, 40, 40, 60),
                           smoking = c(20, 30, 12, 10))
newheartdata

  biking smoking
1     30      20
2     40      30
3     40      12
4     60      10

predict(regHeart, newdata=newheartdata , interval="predict")

        fit       lwr       upr
1 12.547345 11.260464 13.834225
2 12.329353 11.039054 13.619652
3  9.119343  7.832795 10.405891
4  4.760014  3.471742  6.048286

Mathematical Formula: Least-square estimator

$Y={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+...+{\beta }_p{X}_p+ϵ$

$$Y=\left[\begin{array}{c}{Y}_1\\ Y_2\\ .\\ .\\ .\\ Y_n\end{array}\right]$$

$$X=\left[\begin{array}{c}1,x_{11},x_{12},...,x_{1p}\\ 1,x_{21},x_{22},...,x_{2p}\\ .\\ .\\ .\\ 1,x_{n1},x_{n2},...,x_{np}\end{array}\right]$$

$$\hat{\beta }=(X^{\prime }X{)}^{-1}{X}^{\prime }Y$$

Mathematical Formula

Confidence intervals on the regression coefficients

[ $\widehat{{\beta }_j}-t_{\alpha /2,n-p}\sqrt{{\hat{\sigma }}^2{C}_{jj}}$, $\widehat{{\beta }_j}+t_{\alpha /2,n-p}\sqrt{{\hat{\sigma }}^2{C}_{jj}}$ ]

$C_{jj}$ is the $j$th diagonal element of the $(X^{\prime }X{)}^{-1}$

Unbiased estimator for ${\sigma }^2$ is given by

${\hat{\sigma }}^2$ = MSE

Mathematical Formula

Prediction of the mean response Confidence interval for:

Mean Response at $x_{01},x_{02},..,x_{0p}$, $E[Y|X_1=x_{01},X_2=x_{02}...,X_p=x_{0p}]={\mu }_{Y|X_1=x_{01},X_2=x_{02}...,X_p=x_{0p}}$

Fitted value at $x_{01},x_{02},..,x_{0p}$

$[x_0{]}^{\prime }$ = $[1,x_{01},x_{02},...,x_{0k}]$

${\hat{y}}_0=x_0^{\prime }\hat{\beta }$

[ $\widehat{y_0}-t_{\alpha /2,n-p}\sqrt{{\hat{\sigma }}^2{x}_0^{\prime }(X^{\prime }X{)}^{-1}{x}_0}$, $\widehat{y_0}+t_{\alpha /2,n-p}\sqrt{{\hat{\sigma }}^2{x}_0^{\prime }(X^{\prime }X{)}^{-1}{x}_0}$ ]

Mathematical Formula (cont.)

Prediction of a future value

[ $\widehat{y_0}-t_{\alpha /2,n-p}\sqrt{{\hat{\sigma }}^2(1+x_0^{\prime }(X^{\prime }X{)}^{-1}{x}_0})$, $\widehat{y_0}+t_{\alpha /2,n-p}\sqrt{{\hat{\sigma }}^2(1+x_0^{\prime }(X^{\prime }X{)}^{-1}{x}_0})$ ]