Simple Linear Regression

Assumes there is approximately a linear relationship between $X$ and $Y$:

${\beta }_0$ and ${\beta }_1$ are known as coefficients or parameters
Use training data to produce estimates $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$, then predict future response values by:

Estimating Coefficients

Let

represent $n$ observation pairs, each which consists of a measurement of $X$ and a measurement of $Y$
Measure closeness using least squares (this is one of multiple ways to measure closeness)
Let ${\hat{y}}_i=\widehat{{\beta }_0}+\widehat{{\beta }_1}{x}_i$ be the prediction for $Y$ based on the $i^{th}$ value of $X$. Then $e_i=y_i-{\hat{y}}_i$ represents the $i^{th}$ residual - the difference between the $i^{th}$ observed response value and the $i^{th}$ response value that is predicted. We define the residual sum of squares (RSS) as

or

Least squares chooses $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ to minimize RSS. Minimizers are:

where $\overline{y}\equiv \frac{1}{n}\sum _{i=1}^n{y}_i$ and $\overline{x}\equiv \frac{1}{n}\sum _{i=1}^n{x}_i$ are the sample means.

Assessing Coefficient Accuracy

Standard error of $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$:

where ${\sigma }^2=Var(ϵ)$, assuming the errors ${ϵ}_i$ have a common variance ${\sigma }^2$ and are uncorrelated.
Estimate $\sigma$ using residual standard error:

95% confidence interval can be calculated by:

and

Hypothesis testing:

where $t$ distribution has $n-2$ degrees of freedom and $n>30$.

Assessing Model Accuracy

Residual Standard Error (RSE)
$R^2$ Statistic
Measures the proportion of variance:

where

$R^2$ measures the proportion of variability in $Y$ that can be explained using $X$. An $R^2$ statistic that is close to 1 indicates that a large proportion of the variability in the response is explained by the regression. A number near 0 indicates that the regression does not explain much of the variability in the response; this might occur because the linear model is wrong, or the error variance ${\sigma }^2$ is high, or both.
For a simple linear regression (*this does not hold for a Multiple Linear Regression), $R^2=r^2$ where $r=Cor(X,Y)$:

Sources: 1