Linear Regression Analysis

The following sections are included:

• Preface

• Contents

• List of Figures

• List of Tables

The following sections are included:

• Regression Model

• Goals of Regression Analysis

• Statistical Computing in Regression Analysis

The following sections are included:

• Introduction

• Least Squares Estimation

• Statistical Properties of the Least Squares Estimation

• Maximum Likelihood Estimation

• Confidence Interval on Regression Mean and Regression Prediction

• Statistical Inference on Regression Parameters

• Residual Analysis and Model Diagnosis

• Example

The following sections are included:

• Vector Space and Projection
  • Vector Space
  • Linearly Independent Vectors
  • Dot Product and Projection

• Matrix Form of Multiple Linear Regression

• Quadratic Form of Random Variables

• Idempotent Matrices

• Multivariate Normal Distribution

• Quadratic Form of the Multivariate Normal Variables

• Least Squares Estimates of the Multiple Regression Parameters

• Matrix Form of the Simple Linear Regression

• Test for Full Model and Reduced Model

• Test for General Linear Hypothesis

• The Least Squares Estimates of Multiple Regression Parameters Under Linear Restrictions

• Confidence Intervals of Mean and Prediction in Multiple Regression

• Simultaneous Test for Regression Parameters

• Bonferroni Confidence Region for Regression Parameters

• Interaction and Confounding
  • Interaction
  • Confounding

• Regression with Dummy Variables

• Collinearity in Multiple Linear Regression
  • Collinearity
  • Variance Inflation

• Linear Model in Centered Form

• Numerical Computation of LSE via QR Decomposition
  • Orthogonalization
  • QR Decomposition and LSE

• Analysis of Regression Residual
  • Purpose of the Residual Analysis
  • Residual Plot
  • Studentized Residuals
  • PRESS Residual
  • Identify Outlier Using PRESS Residual
  • Test for Mean Shift Outlier

• Check for Normality of the Error Term in Multiple Regression

• Example

• Problems

The following sections are included:

• Model Diagnosis for Multiple Linear Regression
  • Simple Criteria for Model Comparison
  • Bias in Error Estimate from Under-specified Model
  • Cross Validation

• Detection of Outliers in Multiple Linear Regression

• Detection of Influential Observations in Multiple Linear Regression
  • Influential Observation
  • Notes on Outlier and Influential Observation
  • Residual Mean Square Error for Over-fitted Regression Model

• Test for Mean-shift Outliers

• Graphical Display of Regression Diagnosis
  • Partial Residual Plot
  • Component-plus-residual Plot
  • Augmented Partial Residual Plot

• Test for Inferential Observations

• Example

• Problems

The following sections are included:

• Effect of Underfitting and Overfitting

• All Possible Regressions
  • Some Naive Criteria
  • PRESS and GCV
  • Mallow's C_P
  • AIC, AIC_C, and BIC

• Stepwise Selection
  • Backward Elimination
  • Forward Addition
  • Stepwise Search

• Examples

• Other Related Issues
  • Variance Importance or Relevance
  • PCA and SIR

• Problems

The following sections are included:

• Test Heteroscedasticity
  • Heteroscedasticity
  • Likelihood Ratio Test, Wald, and Lagrange Multiplier Test
  • Tests for Heteroscedasticity
    • White's Test
    • Park, Glesjer, and Breusch-Pagan-Godfrey Tests
    • Goldfeld-Quandt test

• Detection of Regression Functional Form
  • Box-Cox Power Transformation
  • Additive Models
  • ACE and AVAS
  • Example

• Problems

The following sections are included:

• Non-Full-Rank Linear Regression Models
  • Generalized Inverse
  • Statistical Inference on Null-Full-Rank Regression Models

• Generalized Least Squares
  • Estimation of (β, σ²)
  • Statistical Inference
  • Misspecification of the Error Variance Structure
  • Typical Error Variance Structures
  • Example

• Ridge Regression and LASSO
  • Ridge Shrinkage Estimator
  • Connection with PCA
  • LASSO and Other Extensions
  • Example

• Parametric Nonlinear Regression
  • Least Squares Estimation in Nonlinear Regression
  • Example

• Problems

The following sections are included:

• Introduction: A Motivating Example

• Components of GLM
  • Exponential Family
  • Linear Predictor and Link Functions

• Maximum Likelihood Estimation of GLM
  • Likelihood Equations
  • Fisher's Information Matrix
  • Optimization of the Likelihood

• Statistical Inference and Other Issues in GLM
  • Wald, Likelihood Ratio, and Score Test
  • Other Model Fitting Issues

• Logistic Regression for Binary Data
  • Interpreting the Logistic Model
  • Estimation of the Logistic Model
  • Example

• Poisson Regression for Count Data
  • The Loglinear Model
  • Example

• Problems

The following sections are included:

• Bayesian Linear Models
  • Bayesian Inference in General
  • Conjugate Normal-Gamma Priors
  • Inference in Bayesian Linear Model
  • Bayesian Inference via MCMC
  • Prediction
  • Example

• Bayesian Model Averaging

• Problems

The following sections are included:

• Bibliography

• Index