Linear Models and Regression with R

The following sections are included:

• Dedication
• Preface
• Contents
• Glossary of Abbreviations
• Glossary of Matrix Notations

It is in human nature to try and understand the physical and natural phenomena that occur around us. For example, all of us know that death is a certainty and that it is only the time of death that is not known beforehand. People have tried to understand the time of death as much as possible, by quantifying the chances of this event at various ages. In the mid-nineteenth century, William Guy published a series of articles (see e.g. Guy, 1859) on comparative studies of the duration of life among persons of different professions. Figure 1.1 depicts a part of the age-at-death data published by him, where the deceased person’s profession has been classified into eight categories: ‘historians’, ‘poets’, ‘painters’, ‘musicians’, ‘mathematicians and astronomers’, ‘chemists and natural philosophers’, ‘naturalists’ and ‘engineers, architects and surveyors’. Despite the variation in ages at death within each category, it would be interesting to know whether people in these various professions generally live for the same duration. For instance, do mathematicians live significantly longer than poets or historians? This is not merely a matter of casual curiosity. Today actuaries routinely classify potential customers for life insurance by various characteristics that are perceived to have a bearing on the hazard of death. Since it is very difficult to identify a large group of people exactly matching the profile of a specific customer, we look for a broader picture. We seek to describe the probable lifetimes of individuals through a mathematical model that is applicable for the entire set of data, covering all the categories and other relevant characteristics that might affect a lifetime…

The regression method may be considered as a way of approximating a random variable of interest, as a function of several other related random variables. This task can only be effective if there is some sort of dependence between the variable of interest (called the response) and the approximating (or explanatory) variables. We will assume here that there is such a stochastic dependence, and that the realized values of the explanatory variables are observed, while the corresponding value of the response is unknown and is to be assessed. In such a situation, we try to identify the best possible approximation, and study its properties under certain assumptions…

Consider the homoscedastic linear model

In Chapter 3 we considered point estimation of estimable linear parametric functions (LPFs) in the linear model (y, X β, σ² I), including scalar LPFs (pʹ β) and vector LPFs (Aβ). In this chapter we discuss confidence regions for such LPFs, tests of hypotheses, and model-based prediction. In Section 4.1 we derive the distribution theory for the usual estimators of Aβ and σ² under the assumption that the errors are independent and normal. (As we shall see in Section 11.5, this assumption can be relaxed somewhat in large samples.) This basic result allows us to test linear hypotheses (Section 4.2) on the parameters and construct confidence regions (Section 4.3) for them. We address the problem of optimal prediction in Section 4.4. In Section 4.5 we discuss the problem of collinearity and examine its effect on confidence regions, prediction, and tests of hypotheses.

As mentioned in Section 2.3, the regression of y on x₁, …, x_k, namely E(y❘x₁, …, x_k) represents the ‘best’ approximation (in the sense of smallest mean squared error) of a random variable y, as a function of other random variables x₁, …, x_k. The particular linear model (y, X β, σ² I) presents us with a simple setup for studying such a regression function and the associated approximation error. Specifically, the model for a single response variable y becomes

One of the invariables in our life, it has been observed, is variability itself. All human beings do not have the same height. Yield of wheat per acre of land is not the same everywhere. Even the microchips, for which we desire very low tolerances, vary in their performance characteristics, due to factors beyond the manufacturer’s control…

In this chapter we discuss the ‘general’ case of the linear model, (y, X β, σ²V), where V is not necessarily the identity matrix. In contrast to the homoscedastic model, this general choice for V allows the observations to be correlated as well as to have different variances. Although we allowed X to be rank deficient in the preceding chapters, here for the first time, we also allow the dispersion matrix V to be rank-deficient i.e. singular. We refer to the linear model with a singular dispersion matrix as the singular linear model…

The results of Chapters 3, 4 and 7 are based on the crucial assumption that the error dispersion matrix σ²V in the linear model (y, X β, σ²V) is known, up to the unspecified scale factor σ². The expression for the BLUE of an estimable parametric function, for instance, depends on this dispersion matrix and cannot be computed when the matrix V is unknown. As we shall see in this chapter however, there are many practical situations where the dispersion matrix is unknown…

Consider the linear model (y, X β, σ²V) where the parameters β and σ² are unknown. The statistical quantities of interest include the best linear unbiased estimators (BLUEs) of the estimable parametric functions, variance-covariance matrices of such estimators, the error sum of squares and the likelihood ratio tests for testable linear hypotheses. In this chapter we are concerned with the changes in these quantities when some observations are included or excluded, as well as when some explanatory variables are included or excluded…

Suppose we wish to determine the efficacy of a new drug by observing n patients over a period of time. Consider the situation where the response of the ith patient (i = 1, …, n) is measured via the value of the set of q variables (y_i1, …, y_iq) which are quantitative measures of relief, observed at the different time points 1, …, q after the treatment begins. As in typical regression, each response may depend on several explanatory variables such as the drug used (new or conventional), the age and gender of the patient etc. A linear model for such multivariate response can take the form

The classical theory of statistical inference (briefly reviewed in Section B.1 of Appendix B), has a strong parallel to inference about the parameter β in the linear model (y, X β σ²V), if we restrict our attention to linear statistics of the form Ly. Early work on such connections can be found in Barnard (1963) and Baksalary and Kala (1981). Drygas (1983) gives the theory a more elaborate structure, while subsequent work (see e.g. Kala et al., 2017) provides modifications and many extensions in various directions. In Section 11.1 we present a systematic exposition of the theory and show how it is connected to the definitions as well as the propositions contained in Chapters 3 and 7. This development culminates in a decomposition of the response vector in the general linear model into uncorrelated parts that are either BLUE or LZF and are either degenerate or having a common variance. Section 11.2 carries the theory of linear inference further by discussing linear versions of admissible, Bayes and minimax estimators. In order to complete the story, some other classes of linear estimators are discussed in Section 11.3…

The following sections are included:

• Appendix A: Matrix and Vector Preliminaries
• Appendix B: Review of Statistical Theory
• Appendix C: Solutions to Selected Exercises

The following sections are included:

• Bibliography and Author Index
• Index