The Fence Methods

The following sections are included:

• Dedication
• Preface
• Contents

On the morning of March 16, 1971, Hirotugu Akaike, as he was taking a seat on a commuter train, came out with the idea of a connection between the relative Kullback-Liebler discrepancy and the empirical log-likelihood function, a procedure that was later named Akaike's information criterion, or AIC [Akaike (1973, 1974); see Bozdogan (1994) for the historical note]. The idea has allowed major advances in model selection and related fields. See, for example, de Leeuw (1992)…

Before we move on to introduce some of the most important advances in the fence methods, we would like to use a number of examples, including simulations and real data analyses, to demonstrate the fence procedures as well as the scope of potential applications.

A question that remains to be answered would be how to choose the tuning constant for the fence, that is, c in (1.20). In this chapter, we offer an approach to answer the question. The idea is called adaptive fence, originally due to Jiang et al. (2008) in the context of mixed model selection. Since then, several variations, and simplification, of the idea have been developed, which we shall discuss in this chapter, and in subsequent chapters.

The SAF discussed in Section 3.2 reduces the computational burden of the adaptive fence by avoiding the evaluation of . On the other hand, even with this simplification, one may still encounter computational difficulties when applying the fence to high dimensional and complex problems. The main difficulty rests in the evaluation of a large number of 's, if, for example, the number of candidate variables is fairly large. More specifically, if all-subset selection is considered, and there are k candidate variables, a total of 2^k evaluations of different Q(M)'s are required. For example, for k = 10, the total number of evaluations is 1,024; for k = 30, the total number of evaluations is 1,073,741,824 (over a billion); and the number goes up quickly. Earlier, effort was taken to avoid the all-subset selection by considering a stepwised fence procedure. See Section 1.5. Note that the amount of computation for the adaptive fence, or SAF, is usually much higher than that for the non-adaptive fence due to the need for bootstrapping. In this chapter, we consider a different approach to making the fence method computationally more attractive.

Another variation of the fence that is intended for high-dimensional selection problems is invisible fence [IF; Jiang et al. (2011b)]. A critical assumption in Jiang et al. (2008) is that there exists a correct model among the candidate models. Although the assumption is necessary in establishing consistency of the fence, it limits the scope of applications because, in practice, a correct model simply may not exist, or exist but not among the candidate models. In this regard, George Box once famously said that “all models are wrong … but some are useful”. Along the same line, we can extend the fence to make it applicable to (much) more general situations. The extension has been done in Section 1.3, where the notions of measure of lack-of-fit and true model were extended. To introduce the IF, we first take another look at the fence methods.

Small area estimation (SAE) has received increasing attention in recent literature. Here the term small area typically refers to a population for which reliable statistics of interest cannot be produced due to certain limitations of the available data. Examples of small areas include a geographical region (e.g., a state, county, municipality, etc.), a demographic group (e.g., a specific age × sex × race group), a demographic group within a geographic region, etc. In absence of adequate direct samples from the small areas, methods have been developed in order to “borrow strength”. See Rao (2003) for a comprehensive account of various methods used in SAE. Statistical models, especially mixed effects models, have played important roles in SAE. Also see Jiang and Lahiri (2006) for an overview of mixed effects models in SAE…

Simultaneous variable selection and estimation by penalized likelihood methods have received considerable interest in the recent literature on statistical learning [see Fan and Lv (2010) for an overview]. In a seminal paper, Tibshirani (1996) proposed the least absolute shrinkage and selection operator (Lasso) to automatically select variables via continuous shrinkage. The method has gained popularity thanks to the developments of several computation algorithms [see Fu (1998), Osborne et al. (2000), and Efron et al. (2004)]. In particular, the LARS algorithm proposed by Efron et al. (2004) finds the entire regularization path at the same computational cost of one single ordinary LS fit. Fan and Li (2001) proposed the smoothly clipped absolute deviation method (SCAD) which has the oracle property under certain conditions. For the most part, the oracle property means that the parameter estimators under the selected model are as efficient as those when the true underlying model is known. Local quadratic method [Fan and Li (2001)], minorize-maximize [MM; Hunter and Li (2005)], local linear approximation method (LLA; Zou and Li (2008)] have been proposed to compute the SCAD estimates. In particular, the LLA reduces the computational cost for the SCAD dramatically by applying the LARS algorithm. Furthermore, Zou (2006) proposed the adaptive Lasso and shows it also has the oracle property. In this chapter, we discussed a few problems for which solutions were obtained, or may be sought, using the fence idea. Although these methods may seem somewhat different from the fence, it may be argued, as we shall see, that they are, indeed, special cases of the fence.

The following sections are included:

• Introduction
• A double-dipping problem
• The EMAF algorithm
• The E-MS algorithm
• Two simulated examples
  • EMAF in backcross experiment
  • Linear regression: Comparison of strategies
• Missing data mechanisms
• Real data example
• Exercises

The following sections are included:

• Introduction
• Consistency of fence, F-B fence and AF
• Asymptotic properties of shrinkage selection
• Signal consistency of IF
• Convergence and consistency of E-MS with fence
• Concluding remark
• Exercises

The following sections are included:

• Bibliography
• Index