Random Matrices: Theory and Applications

This paper considers the problem of detecting a few signals in high-dimensional complex-valued Gaussian data satisfying Johnstone's [On the distribution of the largest eigenvalue in principal components analysis, Ann. Statist.29 (2001) 295–327] spiked covariance model. We focus on the difficult case where signals are weak in the sense that the sizes of the corresponding covariance spikes are below the phase transition threshold studied in Baik et al. [Phase transition of the largest eigenvalue for non-null complex sample covariance matrices, Ann. Probab.33 (2005) 1643–1697]. In contrast to the majority of previous studies, we base the signal detection on the information contained in all the eigenvalues of the sample covariance matrix, as opposed to a few of the largest ones. This allows us to detect weak signals with non-trivial asymptotic probability when the dimensionality of the data and the number of observations go to infinity proportionally. We derive a simple analytical expression for the maximal possible asymptotic probability of correct detection holding the asymptotic probability of false detection fixed. To accomplish this derivation, we establish a novel representation for the hypergeometric function of two p × p matrix arguments, one of which has a deficient rank r < p, as a repeated contour integral of the hypergeometric function of two r × r matrix arguments.

We consider adding arbitrary covariance to the β-Jacobi random matrix model. We recall that for β = 1 the Jacobi random matrix model may be thought of as the eigenvalues, λ_i, of Y^tY(X^TX + Y^tY)^-1 where X and Y are matrices whose elements are i.i.d. standard normals. Equivalently we can take the generalized cosine singular values of (Y, X), c_i, and use . When β = 1 we add covariance by considering Y^tY(Y^tY + ΩX^tXΩ)^-1, for a positive definite diagonal matrix Ω. Equivalently, and preferably, we consider the generalized singular value decomposition (gsvd) of (Y, XΩ). We refer to Ω = I as the Jacobi case and the general Ω case as the MANOVA case. In this paper, we provide a matrix model for the general β-MANOVA ensemble. In particular, we provide an algorithm for the numerical sampling of eigenvalues or generalized cosine singular values. The β-MANOVA algorithm uses the β-Wishart algorithm of Forrester and Dubbs–Edelman–Koev–Venkataramana as a subroutine, perhaps making β-MANOVA the first "second-order" continuous-β random matrix algorithm. Our proofs make use of a conjecture of MacDonald (proven by Baker and Forrester), a theorem of Kaneko, and many identities from Forrester's Log-Gases and Random Matrices. We supply numerical evidence that our theorems are correct.

Formulae of Plancherel–Rotach type are established for the average characteristic polynomials of Hermitian products of rectangular Ginibre random matrices on the region of zeros. These polynomials form a general class of multiple orthogonal hypergeometric polynomials generalizing the classical Laguerre polynomials. The proofs are based on a multivariate version of the complex method of saddle points. After suitable rescaling the asymptotic zero distributions for the polynomials are studied and shown to coincide with the Fuss–Catalan distributions. Moreover, introducing appropriate coordinates, elementary and explicit characterizations are derived for the densities as well as for the distribution functions of the Fuss–Catalan distributions of general order.