ABSTRACT
The spiked covariance model has gained increasing popularity in high-dimensional data analysis. A fundamental problem is determination of the number of spiked eigenvalues, K. For estimation of K, most attention has focused on the use of top eigenvalues of sample covariance matrix, and there is little investigation into proper ways of using bulk eigenvalues to estimate K. We propose a principled approach to incorporating bulk eigenvalues in the estimation of K. Our method imposes a working model on the residual covariance matrix, which is assumed to be a diagonal matrix whose entries are drawn from a gamma distribution. Under this model, the bulk eigenvalues are asymptotically close to the quantiles of a fixed parametric distribution. This motivates us to propose a two-step method: the first step uses bulk eigenvalues to estimate parameters of this distribution, and the second step leverages these parameters to assist the estimation of K. The resulting estimator aggregates information in a large number of bulk eigenvalues. We show the consistency of
under a standard spiked covariance model. We also propose a confidence interval estimate for K. Our extensive simulation studies show that the proposed method is robust and outperforms the existing methods in a range of scenarios. We apply the proposed method to analysis of a lung cancer microarray dataset and the 1000 Genomes dataset.
Supplementary material
The supplementary material contains the description of two GetQT algorithms, the proof of main theorems and lemmas, and the results of BEMA on real datasets with varying α.
Software
The code for implementing BEMA is available at https://github.com/ZhengTracyKe/BEMA
Acknowledgments
The authors thank to Rounak Dey and Derek Shyr for their help on downloading and pruning the 1000 Genomes dataset, and Zhigang Bao and Xiucai Ding for helpful pointers on random matrix theory.
Notes
1 The factor is due to considering the zero-excluded ESD. If we consider the classical ESD, this factor should be
.
2 We remark that the comparison is for the standard spiked covariance model only. For this model, our method has the weakest conditions for consistent estimation of K. On the other hand, other methods apply to some other settings, which are not considered in the comparison.
3 In our model (see Assumption 1), the spiked eigenvalues of are
. Therefore,
is a lower bound of these spiked eigenvalues.