A new multivariate test formulation: theory, implementation, and applications to genomescale sequencing and expression
 Lei Xu^{1, 2}Email author
Received: 12 November 2015
Accepted: 24 December 2015
Published: 13 January 2016
Abstract
A new formulation is proposed for multivariate test, consisting of not only a hierarchy of numerous tests organised in a lattice taxonomy of properties that come from different combinations of multivariates and represent different factors associated with the rejection of null hypothesis, but also by a theory of propertyoriented rejection. Located on the bottom level of this taxonomy is a conventional formulation of multivariate test, featured by a property with the weakest collegiality and a rejection with the largest p value. From one level up to the next, the dimension of rejection increases, the collegiality of properties strengthen, and the p values reduce, until the top level that is featured by a property with the strongest collegiality and a rejection with the smallest p value. Instead of traversing all the combinations in the taxonomy, an easy implementation is developed to identify distinctive properties by the best first path (BFP) in a lattice taxonomy of an appropriate number of intrinsic factors that are obtained after decoupling secondorder dependence cross multivariate statistics and discarding those nondistinctive components. Even away off this BFP, if needed, a particular combination of intrinsic factors may be conveniently tested in such a taxonomy too. Moreover, further improvement is made by considering some dependence of higher than second order, with the top level p value refined into one upper bound that is obtained by directional test. Furthermore, detailed implementations are also provided for applications to genomescale sequencing and expression, with particular emphasis on multivariate phenotypetargeted test for expression profile analyses.
Keywords
Multivariate test Lattice taxonomy Intrinsic factors Propertyoriented rejection Best first path Dependence decoupling Directional test Case–control study Twovariate PT testBackground
Illustrated in Fig. 1 are two typical examples of multivariate test, coming from the case–control study. Given two populations of vectorvariate samples \(X_{\omega }= \{ \mathbf{x}_{t, \omega }, t=1, \ldots , N_{\omega }, \omega =0, 1 \}\), where one with \(\omega =1\) is called the case population while the one with \(\omega =0\) is called the control population. One important task is examining whether there is a significant difference between two populations of samples. Shown in Fig. 1a is an example of twosample test in a twodimensional data space. Test is made on Eq. (1) with \(\theta =\mathbf{c}_1\mathbf{c}_0\) that has a multivariate Gaussian distribution when \(\mathbf{c}_1, \mathbf{c}_0\) are estimated by sample means. Shown in Fig. 1b is an example of testing logistic regression. Test is made on Eq.(1) on the coefficient vector \(\theta =\mathbf {\beta }\) that also has a multivariate Gaussian distribution when \(\mathbf {\beta }\) is a maximum likelihood estimate.
In these methods, asymptotic approximations are used for deriving the significance levels of the test statistics under the null. However, when sample size is small and/or the data has a high missing rate, the null distributions of the test statistics may differ substantially from their asymptotic approximations. Therefore, these studies remain to be theoretical. Instead, real efforts either use the Hotelling’s \(T^2\) test directly (Fan and Knapp 2003) or impose some structure on the covariance matrix (Swanson et al. 2013), and even approximately simplify the covariance matrix into diagonal one but its ability of encoding LD information is lost (Kiezun et al. 2012).
The second thread is featured by the extensions of Wald test and Score test for jointly multiple hypotheses on single/multiple parameters, and efforts along this thread are widely encountered in studies of computational genomics. For a two sample test shown in Fig. 1a, it actually leads to the above first thread. Generally, the task is encountered in testing Eq. (1) on the coefficients \(\theta =\mathbf {\beta }\) of multivariate linear regression, multivariate logistic regression, and multivariate linear mixed model (Gudmundsson et al. 2012; Demidenko 2013; Zhou and Stephens 2014; Adhikari et al. 2015), as well as Cox regression analysis (Li and Gui 2004). Still, a use of the Fisher information matrix shares with the same problems caused by the covariance matrix. The problems will remain, though using multivariate Ftest and likelihood ratio test on Eq. (1) may help to improve the Wald test and score test.
The third thread originated from Fisher’s combined probability test for combining p values (Fishe 1932). However, each p value is merely a positive number that indicates the false alarm probability, already losing useful information such as an overall estimate of effect size, the direction of effects, and the dependence across effects. Progresses have been made by transforming p values into Z statistics or others on which some missing issues may be considered (Zaykin 2011), without or with help of information computed directly from datasets. Applied to rare variants, efforts made along this thread are typically referred under the term Metaanalysis (Evangelou and Ioannidis 2013).
The newest thread is featured by efforts in recent years for extending the existing GWAS from single variant to multiple rare variants. The basic idea is to let multiple variants of a unit (e.g. gene, exome, or one other biological unit) to be collapsed or summed up into a single one (Li and Leal 2008). Further developments are featured by various weighted sums via fixed weights or thresholds (Morgenthaler and Thilly 2007; Chapman and Whittaker 2008; WuM et al. 2011; Han and Pan 2010; Lee et al. 2012; Morris 2010; Price and Kryukov 2010). Turning multiple variants into a single one, the dimension of covariance matrix is thus reduced to lessen the problem of the above first thread. Most of these efforts are associated with a generalised linear regression model to test the null hypothesis that either the regression coefficients are zero or their variances are zero (Chapman and Whittaker 2008; WuM et al. 2011; Han and Pan 2010; Lee et al. 2012; Morris 2010; Price and Kryukov 2010). These studies are recently summarised under the names of burden tests and nonburden tests. Burden tests assume that all the variants in the target region have effects on the phenotype in the same direction and of similar magnitude, while nonburden tests cover various extensions beyond the assumption, for which details are further referred to one recent review (Lee et al. 2014).
 1.
A new multivariate test formulation that is featured not only by a hierarchy of numerous tests organised in a lattice taxonomy of properties that represent different causes and different dimensions of the null hypothesis rejection, but also by a theory of propertyoriented rejection.
 2.
An easy implementation that identifies distinctive properties by the best first path (BFP) in a lattice taxonomy that comes from an appropriate number of intrinsic factors by decoupling secondorder dependence cross multivariate statistics and discarding those nondistinctive components. Also, a particular combination that does not locate on this BFP may also be conveniently tested in such an taxonomy, if needed.
 3.
A further improvement is made by considering some dependence of higher than second order, with the p value of the top level refined into one upper bound obtained by a directional rejection based on a vectorial property possessed by the alarm in evaluation.
Finally, we discuss several potential applications to expression profilebased biomarker identification and exome sequencingbased joint SNV detection.
Methods
Existing methods from a vectorial statistics view: two limitations
Univariate tests are typically implemented in two complementary manners. One is illustrated in Fig. 2a. Given a significant level \(\alpha\), we get a boundary point and the red coloured rejection domain. For a set of statistics (e.g. those indicated by ‘\(*****\)’), every statistics \(\tilde{s}\) is classified into either the rejection domain or the acceptance domain. Those falling into the rejection domain can all identify a significant rejection of \(H_0\), featured by a worstcase false alarm probability \(\alpha\). Here, each statistics \(\tilde{s}\) has not been provided with its accurate false alarm probability though such a probability could be much smaller than \(\alpha\) especially when the corresponding statistics locates far away from the boundary point.
Also, we may understand the Hotelling test, Wald test, and Score test from a perspective of vectorial statistics in a multidimensional Sspace (Xu 2015).

\(H_0\) by Eq. (1) means that all the dimensions of \(\mathbf{s}\) are zero and we reject \(H_0\) as long as at least one of these dimensions is rejected to be zero. In other words, differentiation is considered in a lumped sense without considering the roles of different dimensions and their combinations, as illustrated in Fig. 3.

Rejection is made according to how far \(\mathbf{s}\) is away from the origin (possibly weighted by the orientation of \(\mathbf{s}\), e.g. see Eq. (3)), but without taking the direction of \(\mathbf{s}\) in consideration. In many applications, direction does take its role. In GWAS study, multiple SNPs’ joint effect is reflected in the direction of vectorial statistics, as addressed in Ref. (Bansal et al. 2010) and particularly in its Figure 2.
In the next three subsections, the first limitation will be tackled by considering various situations featured by differentiations associated with different dimensions and their combinations. Then, the second limitation is further tackled in the subsequent subsection, featured by directional tests.
Lattice taxonomy of tests with different dimensions of rejection
As illustrated in Fig. 3a, the rejection domain considered in the existing studies is the complement of the acceptance domain to the entire twodimensional Sspace, lumping up three rejection domains illustrated in Fig. 3b–d. Covering the largest area, it represents the rejection of \(H_0\) via either \(\lnot (s^{(1)}=0)\) or \(\lnot (s^{(2)}=0)\) in the weakest collegiality. On the other hand, the collegiality that \(\mathbf{s}\) falls in the rejection domain shown in Fig. 3d is strongest, requiring both \(\lnot (s^{(1)}=0)\) and \(\lnot (s^{(2)}=0)\), which is suitable for the cases that we make a rejection in one most conservative way.

Four choices of \([\lnot (s^{(1)}=0) \& \lnot (s^{(2)}=0)] \ or \ \lnot (s^{(3)}=0)\), \(\lnot (s^{(1)}=0) \ or \ [ \lnot (s^{(2)}=0) \& \lnot (s^{(3)}=0)]\), \([\lnot (s^{(1)}=0) \ or \ ( \lnot (s^{(2)}=0)] \& \lnot (s^{(3)}=0)\), and \(\lnot (s^{(1)}=0) \& [\lnot (s^{(2)}=0) \ or \ \lnot (s^{(3)}=0)]\);

Three choices of \(\lnot (s^{(1)}=0) \ or \ \lnot (s^{2)}=0)\), \(\lnot (s^{(2)}=0) \ or \ \lnot (s^{(3)}=0)\), and \(\lnot (s^{(1)}=0) \ or \ \lnot (s^{3)}=0)\);

Three choices of \(\lnot (s^{(1)}=0) \& \lnot (s^{2)}=0)\), \(\lnot (s^{(2)}=0) \& \lnot (s^{(3)}=0)\), and \(\lnot (s^{(1)}=0) \& \lnot (s^{3)}=0)\);

Three choices of \(\lnot (s^{(1)}=0)\), \(\lnot (s^{(2)}=0)\), and \(\lnot (s^{(3)}=0)\).
Generally, in the ndimensional space of vectorial statistics, testing \(H_0\) by Eq. (1) involves testing various types of rejections featured by subsets of n different dimensions and their & and or connected combinations. Although an exhaustive search of all the possible types of rejections will be very tedious, we may still make a rather systematical investigation that organises major types of rejections in a partial order structure, namely two cascaded taxonomies as illustrated in Fig. 4a.
In such a way, a multivariate test is not just a single test as usually considered in the existing studies. Examining whether \(H_0\) breaks in a lumping way is just one extreme (i.e. the bottom case) that puts the most loose requirement on making a rejection of \(H_0\), featured by a rejection with the biggest p value and weakest collegiality. Actually, multivariate testing consists of tests in different levels of collegiality and different types, examining a total number of \(2\sum _{i=2}^n (^n_i)\) different combinations of these dimensions that may cause a significant rejection of \(H_0\). The collegiality enhances from the bottom up towards the middle level that considers each of n different dimensions individually. From the middle level up, the collegiality further enhances level by level, until the top that represents another extreme featured by the smallest p value for a rejection in the strongest collegiality.
The gap \(\overline{p}_j  \underline{p}_j\) provides an information on a possible variety of significant differentiation on this level. The collegiality of rejection increases from the bottom up as indicated by the blue arrow on the 1st layer but reduces as indicated by the red arrow on the 2nd layer. Thus, \(\underline{p}_j\) decreases, \(\overline{p}_j\) increases from the bottom up, and thus the gap \(\overline{p}_j  \underline{p}_j\) increases as j increases.
Illustrated in Fig. 4c is the outcome of all the tests organised in the taxonomy, from which we get a roadmap about how each dimension or a combination of dimensions contributes to a significant rejection of the null hypothesis. Comparing \(\underline{p}_{j1}^{\omega }\underline{p}_j^{\omega }\) and \(\overline{p}_{j1}^{\omega }\overline{p}_j^{\omega }\), as well as comparing the p values between two consecutive levels, we may understand the incremental role taken by each dimension.
 Problem 1
How to effectively compute \(\underline{p}_j^{\omega }, \overline{p}_j^{\omega }\) in Eq. (12) ?
 Problem 2
It is usually infeasible to enumerate all the \(\sum _{i=2}^n (^n_i)\) different combinations. How to effectively make such enumeration ?
 Problem 3
There is dependence and redundancy among dimensions of \(\mathbf{s}\), which affects seriously the above two problems. How to remove the dependence and redundancy and to select appropriate number of dimensions ?
Testing implementation: independence case and latent independence
However, the above solution is too rough. Not only it is not easy to determine \(d_i\), but also it does not consider the joint effect of different dimensions. Returning to the thinking line of Eq. (17), we seek to correct each \(\underline{p}_j\) by a normalisation term, see Eqn. (93) in Ref. (Xu 2015a).
The above mentioned unknown \(\delta ^{(i)}\) for some dimension of “donotcare” may be approximately regarded as unchanged over different permutations. Hence, its effect to both the denominator and the numerator will be cancelled out by Eq. (19), and we get closer to Eq. (17) after \(p_j\) is replaced by \(pp_j\).
One essential problem is that the assumption by Eq. (13) is difficult to be satisfied and thus the resulted \(\underline{p}_j^{\omega }\) by Eq. (14) is usually too optimistic. Instead, we may further consider the latent independence as illustrated in Fig. 5a. That is, we observe a latent coordinate wherein components are mutual independent subject to an additive noise e that is independent of \(\mathbf{s}_u\) and typically Gaussian with a spherical covariance matrix. In the new coordinate, we can get not only the effect of noise e in consideration but also Eq. (13) satisfied at least conceptually. Implementation may just follow those addressed between Eqs. (13) and (20), simply with each appearance of \(\mathbf{s}\) replaced by \(\mathbf{s}_u\).
When each component of \(\mathbf{s}_u\) comes from a nonGaussian univariate, the latent model is called nonGaussian factor analysis (NFA) (Xu 2003, 2009; Tu and Xu 2014) and the mapping from \(\mathbf{s}\) to \(\mathbf{s}_u\) is featured by a distribution with a nonlinear regression as illustrated in Fig. 5c. When each component of \(\mathbf{s}_u\) is a Gaussian univariate, the latent model becomes the classical factor analysis (FA) and the mapping from \(\mathbf{s}\) to \(\mathbf{s}_u\) is a distribution with a linear regression as illustrated in Fig. 5b. Particularly, a FA model may be called either FAb with an additional constraint \(AA^T=I\) or FAa for a conventional setting. For the maximum likelihood learning, FAa and FAb are equivalent. However, FAb becomes more favourable for determining m. Readers are referred to Sect.2.2 in Xu (2011) and Tu and Xu (2011) for further studies on FAb versus FAa.
Generally, we may estimate \(\Sigma _\mathbf{s}\) from a set \(\{ {\tilde{\mathbf{s}}}^{\pi }\}\) (including \({\tilde{\mathbf{s}}}\)), with each \({\tilde{\mathbf{s}}}^{\pi }\) obtained by a set of samples that comes from a permutation \(\pi\) of the original samples sets. Specifically, we let \(\Sigma _\mathbf{s}=\Sigma _{\pi }\) with \(\Sigma _{\pi }\) given by Eqn. (69) in Ref. Xu (2015a).
Also, we may get \(\Sigma _\mathbf{s}\) by the Fisher information matrix (i.e. Eqn. (6) in Ref. Xu 2015a) for a regression coefficient test. For a two sample test, we may use \(\Sigma\) in Eq. (2) as \(\Sigma _\mathbf{s}\) under the assumption that not only samples are i.i.d. but also case population and control population are uncorrelated.
As illustrated in Fig. 6a, b, the Cartesian coordinate is rotated into the one spanned by the eigenvectors \(\mathbf{u}_j, j=1, \ldots , m\). After the rotation, the acceptance domain is an ellipse and further becomes a sphere after the normalisation by \(\Lambda ^{0.5}\). Approximately, we may implement tests in the new coordinate following those addressed from Eqs.(13) to (19), simply with each appearance of \(\mathbf{s}\) replaced by \(\mathbf{s}_u\).
Higher order independence and propertyoriented test
With help of the secondorder independence by Eq. (21), we get a rejection domain that covers everywhere outside of the ellipse illustrated in Fig. 6b. Its difference from the blue coloured box is illustrated by the grey area that reflects the influence of higher order dependence. Keeping this influence may not only simplify the computation of \(\overline{p}_{I,j}^{\omega }\) by Eq. (14) but also enhance reliability because the problem of removing higher order dependence becomes more difficult especially based on merely a small size of samples.
In addition to turning a multivariate test into multilevels of tests in a taxonomy illustrated in Fig. 4, the above quad by Eq. (30) alone represents a further development of multivariate test already. The first two \(\overline{p}_{I,j}^{\omega }, \overline{p}_{e,j}^{\omega }\) represent a conventional practice of making a multivariate test on a vectorial statistics \(\mathbf{s}\) of the mutual independence or loosely the secondorder independence cross the components of \(\mathbf{s}\). The other two \(\underline{p}_{h,j}^{\omega }, \underline{p}_{I,j}^{\omega }\) represent new developments. It follows from Eq. (14) that \(\underline{p}_{I,j}^{\omega }\) is featured by a probabilistic product of multiple univariate onetailed or twotailed tests for a vectorial statistics \(\mathbf{s}\) of mutual independence, while \(\underline{p}_{h,j}^{\omega }\) takes certain high order dependence in consideration.
All the above and previously addressed tests can be regarded as examples of propertyoriented tests. Recalling the univariate test introduced in Fig. 2b, one key issue is estimating the probability that the false alarms disturb the judgement on a given scalar statistics \({\tilde{s}}\) according to a property owned by the statistics, which can be further generalised into the property sharing condition given in Table 1.
For a scalar statistics \({\tilde{s}}\), there is only one property \(s \ge {\tilde{s}}\) to consider, as illustrated by the red coloured rejection range in Fig. 2b. However, for vectorial statistics \(\mathbf{s}\) there are various choices to be considered. First, we may consider the properties of \(\mathbf{s}\) either directly in its own Cartesian coordinate or one of subspaces spanned by different combinations of its eigenvectors \(\mathbf{u}_1, \ldots , \mathbf{u}_m\) given in Eq. (21), with some dependence and redundancy removed, as well as some disturbing noises discarded. Second, we may consider various properties featured by different types of combinations of the components \(\mathbf{s}_u =[ s_u^{(1)}, \ldots , s_u^{(m)}]^T\).
Summarised below are four types introduced previously:
Basic property a property owned by each scalar component \(s^{(i)}=0, i\in \{1,2,\ldots , n\}\) individually, i.e. the bottom level illustrated in Fig. 4.
Logical combination a property obtained by combining several basic properties via logical connections & and or, i.e. other levels illustrated in Fig. 4b.
Linear boundary equation a property with its corresponding rejection domain given by Eq. (26) and featured with a linear equation \(B_{ \tilde{\mathbf{s}}_u }({ \mathbf{s}_u })=0\), e.g. a halfspace illustrated in Fig. 6b and defined by one linear equation in Eq. (28). Actually, it includes each basic property above as its degenerated case.
A theory of propertyoriented rejectionbased test
Key point  Description 

Property sharing condition  A necessary condition for false alarms to disturb the judgment on a given statistics is sharing with the statistics’ property that we consider 
Alarm scaleup nature  If an alarm vector \(S^{0.5}(\mathbf{s}{\tilde{\mathbf{s}}})\) falls within a rejection domain, \(\gamma S^{0.5}(\mathbf{s}{\tilde{\mathbf{s}}})\) will also fall within the rejection domain for any \(\gamma >1\) 
Least complexity principle  A rejection domain is modelled by a smallest number of parameters that can be well determined from given samples 
There are many other properties to be considered too. The last two above can be further extended by considering \(B_{ \tilde{\mathbf{s}}_u }({ \mathbf{s}_u })=0\) in a higher order equation. Beyond Eq. (25), it is also possible to use an even general mathematical model \(\Gamma ({\tilde{\mathbf{s}}})\) to express rejection domain, e.g. the twobranching curved boundary illustrated in Fig. 6c.
Naturally, we come to a question, is there any necessary condition that such a rejection domain should satisfy ?
If an alarm is able to disturb the judgement on \({\tilde{s}}\), one usually expects that enlarging the magnitude of this alarm should make this disturbance more stronger, which leads to the scaleup nature given in Table 1, based on which we may exclude many bad choices of rejection domain.
However, it is still not enough yet to fix a reasonable rejection domain. On one hand, the p value reduces as the rejection domain becomes smaller, which seemly leads us to choose a rejection domain as small as we want. On the other hand, in order to specify a rejection domain, what we can rely on are merely the known \(\tilde{\mathbf{s}}_u\) and \(\lambda _1, \ldots , \lambda _m\), which have already been used in Eqs. (25) and (28) for defining a linear or quadratic equation \(B_{ \tilde{\mathbf{s}}_u }({ \mathbf{s}_u })=0\). For a complicated rejection domain, e.g. the green coloured one shown in Fig. 6c, there are more unknowns to be specified. We need to either let an enough large part of them becoming known or get some priories that enable to fix those unknowns. In other words, we encounter the problems of unreliability and overfitting, especially when there is a finite size of samples for us to compute Eq. (21), which thus leads to the least complexity principle given in Table 1.
The situation is similar to the problem of selecting an appropriate number \(k^*\) of dimensions, as addressed between Eqs. (16) and (19). Although it remains an open challenge to choose a rejection domain modelled by a smallest number of parameters, we are at least able to determine \(k^*\) by Eq. (20), which provides another perspective to understand the rationale of examining \(\underline{p}_{h,j}^{\omega }, \underline{p}_{I,j}^{\omega }\) for all \(j\le k^*\) in the taxonomy illustrated in Fig. 4c.
Moreover, the statistics obtained from random samples is probabilistic, and thus the property we consider is probabilistic too. To increase the reliability, we may use a bootstrap method.
Finally, summarised in Algorithm 1 are the main steps of implementing tests in a lattice taxonomy.
Directional test, matrixvariate test, and phenotypetargeted test
Propertyoriented multivariate tests can be divided into two categories, namely directional tests versus nondirectional tests. As previously addressed in Fig. 1 and the last paragraph of the introduction section, the existing multivariate tests, and the ones with \(\overline{p}_{I,j}^{\omega }, \overline{p}_{\lnot e,j}^{\omega }\) as well, are mostly nondirectional tests, which can be regarded as extensions of univariate twotailed tests, featured by merely considering how far the vectorial statistics is away from the origin (possibly weighted by its orientation) but without taking its direction in consideration.
In contrast, a directional test is featured by that its rejection domain relates to certain direction. Precisely, this rejection domain at least contains a nonempty set D of unit vectors such that \(\gamma _0 \mathbf {d}_0\) locates outside of the rejection domain for some \(\mathbf {d}_0 \in D\) and a large enough \(\gamma _0>0\), i.e. at least the rejection domain does not contain some directions. Directional tests can be regarded as extensions of univariate onetailed tests to multivariate tests. In addition to the previously addressed halfspace associated with \(\underline{p}_{h,j}^{\omega }\) (i.e. the one illustrated by the black line), examples shown in Fig. 6c are all directional tests. The one outside the ellipse and lilac coloured can be regarded as a directional counterpart of the bottom one in Fig. 4b, associated with the largest p value. The other side of the black line contains the green coloured rejection domain with its p value smaller than \(\underline{p}_{h,j}^{\omega }\). They may all be regarded as examples of the boundarybased test (BBT) (see Table 6 in Ref. Xu 2015a), with their rejection domains featured by quadratic, linear, and twobranching curved boundary, respectively. Moreover, \(\underline{p}_{I,j}^{\omega }\) is featured by a probabilistic product of multiple univariate onetailed tests, and its corresponding rejection domain is actually an orthant of the Sspace along the direction of the vector \(sign({\tilde{\mathbf{s}}}_u)\).
Alternatively, the directional test associated with the halfspace type rejection domain given in Eq. (28) can be implemented from another aspect. We may obtain \(\underline{p}_{h,j}^{\omega }\) by a univariate onetailed test on the Bscore as given in Table 2. Also, we may consider other two types of projection scores in that table. One is the FDA score with the projection direction being the normal direction of the best linear separating hyperplane as shown in Fig. 1. Another measure is the misclassification rate of the case–control samples by the linear boundary. Several machine learning methods are available for learning such a linear boundary, with two examples given in Table 2.
Three typical projection scores
Type  Description 

Bscore  We get a projection score \(B_{ \tilde{\mathbf{s}}_u }(\mathbf{s})\) by Eq. (28), namely, the projection of statistics \(\mathbf{s}_u \tilde{\mathbf{s}}_u\) onto a particular direction \(S_j^{0.5} {\mathop{ {\beta}}\limits^{\rightarrow}}\). Actually, we are lead to a univariate onetailed test on this projection score, which may be simply implemented by either onetailed ztest or onetailed ttest. Also, we may estimate the univariate distribution of the score, and then compute \(\underline{p}_{h,j}^{\omega }\) based on the estimated distribution 
FDA score  We get \({\mathop{ {\beta}}\limits^{\rightarrow}}\) by making the Fisher discriminative analysis (FDA) on the control samples and the case samples, and then obtain the projection score \({\mathop{ {\beta}}\limits^{\rightarrow}}^T\mathbf{s}\) with \(\mathbf{s}\) given by Eq. (4), where the arrow of \({\mathop{ {\beta}}\limits^{\rightarrow}}\) points from the control to the case, as illustrated in Fig. 1b, while the classical FDA does not care about which direction of two choices is taken as the arrow. A directional test can be made by either onetailed ztest or onetailed ttest, using the statistics \(\quad \quad \quad \quad \quad \quad \quad \quad t_{\mathop{ {\beta}}\limits^{\rightarrow}} = \frac{{\mathop{ {\beta}}\limits^{\rightarrow}}^T\mathbf{s} }{{ \sigma }}, \ \sigma ^2=\alpha _0 \sigma ^2_0+ \alpha _1 \sigma ^2_1,\quad \quad \quad \quad \quad \quad \quad \quad \)(32) 
where \(\sigma ^2_0, \sigma ^2_1\) are the sample variances of the projections of control–case samples onto \({\mathop{ {\beta}}\limits^{\rightarrow}}\), respectively, and \(\alpha _0, \alpha _1\) are corresponding proportions  
We may also perform a nondirectional test with the arrow of \({\mathop{ {\beta}}\limits^{\rightarrow}}\) ignored, by using a twotailed ztest or ttest, which is suggested in Table 2(1) of Ref. Xu (2015a) as one example of the boundarybased twosample test or BBT in short  
Learning LDA score  We may also perform either a directional test or a nondirectional test as above, but with \({\mathop{ {\beta}}\limits^{\rightarrow}}\) obtained by 
(a) Support vector machine (SVM) (Suykens 1999; Suykens et al. 2002), as suggested in Table 4(c) of Ref. Xu (2015a)  
(b) Sparse logistic regression (Shevade and Keerthi 2003; Koh et al. 2007) 
Moreover, extensions can be made from a vector \(\mathbf{s}\) to a matrix. As addressed in Ref. Xu (2015a), many tasks of big data analyses demand extending vectorbased sampling units to sampling units in matrix format. Illustrated in Fig. 7a is a data cubic encountered in the case–control studies. Accordingly, we encounter matrixvariate tests in matrixvariate logistic regression (see the details from Eqs. (48) to (57) in Ref. Xu 2015a) and matrixvariate discriminative analysis (see the details from Eqs (33) to (43) in Ref. Xu 2015a). For the former, test is made on regression coefficients in two vectors, which is still a multivariate test. For the latter, the situation becomes quite different, and further details are addressed in the sequel.
Three feasible approximate techniques for matrixvariate twosample test
Method  Description 

(a) Multivariate test per probe  If probes are independent, we make a multivariate test on each \([condition, \ sample]\) matrix slice per probe. Such a multivariate test can be implemented by Algorithm 1 or one of the methods given in Table 2 
(b) LDAbased multivariate test  we make the map \(s_u(f)= \mathbf{s}^T {\mathop{ {\beta}}\limits^{\rightarrow}}\) onto \({\mathop{ {\beta}}\limits^{\rightarrow}}\) per probe f with \({\mathop{ {\beta}}\limits^{\rightarrow}}\) obtained by either FDA or learningbased methods in Table 2, and make a multivariate test on \([ s_u(f_1), \cdots , s_u(f_g)]^T\) to consider multiple probes \(f_1, \ldots , f_g\) jointly 
(c) Bilinear MDAbased test  We make the matrixvariate discriminative analysis (MDA) (see Eq.(33) & Eq.(34) in Ref. Xu (2015a)) to obtain \(\mathbf{v}, {\mathop{ {\beta}}\limits^{\rightarrow}}\), based on which we test \({\mathop{ {\beta}}\limits^{\rightarrow}}= \mathbf{0}\) by a multivariate test on \(\mathbf{s}_v= \mathbf{v}^T \mathbf{S}\) given \(\mathbf{v}\) fixed and test \(\mathbf{v}= \mathbf{0}\) by a multivariate test on \(\mathbf{s}_{\mathop{ {\beta}}\limits^{\rightarrow}}= {\mathop{ {\beta}}\limits^{\rightarrow}}^T \mathbf{S} ^T\) given \({\mathop{ {\beta}}\limits^{\rightarrow}}\) fixed 
Alternatively, we may make test on the scalar statistics \(s_{v{\mathop{ {\beta}}\limits^{\rightarrow}}}= \mathbf{v}^T \mathbf{S} {\mathop{ {\beta}}\limits^{\rightarrow}}\) 
These existing case–control studies can be roughly classified into two classes featured by columnwise versus rowwise bipartitions of the data cubic, as illustrated in Fig.7b–d, respectively. Examples of Fig. 7b can be found in most of the SNP analyses in GWAS and those gene expression studies under a single condition (e.g. from a tumour tissue only). Examples of Fig. 7c, d1 can be found in many current studies of gene expression with tumour versus its paired adjacent tissue. All of these existing efforts are featured by making univariate tests.
One widely adopted existing practice is making a NT test for getting a set \(F_{NT}\) of significant differentiation expression (DE) probes. However, the resulted biomarkers may not be optimal in the sense of differentiating phenotypes, because a differentiation expression between T versus N may not well cope with the distinctions between phenotypes. For prognosis purpose, there are also efforts that further make a PT test in Fig. 7d(1) for getting \(F_{PT}\subseteq F_{NT}\). Even so, the selection of \(F_{PT}\) is merely based on either the tumour expression or the fold change of T over N, without a best use of information contained in the (T,N) pair for differentiations between phenotypes.
Proposed recently in Ref. Xu (2015a), considering both T and N jointly in a 2D vector by a twovariate PT test paves a new road for reconsidering the task. As shown in Fig. 7d(2), samples of \(PT_1\) and \(PT_2\) can be well separated by a line on the 2D scatter plot. Transforming the 2D scatter plot into a plot on vertical line, however, it becomes no longer possible to separate \(PT_1\) and \(PT_2\). In other words, even in the cases that we are unable to identify biomarkers for distinguishing \(PT_1\) versus \(PT_2\) in the existing ways as shown in Fig. 7b–d (1), we may still find biomarkers for distinguishing \(PT_1\) versus \(PT_2\) by the new method.
Discussions
Whole genome sequencing analyses
 1.
Disturbing influences of those “donotcare” variants may be reduced such that the ability of identifying variants for significant differentiation can be considerably improved.
 2.
Directions of risk effect versus preventive effect with the multiple variants are taken in consideration.
 3.
Important samples may be identified by observing whether their critical variants have major contributions to significant differentiation, as addressed at the bottom of Algorithm 1.
 4.
Extensions of the joint multiplevariant sequencing analysis may be made not only to identify variants that significantly differentiate tumour versus normal but also to identify variants that significantly differentiate a pair of phenotypes.
Genomescale expression profile of mRNA and lncRNA expression
In the existing studies on genomescale expression profile of mRNA or/and lncRNA, there are many examples of the case–control study on the data cubic shown in Fig. 7d, with testing made in one of the ways shown in Fig. 7b, d(1). Instead, the new method shown in Fig. 7d(2) provides a better choice.
Twovariate TP tests : implementations and applications
Implementations  Applications 

1. Make the twovariate PT test by the method given in Table 3 (a), which provides an improvement on making the twovariate PT test by Hotelling test as suggested in Table 8(1)(a) of Ref. Xu (2015a)  (a) Identify mRNA and lncRNA biomarkers for tumour vs normal in expression analysis 
2. Make the FDAbased PT test by Table 3 (b), especially onetailed ztest or onetailed ttest by Eq. (32), which is a complementary to the FDAbased PT test listed in Table 2(1) of Ref. Xu (2015a), where a univariate twotailed t test is made  (b) Identify mRNA and lncRNA biomarkers for 3year & 5year survival in expression analysis 
3. Find probes that significantly differentiate not only phenotypes but also abnormal vs normal, as well as their common part  (c) Identify mRNA and lncRNA biomarkers for cancer grades I, II, III, & IV in expression analysis (d) For each of the above cases (a)(b)(c), we use the FDA projection \(s_u(f)= \mathbf{s}^T {\mathop{ {\beta}}\limits^{\rightarrow}}\) to replace the original expression value for painting heatmaps and making the corresponding clustering analysis 
Conclusions
Instead of understanding and making multivariate test in a single rejection, multivariate test actually consists of a hierarchy of numerous tests organised in a lattice taxonomy, with the bottom level in the lowest rejection collegiality (the largest p value) and the top level in the highest rejection collegiality (the smallest p value), while the ones on the intermediate levels represent different situations in which the null hypothesis is rejected and are featured by different p values. The outcomes consist of not only whether the null hypothesis is rejected significantly as a whole, but also those combinations of multiple components that are responsible for a significance of rejecting the null hypothesis, and those probes that contribute considerably to a significance of rejecting the null hypothesis. Not only detailed implementations are presented, but also several potentials are addressed on possible applications to expression profilebased biomarker identification and exome sequencingbased joint SNV detection.
Declarations
Acknowledgements
This work was partialy supported by the National Basic Research Program of China (973 Program 2009CB825404).
Competing interests
The author has no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
 Adhikari K, Reales G, Smith AJ, Konka E, Palmen J, QuintoSanchez M, AcuñaAlonzo V, Jaramillo C, Arias W, Fuentes M et al (2015) A genomewide association study identifies multiple loci for variation in human ear morphology. Nat Commun 6:7500View ArticleGoogle Scholar
 BaiZ D, Saranadasa H (1996) Effect of high dimension: by an example of a two sample problem. Statistica Sinica 6(2):311–329MathSciNetGoogle Scholar
 Bansal V, Libiger O, Torkamani A, Schork NJ (2010) Statistical analysis strategies for association studies involving rare variants. Nat Rev Genetics 11(11):773–785View ArticleGoogle Scholar
 Chapman J, Whittaker J (2008) Analysis of multiple snps in a candidate gene or region. Genetic Epidemiol 32(6):560View ArticleGoogle Scholar
 Demidenko E (2013) Mixed models: theory and applications with R. probability and statistics. John Wiley and Sons, HobokenGoogle Scholar
 Dempster AP (1958) A high dimensional two sample significance test. Ann Math Stat 995–1010Google Scholar
 DempsterA P (1960) A significance test for the separation of two highly multivariate small samples. Biometrics 16(1):41–50MathSciNetView ArticleGoogle Scholar
 Evangelou E, Ioannidis JP (2013) Metaanalysis methods for genomewide association studies and beyond. Nat Rev Genet 14(6):379–389View ArticleGoogle Scholar
 Fan R, Knapp M (2003) Genome association studies of complex diseases by casecontrol designs. Am J Hum Genet 72(4):850–868View ArticleGoogle Scholar
 Ferguson J, Wheeler W, Fu Y, ProkuninaOlsson L, Zhao H, Sampson J (2013) Statistical tests for detecting associations with groups of genetic variants: generalization, evaluation, and implementation. Euro J Human Genet 21(6):680–686View ArticleGoogle Scholar
 Fisher RA (1932) Statistical methods for research workers, 4th edn, Oliver and Boyd, Edinburgh, pp 99–101Google Scholar
 Gudmundsson J, Sulem P, Gudbjartsson DF, Masson G, Agnarsson BA, Benediktsdottir KR, Sigurdsson A, Magnusson OT, Gudjonsson SA, Magnusdottir DN (2012) A study based on wholegenome sequencing yields a rare variant at 8q24 associated with prostate cancer. Nat Genet 44(12):1326–1329View ArticleGoogle Scholar
 Han F, Pan W (2010) A dataadaptive sum test for disease association with multiple common or rare variants. Human Heredity 70(1):42–54View ArticleGoogle Scholar
 Hotelling H (1931) The generalization of student’s ratio. Ann Math Stat 2(3):360–378View ArticleGoogle Scholar
 Kiezun A, Garimella K, Do R, Stitziel NO, Neale BM, McLaren PJ, Gupta N, Sklar P, Sullivan PF, Moran JL (2012) Exome sequencing and the genetic basis of complex traits. Nature genetics 44(6):623–630Google Scholar
 Koh K, Kim SJ, Boyd SP (2007) An interiorpoint method for largescale l1regularized logistic regression. J Mach Learn Res 8(8):1519–1555MathSciNetMATHGoogle Scholar
 Lee S, Abecasis GR, Boehnke M, Lin X (2014) Rarevariant association analysis: Study designs and statistical tests. Am J Human Genet 95(1):5–23View ArticleGoogle Scholar
 Lee S, Wu MC, Lin X (2012) Optimal tests for rare variant effects in sequencing association studies. Biostatistics 13(4):762–775View ArticleGoogle Scholar
 Li H, Gui J (2004) Partial cox regression analysis for highdimensional microarray gene expression data. Bioinformatics 20(suppl 1):208–215View ArticleGoogle Scholar
 Li B, Leal SM (2008) Methods for detecting associations with rare variants for common diseases: application to analysis of sequence data. Am J Human Genet 83(3):311–321View ArticleGoogle Scholar
 Morgenthaler S, Thilly WG (2007) A strategy to discover genes that carry multiallelic or monoallelic risk for common diseases: a cohort allelic sums test (cast). Mut Res/Fund Mol Mech Mutag 615(1):28–56View ArticleGoogle Scholar
 MorrisA P, Zeggini E (2010) An evaluation of statistical approaches to rare variant analysis in genetic association studies. Genetic Epidemiol 34(2):188View ArticleGoogle Scholar
 Price AL, Kryukov GV, de Bakker PI, Purcell SM, Staples J, Wei LJ, Sunyaev SR (2010) Pooled association tests for rare variants in exonresequencing studies. Am J Human Genet 86(6):832–838View ArticleGoogle Scholar
 ShevadeS K, KeerthiS S (2003) A simple and efficient algorithm for gene selection using sparse logistic regression. Bioinformatics 19(17):2246–2253View ArticleGoogle Scholar
 SrivastavaM S (2007) Multivariate theory for analyzing high dimensional data. J Jpn Stat Soc 37(1):53–86View ArticleGoogle Scholar
 Suykens JA, Van Geste lT, De Brabanter J, De Moor B, Vandewalle J (2002) Least squares support vector machines. World Scientific Publishing, SingaporeView ArticleMATHGoogle Scholar
 SuykensJ A, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300View ArticleGoogle Scholar
 SwansonD M, Blacker D, AlChawa T, Ludwig KU, Mangold E, Lange C (2013) Properties of permutationbased gene tests and controlling type 1 error using a summary statistic based gene test. BMC Genet 14(1):108View ArticleGoogle Scholar
 Tu S, Xu L (2011) An investigation of several typical model selection criteria for detecting the number of signals. Front Elect Electronic Eng China 6(2):245–255MathSciNetView ArticleGoogle Scholar
 Tu S, Xu L (2014) Learning binary factor analysis with automatic model selection. Neurocomputing 134:149–158View ArticleGoogle Scholar
 WuM C, Lee S, Cai T, Li Y, Boehnke M, Lin X (2011) Rarevariant association testing for sequencing data with the sequence kernel association test. Am J Hum Genet 89(1):82–93View ArticleGoogle Scholar
 Xu L (2003) Independent component analysis and extensions with noise and time: a bayesian yingyang learning perspective. Neural Inform Process Lett Rev 1:1–52Google Scholar
 Xu L (2009) Independent subspaces. In: Rabunal JR, Dorado J, Sierra AP (eds.) Encyclopedia of Artificial Intelligence. IGI Global Snippet, Hershey, Pennsylvania, pp 892–901Google Scholar
 Xu L (2011) Codimensional matrix pairing perspective of byy harmony learning: hierarchy of bilinear systems, joint decomposition of datacovariance, and applications of network biology. Front Electr Electron Eng China 6:86–119. A special issue on Machine Learning and Intelligence Science: IScIDE2010 (A)Google Scholar
 Xu L (2015a) Bilinear matrixvariate analyses, integrative hypothesis tests, and casecontrol studies. Appl Inform 2(1):1–39View ArticleGoogle Scholar
 Xu L (2015b) Further advances on bayesian ying yang harmony learning. Appl Inform 2(5)Google Scholar
 ZaykinD V (2011) Optimally weighted ztest is a powerful method for combining probabilities in metaanalysis. J Evol Biol 24(8):1836–1841View ArticleGoogle Scholar
 Zhou X, Stephens M (2014) Efficient multivariate linear mixed model algorithms for genomewide association studies. Nat Methods 11(4):407–409View ArticleGoogle Scholar