Projection-embedded BYY learning algorithm for Gaussian mixture-based clustering

Abstract

On learning the Gaussian mixture model, existing BYY learning algorithms are featured by a gradient-based line search with an appropriate stepsize. Learning becomes either unstable if the stepsize is too large or slow and gets stuck in a local optimal solution if the stepsize is too small. An algorithm without a learning stepsize has been proposed with expectation-maximization (EM) like two alternative steps. However, its learning process may still be unstable. This paper tackles this problem of unreliability by a modified algorithm called projection-embedded Bayesian Ying-Yang learning algorithm (pBYY). Experiments have shown that pBYY outperforms learning algorithms developed from not only minimum message length with Jeffreys prior (MML-Jef) and Variational Bayesian with Dirichlet-Normal-Wishart (VB-DNW) prior but also BYY with these priors (BYY-Jef and BYY-DNW). pBYY obtains the superiority with an easy implementation, while DNW prior-based learning algorithms suffer a complicated and tedious computation load. The performance of pBYY has also been demonstrated on the Berkeley Segmentation Dataset for the topic of unsupervised image segmentation. The resulted performances of semantic image segmentation have shown that pBYY outperforms not only MML-Jef, VB-DNW, BYY-Jef, and BYY-DNW but also three leading image segmentation algorithms, namely gPb-owt-ucm, MN-Cut, and mean shift.

Background

Introduction

Gaussian mixture model (GMM) has been widely used in different areas, e.g., clustering, image segmentation (Zhang et al. [2001]), speaker identification (Reynolds [1995]), document classification (Nigam et al. [2000]), market analysis (Chiu and Xu [2001]), etc. Learning a GMM consists of parameter learning for estimating all unknown parameters and model selection for determining the number of Gaussian components k. Parameter learning is usually implemented under the maximum likelihood principle by an expectation-maximization (EM) algorithm (Redner and Walker [1984]). A conventional model selection approach is featured by a two-stage implementation, which suffers from a huge computation because it requires parameter learning for each candidate GMM. Moreover, parameter learning will become less reliable as k becomes larger, which implies more free parameters.

One road to tackle these problems is referred as automatic model selection that automatically determines k during parameter learning. An early effort is rival penalized competitive learning (RPCL) (Xu et al. [1992]; Xu [1998]) with the number k automatically determined during learning. Automatic model selection may also be approached via appropriate priors on unknown parameters by Bayesian approaches. Two examples are minimum message length (MML) (Figueiredo and Jain [2002]) and variational Bayesian (VB) (Corduneanu and Bishop [2001]). Firstly proposed in (Xu [1995]) and systematically developed in the past two decades, Bayesian Ying-Yang (BYY) learning provides not only a new model selection criteria but also a family of learning algorithms that is capable of automatic model selection during parameter learning, with details referred to recent tutorial and survey by (Xu [2010], [2012]).

A systematic comparison has been recently made by (Shi et al. [2011]) among MML, VB, and BYY with two types of priors. One is the Jeffreys prior and another is a parametric conjugate prior that imposes a Dirichlet prior on mixing weights and a joint normal-Wishart prior on mean vectors and covariance matrices, shortly denoted as DNW. Automatic model selection performances of these approaches are evaluated through extensive experiments, with several interesting empirical findings. Among them, it has been shown that BYY considerably outperforms both VB and MML. Different from VB and MML that rely on appropriate priors to perform model selection, BYY is capable of selecting model automatically even without imposing any priors on parameters, while its performance can be further improved with appropriate priors incorporated. Similar findings have also been obtained (Zhu et al. [2013]), where a simplified BYY learning algorithm with DNW priors is shown to outperform or at least be competitive to the existing state-of-the-art image segmentation methods.

The algorithms by (Shi et al. [2011]) for implementing BYY are featured by a gradient-based line search with an appropriate stepsize. Learning becomes either unstable if this stepsize is too large or slow and gets stuck in a local optimal solution if the stepsize is too small. Given in Algorithm two (Xu [2009]) and Equation (11) (Xu [2010]), there is a Ying-Yang two-step alternation algorithm that is similar to the EM algorithm without a learning stepsize for the learning procedure. However, the Ying step (Xu [2010]) ignores the constrain that the covariance matrix of each Gaussian component must be positive definite matrix, so the learning procedure may become unstable.

To constrain the covariance matrix as a positive definite matrix, this paper proposes a projection operation into the Yang step, which results in a modified algorithm called projection-embedded BYY learning algorithm or shortly denoted as pBYY. To facilitate its implementation, we also add a Kullback Leibler divergence-based indicator into the algorithm to improve the detection of redundant Gaussian components. Experiments have shown that pBYY significantly outperforms not only the Jeffreys-based MML (Figueiredo and Jain [2002]) and the DNW-based VB but also the BYY learning algorithms with these two types of priors (Shi et al. [2011]), and it further avoids the cost of complicated and tedious computation brought by the DNW prior.

Gaussian mixture model and four learning principles

GMM assumes that an observation x∈R^d is drawn from the following mixture of k Gaussian distributions:

\begin{array}{l}q\left(x\right|\theta )=\sum _{i=1}^k{\alpha }_{i}G\left(x|{\mu }_i,{\Sigma }_i\right),\\ \theta ={\left\{\alpha ,{\mu }_i,{\Sigma }_i\right\}}_{i=1}^k,{\alpha }_i\ge 0,\sum _{i=1}^k{\alpha }_i=1,\end{array}

(1)

where G(x|μ,Σ) denotes a Gaussian density with a mean μ and a covariance matrix Σ.

GMM can be also regarded as a latent variable model by introducing a binary latent vector y= [y ₁,y ₂,…,y _k ]^T, subject to y _i ∈{0,1},∀i, and \sum _{i=1}^k{y}_i=1, the latent variable y _i =1 means that the random variable x is drawn from i th Gaussian component. The generative process of an observation x is interpreted as that y is sampled from a multinomial distribution with probabilities α and then x is randomly generated by the i th Gaussian component with y _i =1. Let X∈R^d×n denote the set of n i.i.d. d-dimension observation samples, Y∈R^k×n denote the set of latent vectors for the observable set X, we have the following:

\begin{array}{cc}q(X,Y|\theta )& =q\left(X\right|Y,\theta \left)q\right(Y\left|\theta \right),\\ q\left(X\right|Y,\theta )& =\prod _{t=1}^n\prod _{i=1}^{k}G{\left(x_t\right|{\mu }_i,{\Sigma }_i)}^{y_{\mathit{\text{it}}}},\\ q\left(Y\right|\theta )& =\prod _{t=1}^n\prod _{i=1}^k{\alpha }_i^{y_{\mathit{\text{it}}}}.\end{array}

(2)

Learning a GMM consists of parameter learning for estimating all the unknown parameters in θ and model selection for determining the number of Gaussian components k, which can be implemented differently under different learning principles.

The most widely used principle is called the maximum likelihood (ML), that is, we estimate θ by

\begin{array}{cc}\underset{\theta }{max}& q\left(X\right|\theta ),\\ q\left(X\right|\theta )& =\sum _{Y}q\left(X\right|Y,\theta \left)q\right(Y\left|\theta \right)=\prod _{t=1}^{n}q\left(x_t\right|\theta ).\end{array}

(3)

The ML learning with a known k is typically made by the well known EM algorithm (Redner and Walker [1984]). However, an unknown k is poorly estimated by Equation (3) when the sample number n is not large enough. The task of determining an appropriate k is called model selection, which is usually made in a two-stage implementation with the help of a model selection criterion. However, such a two-stage implementation suffers from a huge computation and an unreliable estimation. The problems are tackled by automatic model selection that automatically determines k during learning θ without such a two-stage implementation.

There are three Bayesian related learning principles that can be implemented with such the property of automatic model selection.

One is called minimum message length (MML) (Wallace and Dowe [1999]), which is actually an information theoretic restatement of Occam’s Razor. The MML was introduced to learn GMM with the property of automatic model selection (Figueiredo and Jain [2002]). Learning is made by the following maximization:

\begin{array}{cc}\underset{\theta }{max}& J_{\mathit{\text{MML}}}\left(X\right|\theta ),\\ J_{\mathit{\text{MML}}}\left(X\right|\theta )& =lnq\left(X\right|\theta )+lnq(\theta )-\frac{1}{2}ln|\mathbf{\text{I}}\left(\theta \right)|,\end{array}

(4)

where |I(θ)| represents the determinant of Fisher information matrix with respect to (w.r.t) Θ. Equation (4) is mathematically equivalent to a maximum a posteriori (MAP) approach with modifying a proper prior q(θ) into being proportional to q(θ)/|I(θ)|^1/2.

Using the Jeffreys prior q(θ)∝|I(θ)|^1/2 directly, Equation (4) degenerates to be ML learning principle. To avoid this situation, Figueiredo and Jain ([2002]) considered the following:

\begin{array}{l}ln\frac{q\left(\theta \right)}{\left|\mathbf{\text{I}}\right(\theta ){|}^{1/2}}\approx -\frac{\rho }{2}\sum _{i=1}^{k}ln{\alpha }_i-\frac{k(\rho +1)}{2}lnN,\end{array}

(5)

where ρ=d+0.5d(d+1) is the number of free parameters in each Gaussian component. In (Shi et al. [2011]), it has shown that some improvement can be obtained by an algorithm that implements the MML principle with the help of a Dirichlet prior and a joint normal-Wishart prior (shortly DNW prior).

The other Bayesian related learning principle is called variational Bayesian (Corduneanu and Bishop [2001]). The naive Bayes considers q(X|θ)q(θ) with a prior q(θ) which takes a strong role. Unfortunately, a poor q(θ) may affect the learning performance seriously. Such a bad influence can be smoothed out by considering the following marginal distribution:

\begin{array}{l}q\left(X\right)=\int q\left(X\right|\theta \left)q\right(\theta )\mathrm{d\theta .}\end{array}

(6)

However, it is difficult in computation with integral. The VB tackles this difficulty via constructing a lower bound J _VB with the help of Jensen’s inequality as follows:

\begin{array}{l}max{J}_{\mathit{\text{VB}}},\\ J_{\mathit{\text{VB}}}=\int p(\theta ,Y|X)ln\left[\frac{q(X,Y|\theta \left)q\right(\theta )}{p(\theta ,Y|X)}\right]\mathit{\text{dY}}\mathrm{d\theta .}\\ lnq\left(X\right)\ge J_{\mathit{\text{VB}}}.\end{array}

(7)

The goal is to choose a suitable posterior distribution p(θ,Y|X) from a distribution family , so that the lower bound J _VB can readily be evaluated and yet sufficiently flexible. One challenge is to provide a suitable distribution family . In (Corduneanu and Bishop [2001]), the family of prior distribution can be approximately factorized as follows:

\begin{array}{l}p(\theta ,Y|X)=p(Y\left|X\right)\prod _{i}p\left({\theta }_i\right|X).\end{array}

(8)

With q(X,Y|θ) by Equation (2) and a DNW prior q(θ), the above p(θ _i |X) can be obtained with p(Y|X) and p(θ _j |X)∀j≠i given by the following equation (Bishop and Nasrabadi [2006]):

\begin{array}{l}p\left({\theta }_i\right|X)=\frac{\underset{j\ne i}{\int }p\left(Y\right|X\left)p\right({\theta }_j\left|X\right)lnq(X,Y,\theta )\mathrm{d\theta dY}}{\underset{j\ne i}{\int }p\left(Y\right|X\left)p\right({\theta }_j\left|X\right)lnq(X,Y,\theta )\mathrm{d\theta dY}\mathit{\text{dX}}}.\end{array}

A tight bound is unavailable to be obtained by Equation (8), which affects the learning performances. Also, DNW is quite tedious and has hyperparameters \left\{\lambda ,\xi ,m_i,\frac{{\Sigma }_i^{-1}}{\beta },\Phi ,\gamma \right\} to be updated, which is time-consuming and may fall into local optimal. To avoid the tedious computation of the DNW prior-based VB, an algorithm for implementing VB principle is developed (Shi et al. [2011]) with the help of the Jeffreys prior via approximately using a block-diagonal complete data Fisher information (Figueiredo and Jain [2002]).

The last Bayesian related principle is BYY harmony learning. Firstly proposed by (Xu [1995]) and systematically developed in the past two decades, BYY harmony learning on typical structures leads to new model selection criteria, new techniques for implementing learning regularization, and a class of algorithms that approach automatic model selection during parameter learning. Readers are referred to (Xu [2010], [2012], [2014]) for latest systematical introductions about BYY harmony learning.

Briefly, a BYY system consists of Yang machine and Ying machine, corresponding to two types of decomposition, namely, Yang p(R|X)p(X) and Ying q(X|R)q(R) respectively, where the data X is regarded to be generated from its inner representation R={Y,θ} that consists of latent variables Y and parameters θ, supported by a hyperparameter set Ξ. The harmony measure is mathematically expressed as follows:

\begin{array}{l}H\left(p\left|\right|q\right)=\int p\left(R|X\right)p\left(X\right)ln\left[q\left(X\right|R\left)q\right(R)\right]\mathrm{dXdR.}\end{array}

(9)

Maximizing this H(p||q) leads to not only a best matching between the Ying-Yang pair but also a compact model with a least complexity. Such an ability can be observed from several perspectives (see Section 4 in (Xu [2010])).

Applied to GMM by Equation (2), we have R={Y,θ} and q(R)=q(Y|θ)q(θ|Ξ). Comparing Equation (9) and Equation (7), the key difference is that there is only q(X,Y|θ)q(θ) inside the basket ln[∗] for the BYY harmony learning while there is also a denominator p(θ,Y|X) for the VB learning. Maximizing J _VB leads to a best match between q(X,Y|θ)q(θ) and p(θ,Y|X), while maximizing H(p||q) leads to not only such a best match but also a modeling of q(X,Y|θ)q(θ) in a least complexity. Readers are referred to Section 4 and its figure five in (Xu [2012]) for various aspects of this key difference, as well as how they relate and differ from MML and minimum description length (MDL) (Barron et al. [1998]; Rissanen [1978]).

Maximizing H(p||q) leads to specific algorithms according to not only what types of q(θ|Ξ) are chosen for the Ying machine but also how the structure of p(θ,Y|X) is designed for the Yang machine. Details are referred to Section 4.2 in (Xu [2010]) and Section 3.2 in (Xu [2012]). For the GMM by Equation (2), we introduce two typical examples here.

One example is p(θ,Y|X) given by Equation (8) together with a DNW prior. Putting them into Equation (9), the DNW prior-based BYY harmony learning algorithm has been developed for maximizing H(p||q) in (Shi et al. [2011]). Extensive experiments have shown that the DNW prior-based BYY considerably outperforms both VB and MML for any type of priors and with whether or not hyper-parameters optimized. As the hyper-parameters of DNW prior are optimized by its corresponding learning principle, BYY further improves its performance and outperforms the others significantly, because learning hyper-parameters is a part of the entire BYY harmony learning. However, both VB and MML deteriorate when there are too many free hyper-parameters, especially the performance of VB drops drastically. The reason is that VB and MML maximize the marginal likelihood via variational approximation and Laplace approximation, respectively, where maximizing the marginal likelihood with respect to a free priori q(θ|Ξ) makes it tend to the maximum likelihood.

Another example is the following structure:

\begin{array}{lcr}p(\theta ,Y|X)& =& p\left(Y\right|X,\theta \left)p\right(\theta \left|X\right),\\ p\left(Y\right|X,\theta )& =& \frac{q(X,Y|\theta )}{\int q(X,Y|\theta )\mathit{\text{dY}}},\\ p\left(\theta \right|X)& \text{is free of structure}.\end{array}

(10)

Maximizing H(p||q,Ξ) with respect to p(θ|X) makes Equation (9) simplified into

\begin{array}{l}\underset{\theta }{max}H\left(\theta \right),H\left(\theta \right)=H_0\left(\theta \right)+lnq\left(\theta \right),\\ H_0\left(\theta \right)=\sum _{t=1}^n\sum _{i=1}^{k}p\left(i\right|x_t,\theta )ln\left[G\left(x_t\right|{\mu }_i,{\Sigma }_i){\alpha }_i\right],\\ p\left(i\right|x_t,\theta )=\frac{{\alpha }_{i}G\left(x_t\right|{\mu }_i,{\Sigma }_i)}{\sum _{i=1}^k{\alpha }_{i}G\left(x_t\right|{\mu }_i,{\Sigma }_i)}.\end{array}

(11)

Automatic model selection and two-step alternation

Given a known k, learning the unknown parameters θ on a GMM is usually implemented under the maximum likelihood principle by an EM algorithm (Redner and Walker [1984]), which is one typical instance of Algorithm 1 featured by a two-step alternation. As remarked at the bottom of the table, we get the EM algorithm after simply removing the lines of trimming with

\begin{array}{l}{p}_{\mathit{\text{it}}}=p\left(i|x_t,{\theta }^{\mathit{\text{new}}}\right),{\eta }_i=0,{\rho }_i=0,i=1,\dots ,k,\end{array}

(12)

where p(i|x _t ,θ) is the Bayes posteriori probability as follows:

\begin{array}{lcr}p\left(i\right|x_t,\theta )& =& \frac{{\alpha }_{i}G\left(x_t|{\mu }_i,{\Sigma }_i\right)}{\sum _{i=1}^k{\alpha }_i^{p}G\left(x_t\right|{\mu }_i,{\Sigma }_i)},\\ \theta & =& {\left\{{\theta }_i\right\}}_{i=1}^k,{\theta }_i=\{{\alpha }_i,{\mu }_i,{\Sigma }_i\}.\end{array}

(13)

Generally, η _i ,ρ _i come from a priori distribution that takes a regularization role. This role is shut off by simply setting them to zero. When η _i =0,ρ _i >0, the EM algorithm is extended to the smoothed EM algorithm that was firstly proposed in 1997 (Xu [2010]). Also, we get the EM algorithm for naive Bayes with Jeffreys priori on α _i ,Σ _i with

\begin{array}{l}{\eta }_i=\frac{d+0.5d(d+1)-1}{2n},{\rho }_i=\frac{d}{2n}.\end{array}

(14)

An unknown k is poorly estimated via the ML learning by Equation (3), especially when the sample number n is not large enough. The task of determining an appropriate k is made by model selection, which is usually made in a two-stage implementation. The first stage enumerates k to get a set of candidate models with unknown parameters of each candidate estimated by the EM algorithm. In the second stage, the best candidate is selected by a model selection criterion. Examples of such criteria include Akaike’s information criterion (AIC) (Akaike [1974]), Bayesian inference criterion (BIC), minimum description length (MDL) criterion (which stems from another viewpoint but coincides with BIC when it is simplified to an analytically computable criterion), etc (Barron et al. [1998]; Rissanen [1978]). However, this two-stage implementation suffers from a huge computation because it requires parameter learning for each k\in \mathcal{ℳ}. Moreover, a larger k often implies more unknown parameters, thus parameter estimation becomes less reliable and the criterion evaluation reduces its accuracy (see Section 2.1 in (Xu [2010]) for a detailed discussion).

One road to tackle the problems is referred to automatic model selection that means to automatically determine an appropriate k during parameter learning. An early effort is RPCL (Xu et al. [1992]; Xu [1998]). The key idea is that not only the winning Gaussian component moves a little bit to adapt the current sample but also the rival (i.e., the second winner) Gaussian component is repelled a little bit from this sample to reduce a duplicated information allocation. As a result, an extra Gaussian component is driven far away from data.

A batch learning version of RPCL learning may be also obtained as one instance of Algorithm 1, simply with

\begin{array}{l}{p}_{\mathrm{\ell t}}=\left\{\begin{array}{cc}1,& {\ell }^{\ast }=arg\underset{j}{max}p\left(j|x_t,{\theta }^{\mathit{\text{new}}}\right),\\ -\gamma ,& \ell =arg\underset{\ell \ne {\ell }^{\ast }}{max}p\left(j|x_t,{\theta }^{\mathit{\text{new}}}\right),\\ 0,& \text{otherwise,}\end{array}\right.\end{array}

(15)

by which learning is made on a cluster when p _{ℓ t} =1 and penalizing or de-learning is made on a cluster when p _{ℓ t} =−γ. Usually, the penalizing strength is set γ≈0.005∼0.05. When γ=0, it degenerates to the so called hard-cut EM algorithm, see Equations (19) and (20) in (Xu [1995]).

According to its general formulation (e.g., see the last part of Section 2.1 in (Xu [2010])), automatic model selection is a nature of learning a mixture of k individual substructures with the following two features:

There is an indicator Ψ _j (θ) on θ or its subset, based on which a particular structural component j can be effectively discarded if its corresponding Ψ _j (θ)→0. Taking the GMM as an example, we may consider

\begin{array}{l}{\Psi }_j\left(\theta \right)={\alpha }_j,\mathit{\text{or}}{\Psi }_j\left(\theta \right)={\alpha }_j\mathit{\text{Tr}}\left[{\Sigma }_j\right].\end{array}

(16)

With initial k large enough, there is an intrinsic mechanism that drives such an indicator Ψ _j (θ) towards zero if the corresponding structure is redundant and thus can be effectively discarded.

Three Bayesian-related approaches introduced in the previous subsection can all be implemented with such a nature of automatic model selection. For both MML and VB, this nature comes from an appropriate prior q(θ|Ξ). Favorably, BYY is capable of automatic model selection even without imposing any priors on the parameters, and its performance can be further improved as appropriate priors are incorporated. Actually, the BYY harmony learning by maximizing H(p||q) bases on q(R)=q(Y|θ)q(θ|Ξ) to make model selection, with q(Y|θ) in a role that is not only equally important to q(θ|Ξ) but also easy computing, while q(θ|Ξ) is still handled in a way similar to MML and VB.

The BYY harmony learning by Equation (11) can be implemented by Algorithm 1, with the Yang step given as follows:

\begin{array}{lcr}{p}_{\mathit{\text{it}}}& =& p_{\mathit{\text{it}}}\left({\theta }^{\mathit{\text{new}}}\right),\\ p_{\mathit{\text{it}}}\left(\theta \right)& =& p\left(i\right|x_t,\theta )\left[1+{\delta }_{i,t}\left(\theta \right)\right],\\ {\delta }_{i,t}\left(\theta \right)& =& {\pi }_t\left({\theta }_i\right)-\sum _{i}p\left(i\right|x_t,{\theta }_i\left){\pi }_t\right({\theta }_i),\\ {\pi }_t\left({\theta }_i\right)& =& ln\left[G\left(x_t\right|{\mu }_i,{\Sigma }_i){\alpha }_i\right].\end{array}

(17)

The algorithm implements a BYY harmony learning without a priori lnq(θ) in Algorithm (1) by simply setting η _i =0,ρ _i =0 or a data smoothing based BYY harmony learning when η _i =0,ρ _i >0. Readers are referred to Section 3.1 of (Xu [2010]) for further details. Also, we may implement the Jeffreys priori based on BYY harmony learning by using Equation (14), see table one in (Shi et al. [2011]).

Methods

Learning unreliability and convex combination

As introduced in Section 3.1 of (Xu [2010]), the existing algorithms for implementing BYY principle come from taking a gradient of H(θ) by Equation (11) w.r.t a subset ϕ of parameters. That is, we consider

\begin{array}{l}{\nabla }_{\varphi }H\left(\theta \right)={\nabla }_{\varphi }{H}_0\left(\theta \right)+{\nabla }_{\varphi }lnq\left(\theta \right),\\ {\nabla }_{\varphi }{H}_0\left(\theta \right)=\sum _{t=1}^n\sum _{i=1}^k{p}_{i,t}\left(\theta \right){\nabla }_{\varphi }{\pi }_t\left({\theta }_i\right),\end{array}

(18)

with p _i,t (θ) and π _t (θ _i ) given in Equation (17).

Based on this gradient, one attempt to update parameters is gradient-based local search. The parameter ϕ can be updated iteratively as below:

\begin{array}{l}{\varphi }^{\mathit{\text{new}}}={\varphi }^{\mathit{\text{old}}}+\eta {\nabla }_{\varphi }H\left(\theta \right),\end{array}

(19)

where η>0 is a small learning stepsize. Both the BYY learning algorithm given in figure seven of (Xu [2010]) and the BYY-Jef algorithm given in table one of (Shi et al. [2011]) are derived from Equation (19) with the help of some computing tricks and simplification. However, the performance of such algorithms all depend on an appropriate stepsize. Learning becomes either unstable if η is too large or slow and gets stuck in a local optimal if η is too small. No such a learning stepsize is required for EM algorithms.

Another typical implementation attempts to make the BYY harmony learning by Equation (11) also in a Ying-Yang two-step alternation, as previously suggested in Section 2.1 and table one of (Xu [2012]). This two-step alternation algorithm is actually derived from approximately letting p _i,t (θ) in Equation (18) to be fixed at its value p _it =p _it (θ^new) such that we can solve the root of ∇ _ϕ H(θ)=0 subject to this fixation to get the Ying step in Algorithm 1.

Still, there lacks theoretical analyses that either guarantee the learning convergence or provide the convergence conditions. Oppositely, we find empirically that the learning process of this BYY two-step alternation may become unstable.

Actually, the root of ∇ _ϕ H(θ)=0 subject to p _it =p _it (θ^new) can be considerably deviated from the true root of ∇ _ϕ H(θ)=0 since this true root is coupled with p _it (θ) that varies with θ. Not only correctly solving the root of ∇ _ϕ H(θ)=0 is a challenging task but also it is unclear whether fixing p _it =p _it (θ^new) makes the learning procedure become unstable.

From the likelihood by Equation (3) and Equation (1), it can be observed that

\begin{array}{l}{\nabla }_{\varphi }lnq\left(X\right|\theta )=\sum _{t=1}^n\sum _{i=1}^{k}p(i|x_t,\theta ){\nabla }_{\varphi }{\pi }_t\left({\theta }_i\right),\end{array}

(20)

with p(i|x _t ,θ) given in Equation (13). Fixing p(i|x _t ,θ)=p _it =p (i|x _t ,θ^new), solving the root of ∇ _ϕ lnq(X|θ)=0 leads to the Ying step in Algorithm 1, or precisely the M step of the EM algorithm while letting p _it =p (i|x _t ,θ^new) is just the E step of the EM algorithm. As well known, the convergence of the EM algroithm has been theoretically proved. That is, though the root of ∇ _ϕ lnq(X|θ)=0 is also coupled with p(i|x _t ,θ) that varies with θ, this deviation actually does not affect the convergence.

The difference between p _it =p(i|x _t ,θ)=p (i|x _t ,θ^new) and p _it =p _it (θ^new) is that p(i|x _t ,θ),i=1,…,k remains to be probability with θ _i , while p _it (θ _i ),i=1,…,k given in Equation (17) are no longer the probabilities and even take negative values sometimes. Thus p _it (θ^new) is more sparse than p(i|x _t ,θ^new), and Yang step in the BYY theory introduces a nature of automatical model selection into the iteration procedure.

To further investigate the influence of replacing p (i|x _t ,θ^new) by p _it (θ^new), we now focus on Ying step, which can be reformulated as below:

\begin{array}{lcr}{\alpha }_i& =& \frac{\sum _{t=1}^n{p}_{\mathit{\text{it}}}}{N},\\ {\mu }_i& =& \sum _{t=1}^N\frac{p_{\mathit{\text{it}}}}{\sum _{t=1}^n{p}_{\mathit{\text{it}}}}{x}_t,\\ {\Sigma }_i& =& \sum _{t=1}^N\frac{p_{\mathit{\text{it}}}}{\sum _{t=1}^n{p}_{\mathit{\text{it}}}}(x_t-{\mu }_i){(x_t-{\mu }_i)}^T.\end{array}

(21)

For EM algorithm, both μ _i and Σ _i are constrained in the convex hulls spanned by x _t and (x _t −μ _i )(x _t −μ _i )^T, respectively, because its p _it still remains in the probability space. However, in BYY algorithm, p _it is no longer the probabilities and even take negative values sometimes. Thus, μ _i and Σ _i may break through their corresponding convex hulls. For GMM, the model parameters θ must satisfy following constrains:

\begin{array}{lcr}\sum _{i=1}^k{\alpha }_i=1,{\alpha }_i& \ge & 0,\forall i\in \{1,2,\dots ,k\},\\ {\Sigma }_i& \in & {ℝ}_{+}^{d\times d}\forall i\in \{1,2,\dots ,k\},\end{array}

(22)

where {ℝ}_{+}^{d\times d} denotes the set of positive semidefinite matrix of size d×d. Thus the updated α _i and Σ _i in BYY may no longer exist in their feasible regions sometimes. Instead of projecting α _i and Σ _i to the set of positive semidefinite matrix directly, we are motivated to project ∇ _ϕ H ₀(θ) back to the convex hull of local gradients ∇ _ϕ π _t (θ _i ), t=1,…,n, via projecting p _it (θ^new) onto the following set of probabilities to preserve more information of α _i and Σ _i :

\begin{array}{l}\mathcal{P}=\left\{p_1,\dots ,p_k:p_i\ge 0,\sum _{i=1}^k{p}_i=1\right\}.\end{array}

(23)

For updating each mean vector μ _i , we are encouraged to use p _it (θ^new), because the updating equation of μ _i is no longer a convex combination of all observable samples, and the redundant components can be pushed outside the convex hull; thus, this operation accelerates the speed of model selection.

The relative structure among the original {p _it (θ^new)} is encoded by the position of the vector p_t^H={\left[p_{1t}\left({\theta }^{\mathit{\text{new}}}\right),\dots ,p_{\mathit{\text{kt}}}\left({\theta }^{\mathit{\text{new}}}\right)\right]}^T in R^k. Projecting p_t^H from R^k to in Equation (23) means to find a vector p_t={\left[p_{1t},\dots ,p_{\mathit{\text{kt}}}\right]}^T\in \mathcal{P} that is the nearest one to p_t^H and thus best keeps the relative structure within elements of p_t^H. To be specific, we choose the nearest one in a sense of the least square distance, that is, we consider the following optimization problem:

\begin{array}{l}{p}_t^{\ast }=\mathit{\text{arg}}\underset{p\in \mathcal{P}}{min}{\left|\left|p-p_t^H\right|\right|}^2.\end{array}

(24)

The above implementation maybe regarded as a two-step approach of making the BYY harmony learning by Equation (17) under a principle of multiple convex combination preservation (Xu [2014]).

Fast approximation and pBYY-Jef algorithm

The problem Equation (24) is often encountered in the literature of applied mathematics and scientific computing and tackled by several algorithms such as variants of the method of alternating projections (Bauschke and Borwein [1993]) and variants of Dykstra’s algorithm (Bauschke and Borwein [1994]). However, these algorithms suffer from a huge computing cost, especially on a large-size data set.

Alternatively, we propose a fast approximation algorithm with two steps, motivated by the Kolmogorov’s criterion (see Chapter of 1 (Escalante and Raydan [2011])). Let \prod _{\mathcal{S}}\left(x\right) denote the projection point of an arbitrary point x\in {ℝ}^k onto a non-empty closed convex set \mathcal{S}\subset {ℝ}^n; Kolmogorov’s criterion states that z^{\ast }=\prod _{\mathcal{S}}\left(x\right) if and only if z^{\ast }\in \mathcal{S} and (z−z^∗)^T(x−z^∗)≤0 for all z\in \mathcal{S}, from which we can get the following:

Theorem 1.

Let {\mathcal{F}}_p=\{p_1,\dots ,p_k:\sum _{i=1}^k{p}_i=1\} with \mathcal{P}\subset {\mathcal{F}}_p, we have \prod _{\mathcal{P}}\left(x\right)=\prod _{\mathcal{P}}\prod _{{\mathcal{F}}_p}\left(x\right) for an arbitrary point x\in {ℝ}^k.

Proof.

Let z^{\prime }=\prod _{{\mathcal{F}}_p}\left(x\right), z^{\ast }=\prod _{\mathcal{P}}\left(x\right) and {z^{\prime }}^{\ast }=\prod _{\mathcal{P}}\left(z^{\prime }\right). From (z−z^{′ ∗})^T(z^′−z^{′ ∗})≤0 for all z\in \mathcal{P}, we have (z−z^{′ ∗})^T(z^′−x+x−z^{′ ∗})≤0 or (z−z^{′ ∗})^T(z^′−x)+(z−z^{′ ∗})^T(x−z^{′ ∗})≤0. It follows (z−z^{′ ∗})^T(z^′−x)=0 since z^{\prime }=\prod _{{\mathcal{F}}_p}\left(x\right) is the projection point of x to the hyperplane {\mathcal{F}}_p and thus orthogonal to the vector z−z^{′ ∗} that lies in this hyperplane {\mathcal{F}}_p. Therefore, we get the inequality (z−z^{′ ∗})^T(x−z^{′ ∗})≤0, which holds for all z\in \mathcal{P} and thus z^∗=z^{′ ∗} according to Kolmogorov’s criterion. End. □

Based on this theorem, we split the projection into two steps. First, we consider the following orthogonal projection of p_t^H onto the hyperplane {\mathcal{F}}_p :

\begin{array}{l}{f}_t=\left(I-\mathbf{\text{n}}{\mathbf{\text{n}}}^T\right)\left(p_t^H-f_0\right)+f_0,f_0=\frac{1}{k}\mathbf{\text{1}},\end{array}

(25)

where \mathbf{\text{n}}=\frac{1}{\sqrt{k}}\mathbf{\text{1}} is the normal vector of the hyperplane \sum _{i=1}^k{p}_i=1, f ₀ is the center point of the closed convex set , and all elements in 1∈R^k×1 are equal to 1.

Second, we further project f _t onto . However, accurately calculating the projecting point is still very time-consuming. Instead, we consider a fast approximation along the line between f _t and f ₀ as follows:

\begin{array}{l}{p}_t=\lambda f_0+(1-\lambda )f_t\end{array}

(26)

with a minimum λ that make p _t locate within .

In a summary, we get a modified algorithm as one new instance of Algorithm 1. Its Ying step remains unchanged but its Yang step gets {p _it (θ^new)} by Equation (17) and then makes the nearest projection onto by Equation (25) and Equation (26). For clarity, we rewrite Algorithm 1 into a detailed form in Algorithm-2 that is dedicated to implementing this projection-embedded BYY learning (shortly named pBYY).

The pBYY implementation repeats the Ying step and the Yang step alternatively. It gets out of the repeating circle in two cases. One is that learning is finally completed as the repeating circle converges with an unchanged k. The other is after trimming one Gaussian component with k reducing by 1, after which it goes to the line of initialization and start a new repeating circle. This re-initialization is helpful to avoid accumulation of estimating bias, though it requires extra computing costs. Whether we need this depending on a trade-off of computing cost versus estimating accuracy. We may remove this re-initialization by simply deleting the line ‘go to Initialization’.

Trimming a Gaussian component bases on an indicator Ψ _j (θ) as given in Equation (16). Empirically, we find that there are scenarios and add the following new indicator for detection:

\begin{array}{lcr}{\mathit{\text{KL}}}_{\mathit{\text{ij}}}& =& \int G\left(x\right|{\mu }_i,{\Sigma }_i)ln\frac{G\left(x\right|{\mu }_i,{\Sigma }_i)}{G\left(x\right|{\mu }_j,{\Sigma }_j)}\mathit{\text{dx}}\\ =& \frac{1}{2}\left[ln\frac{\left|{\Sigma }_j\right|}{\left|{\Sigma }_i\right|}-d+\text{Tr}\left(\underset{j}{\overset{-1}{\Sigma }}{\Sigma }_i\right)+d_{i,j}^M\right],\\ d_{i,j}^M& =& {({\mu }_i-{\mu }_j)}^T{\Sigma }_j^{-1}({\mu }_i-{\mu }_j).\end{array}

(27)

That is, we use the Kullback–Leibler (KL) divergence to measure the similarity between two Gaussian components. When K L _ij becomes more close to 0 for any j≠i, we may regard that the i th Gaussian component is redundant and thus discarded.

Results and discussion

Performance measures and algorithms

When samples locate in a space with its dimension less than 3, we can visualize and judge the clustering performance manually. However, samples are usually located in a high dimensional space for practical problems. Also, human evaluation is too subjective. In this paper, we consider four typical measures for clustering performance and model selection on number of clusters.

First, a traditional criterion to measure the performances of model selection could be named as the correct selection rate (CSR), namely how many times the algorithm gets the accurate number of clusters among a large number of trials. Sometimes, this criterion is argued to be too strict. For example, there exists four clusters in the set of observation samples. If an algorithm splits one cluster into two but gets the other three clusters correctly, this trial gets a zero count in computing CSR, though the clustering result still has some reasonable interpretation.

Second, one popular measure in the current literature is called variational information (VI), which evaluates the distance between one clustering result {\mathcal{C}}^{\prime } and the ground-truth as follows:

\begin{array}{lcr}\text{VI}(\mathcal{C},{\mathcal{C}}^{\prime })& =& H\left(\mathcal{C}\right)+H\left({\mathcal{C}}^{\prime }\right)-\text{MI}(\mathcal{C},{\mathcal{C}}^{\prime }),\\ H\left(\mathcal{C}\right)& =& -\sum _{i=1}^{k}P\left(i\right)logP\left(i\right),\\ \text{MI}(\mathcal{C},{\mathcal{C}}^{\prime })& =& \sum _{i=1}^k\sum _{j=1}^{m}P(i,j)\underset{2}{log}\frac{P(i,j)}{P\left(i\right)P\left(j\right)},\\ P\left(i\right)& =& \frac{\left|C_i\right|}{N},P(i,j)=\frac{|C_i\cap C_j^{\prime }|}{N},\end{array}

(28)

with |C _i | denoting the size of cluster C _i , where we get k clusters {C _i } in clustering and m clusters {C _j } in clustering {\mathcal{C}}^{\prime }. This MI denotes the mutual information that describes how much we can reduce the uncertainty about the cluster of a random sample when knowing its cluster in another clustering of the same set of observation samples (Wagner and Wagner [2007]). The smaller the VI value is, the better the performance is.

The last popular measure is called probabilistic Rand index (PRI). It further considers to partition the set of all (unordered) pairs of observation samples in into the disjoint union of the following sets:{\mathcal{R}}_{11}= {pairs that are in the same cluster under and {\mathcal{C}}^{\prime }}{\mathcal{R}}_{00}= {pairs that are in the different clusters under and {\mathcal{C}}^{\prime }}{\mathcal{R}}_{10}= {pairs that are in the same cluster under but in different ones under {\mathcal{C}}^{\prime }}{\mathcal{R}}_{01}= {pairs that are in the different clusters under but in the same under {\mathcal{C}}^{\prime }}.

Assume that each sample is randomly assigned to one cluster. The probability that two samples are in the same cluster in both partitions is p_{11}=\frac{1}{k}·\frac{1}{m}. Corresponding to the {\mathcal{R}}_{10}, {\mathcal{R}}_{01,} and {\mathcal{R}}_{00}, we get p_{10}=\frac{1}{k}·\left(1-\frac{1}{m}\right), p_{01}=\left(1-\frac{1}{k}\right)·\frac{1}{m}, and p_{00}=\left(1-\frac{1}{k}\right)·\left(1-\frac{1}{m}\right). Then, PRI can be expressed as follows (Carpineto and Romano [2012]):

\begin{array}{l}\text{PRI}(\mathcal{C},{\mathcal{C}}^{\prime })=\frac{w_{11}{n}_{11}+w_{00}{n}_{00}}{w_{11}{n}_{11}+w_{10}{n}_{10}+w_{01}{n}_{01}+w_{00}{n}_{00}},\end{array}

(29)

where n _ab =|R _ab | and w _ab =− log2(p _ab ) for a,b∈{0,1}. Simple analysis show that PRI vary between 0 (no agreement on any pair of samples in clusterings and {\mathcal{C}}^{\prime }) and 1 (when two clusterings are equal).

Moreover, one popular application of clustering algorithms is image segmentation. To evaluate the performances of semantic image segmentation, one widely used measure is the covering rate (CR) (Richardson and Green [1997]), by whcih a larger CR value indicates a better performance.

We aim at comparisons of the proposed Algorithm-2 with those typical algorithms investigated in (Shi et al. [2011]). For clarification, we summarize as follows: BYY-Jef and BYY-DNW: both come from table one and table six in (Shi et al. [2011]). MML-Jef: this was taken from table two in (Shi et al. [2011]), same as the one given in (Figueiredo and Jain [2002]). VB-DNW: this was taken from table six in (Shi et al. [2011]), same as the one given in (Bishop and Nasrabadi [2006]; Corduneanu and Bishop [2001]).

All algorithms are programmed in MATLAB R2010b on a 32-bit PC with 3.1 GHz Intel Core i5-2400 CPU and 4 GB memory.

All data sets and source codes used in this paper can be downloaded from the website http://www.cse.cuhk.edu.hk/~gychen/pBYY.

Empirical comparison

We start at three types of synthetic data sets illustrated in Figure 1. Each type of data set is processed 500 independent trails with random initializations. In the algorithm implementations, the mean vector of each Gaussian component is initialized randomly, and the initial mixing weight and initial covariance matrix of each Gaussian component are computed with help of the k mean algorithm.

Figure 1

Three synthetic data sets and their ground-truth clusters. For both (a) and (c), data sets of 60 samples are generated from a 2-dimensional 4-component GMM, with equal mixing weights of \frac{1}{4}; (b) 75 samples are generated from a 2-dimensional 5-component GMM, with equal mixing weight of \frac{1}{5}. (Each red curve indicates to a contour of equal probability density per component, and the read diamonds indicate the Gaussian means).

The comparisons of performance of each algorithm are shown in Table 1. We observe that pBYY significantly outperforms all the other algorithms almost in all the cases, without using any priori. The only exception occurs on the data set GMM-b, where BYY-DNW scored the best VI value though pBYY also got a value that is very close to the VI score. We also observe how the choice of an appropriate learning stepsize affects the performance of BYY-Jef and BYY-DNW. Closely related to the configurations of data sets, this choice is a difficult task. On the configuration type of GMM-b similar to the datasets studied in (Shi et al. [2011]), experiments reconfirm the statement that BYY outperforms its counterparts of VB and MML (Shi et al. [2011]). However, the statement seemly no longer holds for the configuration types of GMM-a and GMM-c, probably due to inappropriate learning stepsizes. Favorably, this statement has been reconfirmed by pBYY on the data sets of GMM-a and GMM-c with re-initialization period T _b being set as 5, namely, pBYY still significantly outperforms not only VB-DNW and MML-Jef but also BYY-Jef and BYY-DNW.

Table 1 Performance of each algorithm on three synthetic data sets after 500 trials, with the initial number of Gaussian components is set as k =20, where ^‘a’ indicates the best within its column

Table 2 presents a set of real-world data, where acidity, enzyme, and galaxy data sets come from (Richardson and Green [1997]). On these data sets, it is difficult to use CSR, PRI, and VI because the information about the correct clustering result is unavailable. Following (Bishop and Nasrabadi [2006]), we compare the performances of these algorithms on modeling the distributions of acidity, enzyme, and galaxy data sets visually. As demonstrated in Figure 2, BYY-Jef, BYY-DNW, and pBYY all obviously outperform VB-DNW and MML-Jef on the acidity and enzime, with pBYY performing best and MML-Jef outperforming. VB-DNW. On the galaxy data set, pBYY and BYY-DNW perform similarly and both outperform BYY-Jef, VB-DNW, and MML-Jef. In summary, these experiments confirm the previous findings obtained on synthetic data sets. In other words, pBYY outperforms not only VB and MML but also BYY-Jef and BYY-DNW.

Figure 2

Density fitting on 1D real-world data sets. From left to right: acidity, enzyme, and galaxy data sets; while from top to bottom: BYY-DNW, BYY-Jef, MML-Jef, VB-DNW, and pBYY algorithms. (Red curve indicates the overall density function and green curve the density function per component).

Table 2 Details of 1D real data sets

To further evaluate the performance of pBYY algorithm, we apply the proposed algorithm to unsupervised image segmentation on 100 testing images from Berkeley Segmentation Data Set (BSDS) www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html, where each image has five ground-truth segmentations hand-drawn by persons, as illustrated in Figure 3. For a clustering-based image segmentation algorithm, an important issue is how to get features as input vectors. In this paper, we use the features proposed by Varma and Zisserman ([2003]), which has been used with promising image segmentation results (Nikou et al. [2010]; Shi et al. [2011]; Zhu et al. [2013]). To concentrate on the performance of clustering algorithms, we do not conduct post-processing operations, such as region merging and graph cut, although they may further improve the segmentation results.

Figure 3

Ground-truth segmentation results hand-drawn by five different human objects on the image ♯ 296058.

We compare the performance of pBYY algorithm with several leading segmentation algorithms, including gPb-owt-ucm (Arbelaez et al. [2011]), multiscale graph decomposition (MN-Cut) (Cour et al. [2005]), and mean shift (Comaniciu and Meer [2002]). To make a fair comparison, these algorithms are implemented under the same prespecified configuration. For MML-Jef, VB-DNW, MN-Cut, and pBYY, the initial cluster number is set to be 20. For mean shift, the minimum region area is set at 5,000 pixels. For gPb-owt-ucm, we use the segmentation results posted by (Arbelaez et al. [2011]), and set the threshold to be 0.5. These settings are fixed throughout all the evaluations. To simplify the computation, we also ignore the re-initiation step in Algorithm 2 to accelerate the speed of pBYY algorithm.

Following the existing convention (Arbelaez et al. [2011]), we use PRI, VI, and CR to measure the comparison performance. The result of PRI, VI, and CR scores are shown in Table 3. Moreover, pairwise comparisons of pBYY with each competing algorithm are illustrated in Figure 4. By the PRI and CR measures, pBYY outperforms almost all the algorithms.

Figure 4

Pairwise comparison of segmentation algorithms on the BSDS500. The coordinates of the blue dots are the PRI, VI, and CR scores pBYY and its competing algorithms obtained per image. Red line represents the boundary of equal performance by the two algorithms, and the boxed digits indicate the number of images where one algorithm is better. For example, the last one shows MN-Cut outscores pBYY merely on 4/100 images.

Table 3 Performance scores on the BSDS

There is one exception at the center of the first row. The mean shift performs better than pBYY on 53 pieces of images according to PRI. Figure 5 shows the comparisons on four images randomly picked from the BSDS. Human judgement may clearly identify that the segmentations by pBYY look much bettter than the counterparts by mean shift. By the VI criterion, pBYY outperforms MML-Jef and VB-DNW but fails to win gPb-owt-ucm, MN-Cut, and mean shift. Observed from Figure 5, a human judgement may identify that the segmentations by pBYY are much better than the counterparts by gPb-owt-ucm, MN-Cut, and mean shift. Seemingly, the VI is more suitable to measure the clustering-based segmentations for a purpose of getting superpixels, while pBYY outperforms all the algorithms for semantic image segmentation but not be so for segmentations towards superpixels.

Figure 5

Comparisons on four images from the BSDS500.

Conclusions

On learning the Gaussian mixture model, the existing BYY learning algorithms are featured by either a gradient-based local search that needs an appropriate stepsize to be prespecified or a EM-like two-step alternation that does not request a learning stepsize but may lead to a unstable learning. The proposed pBYY still implements such a two-step alternation but removes the learning unreliability by an embedded projection, outperforming the existing BYY learning algorithms significantly. In the machine learning literature, Bayesian approach with appropriate priori provides a standard direction of developing learning algorithms for model selection, with VB and MML being two typical instances. In (Shi et al. [2011]), BYY outperforms MML and VB with the help of the same types of priories, but still fail to prevail with no priori. It has been shown in this paper that pBYY without any priori has outperformed MML-Jef, VB-DNW, BYY-Jef, and BYY-DNW, which confirms that the BYY best harmony learning provides a new perspective for automatic model selection even without a prior. Especially, this pBYY uses an easy computation to prevail the tedious computation required for using the DNW prior. More interestingly, the semantic image segmentation performance on the Berkeley Segmentation Data Set of 100 testing images have shown that pBYY outperforms not only MML-Jef, VB-DNW, BYY-Jef and BYY-DNW but also gPb-owt-ucm, MN-Cut, and mean shift.