Variational Bayesian multinomial probit regression

Girolami,M.andRogers,S.(2006)VariationalBayesianmultinomialprobitregressionwithGaussianprocesspriors.NeuralComputation18(8):pp.1790-1817.:[e-mail]cienceUniversityofGlasgowTechnicalReport:TR-2005-205fgirolami,[e-mail]9,20051AbstractItiswellknowninthestatisticsliteraturethataugmentingbinaryandpolychotomousresponsemodelswithGaussianlatentvariablesen-ablesexactBayesiananalysisviaGibbssamplingfromtheparameterposterior.Byadoptingsuchadataaugmentationstrategy,dispensingwithpriorsoverregressioncoecientsinfavourofGaussianProcess(GP)priorsoverfunctions,andemployingvariationalapproximationstothefullposteriorweobtainecientcomputationalmethodsforGaussianProcessclassicationinthemulti-classsetting1.Themodelaugmentationwithadditionallatentvariablesensuresfullaposteri-oriclasscouplingwhilstretainingthesimpleaprioriindependentGPcovariancestructurefromwhichsparseapproximations,suchasmulti-classInformativeVectorMachines(IVM),emergeinaverynat-uralandstraightforwardmanner.ThisisthersttimethatafullyVariationalBayesiantreatmentformulti-classGPclassicationhasbeendevelopedwithouthavingtoresorttoadditionalexplicitapprox-imationstothenon-Gaussianlikelihoodterm.EmpiricalcomparisonswithexactanalysisviaMCMCandLaplaceapproximationsillustratetheutilityofthevariationalapproximationasacomputationallyeco-nomicalternativetofullMCMCanditisshowntobemoreaccuratethantheLaplaceapproximation.1IntroductionIn(AlbertandChib,1993)itwasrstshownthatbyaugmentingbinaryandmultinomialprobitregressionmodelswithasetofcontinuouslatentvariablesyk,correspondingtothek'thresponsevaluewhereyk=mk+,N(0;1)andmk=Pjkjxj,anexactBayesiananalysiscanbeper-formedbyGibbssamplingfromtheparameterposterior.Asanexam-pleconsiderbinaryprobitregressionontargetvariablestn2f0;1g,theprobitlikelihoodforthenthdatasampletakingunitvalue(tn=1)isP(tn=1jxn;)=(Txn),whereisthestandardisedNormalCumula-tiveDistributionFunction(CDF).Now,thiscanbeobtainedbythefollow-ingmarginalisationRP(tn=1;ynjxn;)dyn=RP(tn=1jyn)p(ynjxn;)dynandasbydenitionP(tn=1jyn)=(yn0)thenweseethattherequiredmarginalissimplythenormalizingconstantofalefttruncatedunivariate1Matlabcodetoallowreplicationofthereportedresultsisavailableat(tn=1jxn;)=R(yn0)Nyn(Txn;1)dyn=(Txn).ThekeyobservationhereisthatworkingwiththejointdistributionP(tn=1;ynjxn;)=(yn0)Nyn(Txn;1)providesastraightforwardmeansofGibbssamplingfromtheparameterposteriorwhichwouldnotbethecaseifthemarginalterm,(Txn),wasemployedindeningthejointdistributionoverdataandparameters.Thisdataaugmentationstrategycanbeadoptedindevelopingecientmethodstoobtainbinaryandmulti-classGaussianProcess(GP)(WilliamsandRasmussen,1996)classiersaswillbepresentedinthispaper.Withtheexceptionof(Neal,1998),whereafullMarkovChainMonteCarlo(MCMC)treatmenttoGPbasedclassicationisprovided,allotherapproacheshavefocussedonmethodstoapproximatetheproblematicformoftheposte-rior2whichallowanalyticmarginalisationtoproceed.Laplaceapproxima-tionstotheposteriorweredevelopedin(WilliamsandBarber,1998)whilstlower&upperboundquadraticlikelihoodapproximationswereconsideredin(Gibbs,2000).Variationalapproximationsforbinaryclassicationweredevelopedin(Seeger,2000)wherealogitlikelihoodwasconsideredandmeaneldapproximationswereappliedtoprobitlikelihoodtermsin(OpperandWinther,2000),(Csatoetal,2000)respectively.Additionally,incremen-tal(Quinonero-CandelaandWinther,2003)orsparseapproximationsbasedonAssumedDensityFiltering(ADF)(CsatoandOpper,2002),InformativeVectorMachines(IVM)(Lawrence,etal2003)andExpectationPropagation(EP)(Minka,2001;Kim,2005)havebeenproposed.Withtheexceptionsof(WilliamsandBarber,1998;Gibbs,2000;SeegerandJordan,2004;Kim,2005)thefocusofmostrecentworkhaslargelybeenonthebinaryGPclas-sicationproblem.In(SeegerandJordan,2004)amulti-classgeneralisationoftheIVMisdevelopedwheretheauthorsemployamultinomial-logitsoft-maxlikelihood.However,considerablerepresentationaleortisrequiredtoensurethatthescalingofcomputationandstoragerequiredoftheproposedmethodmatchesthatoftheoriginalIVMwithlinearscalinginthenumberofclasses.Incontrast,byadoptingtheprobabilisticrepresentationof(Al-bertandChib,1993)wewillseethatGPbasedK-classclassicationandecientsparseapproximations(IVMgeneralisationswithscalinglinearinthenumberofclasses)canberealisedbyoptimisingastrictlower-boundofthemarginallikelihoodofamultinomialprobitregressionmodelwhich2Thelikelihoodisnonlinearintheparametersduetoeitherthelogisticorprobitlinkfunctionsrequiredintheclassicationsetting3requiresthesolutionofKcomputationallyindependentGPregressionprob-lemswhilststilloperatingjointly(statistically)onthedata.WewillalsoshowthattheaccuracyofthisapproximationiscomparabletothatobtainedviaMCMC.Thefollowingsectio