Approximate Inference for Robust Gaussian Process

Max–Planck–Institutf¨urbiologischeKybernetikMaxPlanckInstituteforBiologicalCyberneticsTechnicalReportNo.136ApproximateInferenceforRobustGaussianProcessRegressionMalteKuss1,TobiasPfingsten1,2,LehelCsat´o1,CarlE.Rasmussen1March10,20051DepartmentSch¨olkopf{kuss,tpfingst,csatol,carl}@tuebingen.mpg.de2RobertBoschGmbH,CorporateSectorResearchandAdvanceEngineeringThisreportisavailableinPDF–formatviaanonymousftpat:fingsten,LehelCsat´o,CarlE.RasmussenAbstract.Gaussianprocess(GP)priorshavebeensuccessfullyusedinnon-parametricBayesianre-gressionandclassificationmodels.InferencecanbeperformedanalyticallyonlyfortheregressionmodelwithGaussiannoise.Forallotherlikelihoodmodelsinferenceisintractableandvariousapproximationtechniqueshavebeenproposed.Inrecentyearsexpectation-propagation(EP)hasbeendevelopedasageneralmethodforapproximateinference.Thisarticleprovidesageneralsummaryofhowexpectation-propagationcanbeusedforapproximateinferenceinGaussianprocessmodels.FurthermorewepresentacasestudydescribingitsimplementationforanewrobustvariantofGaussianprocessregression.TogainfurtherinsightsintothequalityoftheEPapproximationwepresentexperimentsinwhichwecom-paretoresultsobtainedbyMarkovchainMonteCarlo(MCMC)sampling.1Introduction–Robustness&BayesianRegressionTosolveareal-worldregressionproblemtheanalystshouldcarefullyscreenthedataanduseallpriorinformationathandinordertochooseanappropriateregressionmodel.Themodelisselectedsoastoapproximatethebeliefsaboutthedatageneratingprocess.Amismatchseemsunavoidableinpractice.Robustregressionmethodscanbeunderstoodasattemptstolimitundesireddistractionsanddistortionsthatresultfromthismismatch.Robustregressionisoftenassociatedwiththenotionofoutliers,whichreferstoobservationsthatareinsomesensestructurallyconspicuous.Oftenthepresenceofsuchoutliersisattributedtoobservationalerrors,e.g.dataprocessingerrorsorfailuresofmeasuringinstruments.Commonlyastatisticalmodeliscalledrobustifitleadstoconclusionswhichareinsensitivetotheoccurrenceofsuchoutlierobservations.Notethatthisimpliesthatanobservationcanonlybecalledanoutlierrelativetoagivenmodel.AsJaynes(2003,ch.21)phrasesit:“Oneseeksdataanalysismethodsthatarerobust,whichmeansinsensitivetotheexactsamplingdistributionoferrors,asitisoftenstated,insensitivetothemodel,orare,resistant,meaningthatlargeerrorsinsmallproportionofthedatadonotgreatlyaffecttheconclusions.”TheBayesiananswertorobustregression,i.e.handlingoutliers,resultsautomaticallyfromthecom-monstatementthatamodelshouldbechosensoastoreflectalltheanalyst’sbeliefsanduncertainties.SoaBayesianregressionmodelcanbeconsideredrobustifitexplicitlyaccountsforthepotentialexis-tenceofoutliers.Therefore,unlesstheanalysthasabsolutelynodoubtthatthemodelhehasaccountsforallpossibleobservations—inotherwords,unlessheiscertainthattherearenooutliersrelativetothatmodel—heshouldadjustthemodeltoaccountexplicitlyforthepotentialoccurrenceofoutliers.Aconvenientwaytoreflectthisbeliefisamixturemodel.Jaynes(2003,ch.21)callsita“two-modelmodel”beingamixtureofamodelwhichaccountsfortheregularobservationsandasecondmodelforexplainingoutliers.The“two-modelmodel”willbethelineofthoughtintheremainderofthispaper.Beforewegoon,webrieflydescribeinferenceintheframeworkofnon-parametricBayesianregres-sion.ByinferencewerefertotheprocessofupdatingourbeliefsaccordingtoBayes’rule,i.e.comput-ingtheposteriorfromlikelihoodandprior,integratingtheinformationcontainedinobserveddata.Inregressionanalysistheobjectiveistomakeinferencerelatedtoalatentreal-valuedfunctionf(x)wherex∈RD.Thenon-parametricapproachistoputapriorp0(f|θ1)directlyonthespaceoffunctionsandto1doinferenceonf.ThesimplestandmostcommonprioroverfunctionsisaGaussianprocess,describedinSection2.Inferenceaboutfisbasedonobservedsamplesy(x)=f(x)+εwhicharecorruptedbyadditivenoise.Weassumethenoisetermεtobeindependentandidenticallydistributed(iid.),leadingtothejointlikelihoodp(y|f,X,θ2)=NYn=1p(yn|fn,xn,θ2)(1)wherey=[y1,...,yN]denotestheobservedoutputs,X=[x1,...,xN]arethecorrespondingin-puts,andfn=f(xn)arethelatentfunctionvalues.Weintroduceasetofparametersθ2toparameterisethelikelihoodp(y|f,x,θ2).1Fornon-parametricBayesianmodelstheposterioroverthefiscomputedaccordingtoBayes’ruleppost(f|X,y,θ1,θ2)=p(y|f,X,θ2)p0(f|θ1)p(y|X,θ1,θ2)(2)wherefisarandomfunctionandtheparametersθ1andθ2areconsideredfixed.Thedenominatoristheevidence,ormarginallikelihoodp(D|θ1,θ2)=p(y|X,θ1,θ2),whichisthenormalisingconstantoftheproductoflikelihoodandprior.HereD={X,y}denotestheobserveddataandweusetheslightabuseofnotationp(D|·)tomeanp(y|X,·).Wenowdescribehowwecanconstructamixturelikelihood—atwo-modelmodel—inordertoobtainarobustBayesianregressionmodelwrt.outliersiny.Letpr(yn|fn,θ2)denoteanoisemodelwhichdescribesourbeliefsaboutregularobservations,likethetypicalerrorofameasuringinstrument.Assumewecannotdenythepotentialexistenceofoutlie