Strong consistency of MLE for finite mixtures of l

arXiv:math/0605148v1[math.ST]5May2006StrongconsistencyofMLEforfinitemixturesoflocation-scaledistributionswhenthescaleparametersareexponentiallysmallKentaroTanaka∗andAkimichiTakemura†AbstractInafinitemixtureoflocation-scaledistributionsmaximumlikelihoodestimatordoesnotexistbecauseoftheunboundednessofthelikelihoodfunctionwhenthescaleparameterofsomemixturecomponentapproacheszero.Inordertostudythestrongconsistencyofmaximumlikelihoodestimator,weconsiderthecasethatthescaleparametersofthecomponentdistributionsarerestrictedfrombelowbycn,where{cn}isasequenceofpositiverealnumberswhichtendtozeroasthesamplesizenincreases.Weprovethatundermildregularityconditionsmaximumlikelihoodestimatorisstronglyconsistentifthescaleparametersarerestrictedfrombelowbycn=exp(−nd),0d1.Keywordsandphrases:Mixturedistribution,maximumlikelihoodestimator,consis-tency.1IntroductionInsomefinitemixturedistributionsmaximumlikelihoodestimator(MLE)doesnotexist.Letusconsiderthefollowingexample.DenoteanormalmixturedistributionwithMcomponentsandparameterθ=(α1,μ1,σ21,...,αM,μM,σ2M)byf(x;θ)=MXm=1αmφm(x;μm,σ2m),whereαm(m=1,...,M)arenonnegativerealnumbersthatsumtooneandφm(x;μm,σ2m)arenormaldensities.Letx1,...,xndenotearandomsampleofsizen≥2fromthedensityf(x;θ0).Inviewoftheidentifiabilityproblemofmixturedistributionsdiscussedbelow,∗DepartmentofIndustrialEngineeringandManagement,TokyoInstituteofTechnology,2-12-1O-okayama,Meguro-ku,Tokyo152-8550JAPAN,E-mail:[e-mail]†DepartmentofMathematicalInformatics,GraduateSchoolofInformationScienceandTechnol-ogy,UniversityofTokyo,7-3-1Hongo,Bunkyo-ku,Tokyo113-8656,JAPAN,E-mail:[e-mail]1hereθ0isaparametervaluedesignatingthetruedistribution.Howeverforsimplicitywejustsayθ0isthetrueparameterfromnowon.TheloglikelihoodfunctionisnXi=1logf(xi;θ)=nXi=1log(MXm=1αmφm(xi;μm,σ2m)).Ifwesetμ1=x1,thenthelikelihoodtendstoinfinityasσ21→0.ThusMLEdoesnotexist.Butwhenwerestrictσm≥c(m=1,...,M)bysomepositiverealconstantc,wecanavoidthedivergenceofthelikelihood.Furthermore,inthissituation,itcanbeshownthatMLEisstronglyconsistentifthetrueparameterθ0isintherestrictedparameterspace.Ontheotherhand,thesmallerσ21is,thelesscontributionφ1(x;μ1=x1,σ21)makestothelikelihoodatotherobservationsx2,...,xn.Thereforeaninterestingquestionhereiswhetherwecandecreasetheboundc=cntozerowiththesamplesizenandyetguaranteethestrongconsistencyofMLE.Ifthisispossible,thefurtherquestionishowfastcncandecreasetozero.Thisquestionissimilartothe(sofaropen)problemstatedinHathaway(1985),whichtreatsmixturesofnormaldistributionswithconstraintsimposedontheratiosofvarianceswhileourrestrictionisimposedonvariancesthemselves.Seealsoadiscussioninsection3.8.1ofMcLachlanandPeel(2000).Intheaboveexample,thenormalityofthecomponentdistributionsisnotessentialandthesamedifficultyexistsforfinitemixturesofgenerallocation-scaledistributionssuchasmixturesofuniformdistributions.Furthermoreinthispaperweallowthateachcomponentbelongstodifferentlocation-scalefamilies.Letσm(m=1,...,M)denotethescaleparametersofthecomponentdistributionsandconsidertherestrictionσm≥cn(m=1,...,M).Thenaquestionofinteresthereiswhetherwecandecreasetheboundcntozero.Forthecaseofmixtureofuniformdistributions,inTanakaandTakemura(2005)weprovedthatMLEisstronglyconsistentifcn=exp(−nd),0d1.Heredcanbearbitrarilycloseto1butfixed.Inthispaper,weprovethatthesameresultholdsforgeneralfinitemixturesoflocation-scaledistributionsunderverymildregularityconditions(assumptions1–4below).WeemploythesamelineofproofasinTanakaandTakemura(2005),buttheproofforthegeneralfinitemixtureismuchmoredifficult.Asdiscussedinsection5thenormaldensitysatisfiestheregularityconditionsandourresultimpliesthatMLEisstronglyconsistentforthefinitenormalmixtureifσm≥cn=exp(−nd),0d1,m=1,...,M.Ourframeworkiscloselyrelatedtothemethodofsieve(Grenander(1981)).Inthesievemethodanobjectivefunctionismaximizedoveraconstrainedsubspaceofparameterspaceandthenthissubspaceisexpandedtothewholeparameterspaceasthesamplesizeincreases.SomeapplicationsandconsistencyresultsforthemethodaregiveninGemanandHwang(1982).MLEbasedonasieveiscalledasieveMLE.TheconvergenceratesofsieveMLEforGaussianmixtureproblemsarestudiedinGenoveseandWasserman2(2000)andGhosalandvanderVaart(2001)andtheirideasareveryinteresting.TheyobtaintheconvergenceratesbyboundingtheHellingerbracketingentropyofsubsetsofthefunctionspaceandassumethatthecorrespondingsubsetsoftheparameterspacearecompactsothattheirbracketingentropydoesnotdiverge.InthecaseofsieveMLE,theapproximatingsubspacesareusuallytakentobecompact,whereaswetreatasequenceofnon-compactsubsetsoftheparameterspaceexpandingtothewholeparameterspaceasthesamplesizeincreases.ThereforeresultsonsieveMLEarenotdirectlyapplicableinourframework.Theorganizationofthepaperisasfollows.Insection2wesummarizesomeprelim-inarydescriptions.Insection3westateourmainresultsintheorems1and2.Section4isdevotedtotheproofoftheoremsandlemmas.Finallyinsection5wegivesomediscussions.2Preliminariesonstrongconsistencyandidentifia-bilityofmixturedistributionsAmixtureofMdensitieswithparameterθ=(α1,μ1,σ1,