Bootstrap estimate of Kullback-Leibler information

BootstrapEstimateofKullback-LeiblerInformationforModelSelectionByRiteiShibataTechnicalReportNo.424January1995DepartmentofStatisticsUniversityofCaliforniaBerkeley,CaliforniaBootstrapEstimateofKullback-LeiblerInformationforModelSelectionRiteiSHIBATADepartmentofMathematics,KeioUniversity3-14-1Hiyoshi,Kohoku,Yokohama,223,JapanAbstractEstimationofKullback-Leibleramountofinformationisacru-cialpartofderivingastatisticalmodelselectionprocedurewhichisbasedonlikelihoodprinciplelikeAIC.Todiscriminatenestedmod-els,wehavetoestimateituptotheorderofconstantwhiletheKullback-Leiblerinformationitselfisoftheorderofthenumberofobservations.AcorrectiontermemployedinAICisanexampletofulllthisrequirementbutitisasimplemindedbiascorrectiontothelogmaximumlikelihood.ThereforethereisnoassurancethatsuchabiascorrectionyieldsagoodestimateofKullback-Leiblerinformation.Inthispaperasanalternative,bootstraptypeestimationisconsid-ered.WewillrstshowthatbothbootstrapestimatesproposedbyEfron(1983,1986,1993)andCavanaughandShumway(1994)areatleastasymptoticallyequivalentandthereexistmanyotherequiva-lentbootstrapestimates.Wealsoshowthatallsuchmethodsareasymptoticallyequivalenttoanon-bootstrapmethod,knownasTIC(Takeuchi’sInformationCriterion)whichisageneralizationofAIC.KeyWordsandPhrases:Kullback-LeiblerInformation,Informa-tionCriterion,Bootstrap,BiasEstimationNowstayingatDepartmentofStatistics,U.C.Berkeley.11IntroductionEstimationofKullback-Leiblerinformationisakeytoderivingsocalledin-formationcriterionwhichisnowwidelyusedforselectingastatisticalmodel.Inparticular,Kullback-Leiblerinformationdenedasinthefollowing(1.1)isconsideredameasureofgoodnessoftofastatisticalmodel.Therefore,oneofstrategiesistoselectamodelsoastominimize(1.1).Throughoutthispaper,wemeanbyastatisticalmodelaparametricfamilyofdensitieswithrespecttoa-nitemeasureonndimensionalEuclideanspace,M=ff(x;)=Yifi(xi;);2g;wherex=(x1;x2;:::;xn)Tand=(1;2;:::;p)T.Weassume,ontheotherhand,thatthejointdistributionofindependentobservationsy=(y1;y2;:::;yn)TisGwhichhasadensityg(y)=Qigi(yi)withrespectto.Denoting^=^(y)themaximumlikelihoodestimateofunderamodelM,wedeneKullback-LeiblerinformationformodelMasIn(g();f(;^(y)))=Zg(x)logg(x)f(x;^(y))d(x)(1.1)=Zg(x)logg(x)d(x)Zg(x)logf(x;^(y))d(x):Sincethersttermontherighthandsideofthelastequationin(1.1)isindependentofanyparticularmodel,minimizingtheKullback-Leiblerinfor-2mation(1.1)isequivalenttomaximizingatargetvariable,T=T(y)=Zg(x)logf(x;^(y))d(x):(1.2)ByasimpleTaylorexpansion,wehaveanapproximationofT,T=Zg(x)logf(x;)d(x)12Q+op(1);(1.3)whereisapseudotrueparameter,thatis,thewhichminimizesI(g();f(;))ormaximizesZg(x)logf(x;)d(x):Herewehaveusedthenotations,Q=(^(y))T^J(y;)(^(y))and^J(y;)=@2@@Tlogf(y;):HoweverinpracticewehavetoestimateT,becausetheTdependsonanunknowng().ThelogmaximumlikelihoodisanaiveestimateofTanditcanbeagoodplatform.Itisapproximatedaslogf(y;^(y))=logf(y;)+12Q+op(1)(1.4)=Zg(x)logf(x;)d(x)3+flogf(y;)Zg(x)logf(x;)d(x)g+12Q+op(1):Theorderofmagnitudeoftherstthreetermsontherighthandsideofthelastequationin(1.4)areO(n),Op(pn)andOp(1),respectively.Therefore,onlythersttermissignicantasfarascompetitivemodelsarenotnestedeachother.However,ifmodelsM1M2arenestedandg()isamemberofM1,thenthepseudotrueparameterbecomesthesameforbothmodels,sothatonlythelastterm12Qin(1.4)remainssignicant.Infact,denotingthemaximumlikelihoodestimateofundereachmodelby^1and^2wecanwritethedierenceofthecorrespondinglogmaximumlikelihoodsaslogf(y;^1)logf(y;^2)=12(Q1Q2)+op(1):(1.5)Ontheotherhand,thedierenceofvaluesofthetargetvariableTiswrittenasT1T2=12(Q1Q2)+op(1):(1.6)Therefore,asimplemindedcorrectiontothelogmaximumlikelihoodiscor-rectingonlyasignicantpartofthebiasof(1.5)to(1.6),E(Q1Q2);whichisasymptoticallyequalto(p1p2),wherep1andp2arethenumberofparametersofmodelsM1andM2respectively.Thisyieldsabiascorrec-tionptothemaximumloglikelihoodlogf(y;^(y)).Ifthecorrectedlog4maximumlikelihoodismultipliedby-2forconvenience,Akaike’sinformationcriterionAIC=2logf(y;^(y))+2pfollows.Ofcourse,suchasimplemindedcorrectiondoesnotnecessarilyyieldagoodestimate.Alotofworkshavebeendonetondabettercorrection.Oneofsuchapproachesistoevaluatethebiasaspreciselyaspossible.InspiredbythepioneeringworkbySugiura(1978),HurvichandTsai(1989,1991,1993)derivedamoreprecisebiascorrection,p+(p+1)(p+2)np2;thanthepinAICfornormallinearmodels.Inpractice,suchacorrectionisquiteeective,particularlywhenthepiscloseton.Alsonon-asymptoticbiascorrectionisimportantinselectingadiscretemodellikebinomialormultinomialmodels,wherethedistributionisoftenskewedandnormalap-proximationworkswellonlyforquitelargenumberofobservations.Butinthispaperwedon’tgofurtherintothisproblem.TheauthorshowedanoptimalityoftheselectionsoastominimizeAICundertheassumptionthatthenumberofparametersofincreasesasthenumberofobservationsnincreases(Shibata(1980,1981)).Thisisforex-amplethecasewheng()isoutsideofanymodel.Thenmoreandmoreparametersareneededtogetcloserapproximationtog().Unde