PAMI Pedestrian Detection：An Evaluation of the Sta

PedestrianDetection:AnEvaluationoftheStateoftheArtPiotrDolla´r,ChristianWojek,BerntSchiele,andPietroPeronaAbstract—Pedestriandetectionisakeyproblemincomputervision,withseveralapplicationsthathavethepotentialtopositivelyimpactqualityoflife.Inrecentyears,thenumberofapproachestodetectingpedestriansinmonocularimageshasgrownsteadily.However,multipledatasetsandwidelyvaryingevaluationprotocolsareused,makingdirectcomparisonsdifficult.Toaddresstheseshortcomings,weperformanextensiveevaluationofthestateoftheartinaunifiedframework.Wemakethreeprimarycontributions:1)Weputtogetheralarge,well-annotated,andrealisticmonocularpedestriandetectiondatasetandstudythestatisticsofthesize,position,andocclusionpatternsofpedestriansinurbanscenes,2)weproposearefinedper-frameevaluationmethodologythatallowsustocarryoutprobingandinformativecomparisons,includingmeasuringperformanceinrelationtoscaleandocclusion,and3)weevaluatetheperformanceofsixteenpretrainedstate-of-the-artdetectorsacrosssixdatasets.Ourstudyallowsustoassessthestateoftheartandprovidesaframeworkforgaugingfutureefforts.Ourexperimentsshowthatdespitesignificantprogress,performancestillhasmuchroomforimprovement.Inparticular,detectionisdisappointingatlowresolutionsandforpartiallyoccludedpedestrians.IndexTerms—Pedestriandetection,objectdetection,benchmark,evaluation,dataset,CaltechPedestriandataset.Ç1INTRODUCTIONPEOPLEareamongthemostimportantcomponentsofamachine’senvironment,andendowingmachineswiththeabilitytointeractwithpeopleisoneofthemostinterestingandpotentiallyusefulchallengesformodernengineering.Detectingandtrackingpeopleisthusanimportantareaofresearch,andmachinevisionisboundtoplayakeyrole.Applicationsincluderobotics,entertain-ment,surveillance,carefortheelderlyanddisabled,andcontent-basedindexing.JustintheUS,nearly5,000ofthe35,000annualtrafficcrashfatalitiesinvolvepedestrians[1];hencetheconsiderableinterestinbuildingautomatedvisionsystemsfordetectingpedestrians[2].Whilethereismuchongoingresearchinmachinevisionapproachesfordetectingpedestrians,varyingevaluationprotocolsanduseofdifferentdatasetsmakesdirectcomparisonsdifficult.Basicquestionssuchas“Docurrentdetectorsworkwell?”“Whatisthebestap-proach?”“Whatarethemainfailuremodes?”and“Whatarethemostproductiveresearchdirections?”arenoteasilyanswered.Ourstudyaimstoaddressthesequestions.Wefocusonmethodsfordetectingpedestriansinindividualmonocularimages;foranoverviewofhowdetectorsareincorporatedintofullsystemswereferreadersto[2].Ourapproachisthree-pronged:Wecollect,annotate,andstudyalargedatasetofpedestrianimagescollectedfromavehiclenavigatinginurbantraffic;wedevelopinformativeevaluationmethodol-ogiesandpointoutpitfallsinpreviousexperimentalprocedures;finally,wecomparetheperformanceof16pre-trainedpedestriandetectorsonsixpubliclyavailabledatasets,includingourown.Ourstudyallowsustoassessthestateoftheartandsuggestsdirectionsforfutureresearch.Allresultsofthisstudy,andthedataandtoolsforreproducingthem,arepostedontheprojectwebsite:[3],weintroducedtheCaltechPedestrianDataSet,whichincludes350,000pedestrianboundingboxes(BB)labeledin250,000framesandremainsthelargestsuchdatasettodate.Occlusionsandtemporalcorrespondencesarealsoannotated.Usingtheextensivegroundtruth,weanalyzethestatisticsofpedestrianscale,occlusion,andlocationandhelpestablishconditionsunderwhichdetectionsystemsmustoperate.Evaluationmethodology.Weaimtoquantifyandrankdetectorperformanceinarealisticandunbiasedmanner.Tothiseffect,weexploreanumberofchoicesintheevaluationprotocolandtheireffectonreportedperformance.Overall,themethodologyhaschangedsubstantiallysince[3],resultinginamoreaccurateandinformativebenchmark.Evaluation.Weevaluate16representativestate-of-the-artpedestriandetectors(previouslyweevaluatedseven[3]).Ourgoalwastochoosediversedetectorsthatweremostpromisingintermsoforiginallyreportedperfor-mance.Weavoidretrainingormodifyingthedetectorstoensureeachmethodwasoptimizedbyitsauthors.Inadditiontooverallperformance,weexploredetectionratesundervaryinglevelsofscaleandocclusionandonclearlyvisiblepedestrians.Moreover,wemeasurelocalizationaccuracyandanalyzeruntime.Toincreasethescopeofouranalysis,wealsobenchmarkthe16detectorsusingaunifiedevaluationframeworkonIEEETRANSACTIONSONPATTERNANALYSISANDMACHINEINTELLIGENCE,VOL.34,NO.4,APRIL2012743.P.Dolla´randP.PeronaarewiththeDepartmentofElectricalEngineering,CaliforniaInstituteofTechnology,MC136-93,1200E.CaliforniaBlvd.,Pasadena,CA91125.E-mail:{pdollar,perona}@caltech.edu..C.WojekandB.SchielearewiththeMaxPlanckInstituteforInformatics,CampusE14,Saarbru¨cken66123,Germany.E-mail:{cwojek,schiele}@mpi-inf.mpg.de.Manuscriptreceived2Nov.2010;revised17June2011;accepted3July2011;publishedonline28July2011.RecommendedforacceptancebyG.Mori.Forinformationonobtainingreprintsofthisarticle,pleasesende-mailto:[e-mail],andreferenceIEEECSLogNumberTPAMI-2010-11-0837.DigitalObjectIdentifierno.10.1109/TPAMI.2011.155.0162-8828/12/$31.002012IEEEPublishedbytheIEEE