Face Description with Local Binary Patterns Applic

1FaceDescriptionwithLocalBinaryPatterns:ApplicationtoFaceRecognitionTimoAhonen,StudentMember,IEEE,AbdenourHadid,andMattiPietik¨ainen,SeniorMember,IEEEAbstractThispaperpresentsanovelandefficientfacialimagerepresentationbasedonlocalbinarypattern(LBP)texturefeatures.ThefaceimageisdividedintoseveralregionsfromwhichtheLBPfeaturedistributionsareextractedandconcatenatedintoanenhancedfeaturevectortobeusedasafacedescriptor.Theperformanceoftheproposedmethodisassessedinthefacerecognitionproblemunderdifferentchallenges.Otherapplicationsandseveralextensionsarealsodiscussed.IndexTermsFacialimagerepresentation,localbinarypattern,component-basedfacerecogni-tion,texturefeatures,facemisalignmentI.INTRODUCTIONAutomaticfaceanalysiswhichincludes,e.g.,facedetection,facerecognitionandfacialexpressionrecognitionhasbecomeaveryactivetopicincomputervisionresearch[1].Akeyissueinfaceanalysisisfindingefficientdescriptorsforfaceappearance.DifferentholisticmethodssuchasPrincipalComponentAnalysis(PCA)[2],LinearDiscriminantAnalysis(LDA)[3]andthemorerecent2-DPCA[4]havebeenstudiedwidelybutlatelyalsolocaldescriptorsT.Ahonen,A.Hadid,andM.Pietik¨ainenarewiththeMachineVisionGroup,InfotechOulu,DepartmentofElectricalandInformationEngineering,UniversityofOulu,POBox4500,FIN-90014,Finland.E-mail:ftahonen,hadid,[e-mail].fi.5thJune2006DRAFT2havegainedattentionduetotheirrobustnesstochallengessuchasposeandilluminationchanges.Thispaperpresentsanoveldescriptorbasedonlocalbinarypatterntexturefeaturesextractedfromlocalfacialregions.OneofthefirstfacedescriptorsbasedoninformationextractedfromlocalregionsistheeigenfeaturesmethodproposedbyPentlandetal.[5].ThisisahybridapproachinwhichthefeaturesareobtainedbyperformingPCAtolocalfaceregionsindependently.InLocalFeatureAnalysis[6],kernelsoflocalspatialsupportareusedtoextractinformationaboutlocalfacialcomponents.ElasticBunchGraphMatching(EBGM)[7]describesfacesusingGaborfilterresponsesincertainfaciallandmarksandagraphdescribingthespatialrelationsoftheselandmarks.ThevalidityofthecomponentbasedapproachisalsoattestedbythestudyconductedbyHeiseleetal.inwhichacomponent-basedfacerecognitionsystemclearlyoutperformedglobalapproachesonatestdatabasecontainingfacesrotatedindepth[8].Usinglocalphotometricfeatures[9]forobjectrecognitioninthemoregeneralcontexthasbecomeawidelyacceptedapproach.Inthatsettingthetypicalapproachistodetectinterestpointsorinterestregionsinimages,performnormalizationwithrespecttoaffinetransformationsanddescribethenormalizedinterestregionsusinglocaldescriptors.Thisbag-of-keypointsapproachisnotsuitedforfacedescriptionassuchsinceitdoesnotretaininformationonthespatialsettingofthedetectedlocalregionsbutitdoesbearcertainsimilaritiestolocalfeaturebasedfacedescription.Findinggooddescriptorsfortheappearanceoflocalfacialregionsisanopenissue.Ideally,thesedescriptorsshouldbeeasytocomputeandhavehighextra-classvariance(i.e.,betweendifferentpersonsinthecaseoffacerecognition)andlowintra-classvariance,whichmeansthatthedescriptorshouldberobustwithrespecttoagingofthesubjects,alternatingilluminationandotherfactors.Thetextureanalysiscommunityhasdevelopedavarietyofdifferentdescriptorsfortheappearanceofimagepatches.However,facerecognitionproblemhasnotbeenassociatedtothatprogressintextureanalysisfieldasithasnotbeeninvestigatedfromsuchpointofview.Recently,weinvestigatedtherepresentationoffaceimagesbymeansoflocalbinarypatternfeatures,yieldinginoutstandingresultsthatwerepublishedintheECCV2004conference[10].Afterthis,severalresearchgroupshaveadoptedourapproach.Inthispaper,weprovideamoredetailedanalysisoftheproposedrepresentation,presentadditionalresultsanddiscussfurther5thJune2006DRAFT311011001Binary:11001011Decimal:203Threshold=859921545486121357Fig.1.ThebasicLBPoperator.extensions.II.LBPBASEDFACEDESCRIPTIONTheLBPoperator[11]isoneofthebestperformingtexturedescriptorsandithasbeenwidelyusedinvariousapplications.Ithasproventobehighlydiscriminativeanditskeyadvantages,namelyitsinvariancetomonotonicgraylevelchangesandcomputationalefficiency,makeitsuitablefordemandingimageanalysistasks.ForabibliographyofLBP-relatedresearch,seefi/research/imag/texture/.TheideaofusingLBPforfacedescriptionismotivatedbythefactthatfacescanbeseenasacompositionofmicro-patternswhicharewelldescribedbysuchoperator.A.LocalbinarypatternsTheLBPoperatorwasoriginallydesignedfortexturedescription.Theoperatorassignsalabeltoeverypixelofanimagebythresholdingthe3x3-neighborhoodofeachpixelwiththecenterpixelvalueandconsideringtheresultasabinarynumber.Thenthehistogramofthelabelscanbeusedasatexturedescriptor.SeeFigure1foranillustrationofthebasicLBPoperator.Tobeabletodealwithtexturesatdifferentscales,theLBPoperatorwaslaterextendedtouseneighborhoodsofdifferentsizes[12].Definingthelocalneighborhoodasasetofsamplingpointsevenlyspacedonacirclecenteredatthepixeltobelabeledallowsanyradiusandnumberofsamplingpoints.Bilinearinterpolationisusedwhenasamplingpointdoesnotfallinthecenterofapixel.Inthefollowing,thenotation(P;R)willbeusedforpixelneighborhoodswhichmeansPsamplingpointsonacirc