Face Recognition from Still Images and Videos

1FaceRecognitionfromStillImagesandVideosShaohuaKevinZhouandRamaChellappaCenterforAutomationResearch(CfAR)andDepartmentofElectricalandComputerEngineeringUniversityofMaryland,CollegePark,MD20742Email:{shaohua,rama}@cfar.umd.eduI.INTRODUCTIONInmostsituations,identifyinghumansusingfacesisaneffortlesstaskforhumans.Isthistrueforcomputers?Thisveryquestiondefinesthefieldofautomaticfacerecognition[7],[31],[62],oneofthemostactiveresearchareasincomputervision,patternrecognition,andimageunderstanding.Overthepastdecade,theproblemoffacerecognitionhasattractedsubstantialattentionfromvariousdisciplinesandhaswitnessedaskyrocketinggrowthoftheliterature.Below,wemainlyemphasizesomekeyperspectivesofthefacerecognitionproblem.A.BiometricperspectiveFaceisabiometric.Asaconsequence,facerecognitionfindswideapplicationsinauthentication,security,andsoon.OnerecentapplicationistheUS-VISITsystembytheDepartmentofHomelandSecurity(DHS),collectingforeignpassengers’fingerprintsandfaceimages.Biometricsignaturesofapersoncharacterizethephysiologicalorbehavioralcharacteristics.Physiologicalbio-metricsareinnateornaturallyoccuring,whilebehavioralbiometricsarisefrommannerismsortraitsthatarelearnedoracquired.TableIlistscommonlyusedbiometrics.Biometrictechnologiesprovidethefoundationforanextensivearrayofhighlysecureidentificationandpersonalverificationsolutions.Comparedtoconventionalidentificationandverificationmethodsbasedonpersonalidentificationnumbers(PINs)orpasswords,biometrictechnologiesoffermanyadvantages.First,biometricsareindividualizedtraitswhilepasswordsmaybeusedorstolenbysomeoneotherthantheauthorizeduser.Also,biometricisveryconvenientsincethereisnothingtocarryorremember.Inaddition,biometrictechnologiesarebecomingmoreaccurateandlessexpensive.AmongallbiometricslistedinTableI,thefaceisaveryuniqueonebecauseitistheonlybiometricbelongingtobothphysiologicalandbehavioralcategories.WhilethephysiologicalpartofthefacehasbeenwidelyexploitedPartiallysupportedbyNSFITRGrant03-25119.ZhouisnowwithIntegratedDataSystemsDepartment,SiemensCorporateResearch,Princeton,[e-mail]5,2004DRAFT2TypeExamplesPhysiologicalbiometricsDNA,face,fingerprint,handgeometry,iris,pulse,retinal,andbodyodorBehavioralbiometricsFace,gait,handwriting,signature,andvoiceTABLEIAlistofphysiologicalandbehavioralbiometrics.forfacerecognition,thebehavioralparthasnotyetbeenfullyinvestigated.Inaddition,asreportedin[19],[43],faceenjoysmanyadvantagesoverotherbiometricsbecauseitisanatural,non-intrusive,andeasy-to-usebiometric.Forexample[19],amongsixbiometricsofface,finger,hand,voice,eye,andsignature,facebiometricranksthefirstinthecompatibilityevaluationofamachinereadabletraveldocument(MRTD)systemintermsofsixcriteria:enrollment,renewal,machine-assistedidentityverificationrequirements,redundancy,publicperception,andstoragerequirementsandperformance.Probablythemostimportantfeatureofacquiringthefacebiometricsignatureisthatnocooperationisrequiredduringdataacquisition.Besidesapplicationsrelatedtoidentificationandverificationsuchasaccesscontrol,lawenforcement,IDandlicensing,surveillance,etc.,facerecognitionisalsousefulinhuman-computerinteraction,virtualreality,databaseretrieval,multimedia,computerentertainment,etc.See[31],[62]forrecentsummariesonfacerecognitionappli-cations.B.ExperimentalperspectiveFacerecognitionmainlyinvolvesthefollowingthreetasks[46]:•Verification.Therecognitionsystemdeterminesifthequeryfaceimageandtheclaimedidentitymatch.•Identification.Therecognitionsystemdeterminestheidentityofthequeryfaceimage.•Watchlist.Therecognitionsystemfirstdeterminesiftheidentityofthequeryfaceimageisinthewatchlistand,ifyes,thenidentifytheindividual.Figure1illustratestheabovethreetasksandcorrespondingmetricsusedforevaluation.Amongthreetasks,thewatchlisttaskisthemostdifficultone.Thischapterfocusesonlyontheidentificationtask.WeintroduceafacerecognitiontestprotocolFERET[45]widelyfollowedinthefacerecognitionliterature.FERETstandsfor‘facialrecognitiontechnology’.FERETassumestheavailabilityofthefollowingthreesets,namelyatrainingset,agalleryset,andaprobeset.Thetrainingsetisprovidedfortherecognitionalgorithmtolearnthefeaturesthatarecapableofcharacterizingthewholehumanfacespace.Thegalleryandprobesetsareusedinthetestingstage.Thegallerysetcontainsimageswithknownidentitiesandtheprobesetwithunknownidentities.Thealgorithmassociatesdescriptivefeatureswiththeimagesinthegalleryandprobesetsanddeterminestheidentitiesoftheprobeimagesbycomparingtheirassociatedfeatureswithfeaturesassociatedwithgalleryimages.November5,2004DRAFT3Fig.1.Threefacerecognitiontasks:verification,identification,watchlist(courtesytoP.J.Phillips.)C.TheoreticalperspectiveFacerecognitionisbynatureaninterdisciplinaryresearcharea,involvingresearchersfrompatternrecognition,computervisionandgraphics,imageprocessing/understanding,statisticalcomputingandmachinelearning.Inaddition,automaticfacerecognitionalgorithms/systemsareoftenguidedbythepsychophysicsandneuralstudiesonhowhumansperceivefaces.Agoodsummaryofresearchonfaceperceptionisprese