Appearance-based person re-identification in camer

ORIGINALRESEARCHAppearance-basedpersonreidentificationincameranetworks:problemoverviewandcurrentapproachesGianfrancoDoretto•ThomasSebastian•PeterTu•JensRittscherReceived:30January2010/Accepted:4October2010/Publishedonline:14January2011Springer-Verlag2011AbstractRecentadvancesinvisualtrackingmethodsallowfollowingagivenobjectorindividualinpresenceofsignificantclutterorpartialocclusionsinasingleorasetofoverlappingcameraviews.Thequestionofwhenpersondetectionsindifferentviewsoratdifferenttimeinstantscanbelinkedtothesameindividualisoffundamentalimpor-tancetothevideoanalysisinlarge-scalenetworkofcam-eras.Thisisthepersonreidentificationproblem.Thepaperfocusesonalgorithmsthatusetheoverallappearanceofanindividualasopposedtopassivebiometricssuchasfaceandgait.Methodsthateffectivelyaddressthechallengesasso-ciatedwithchangesinillumination,pose,andclothingappearancevariationarediscussed.Morespecifically,thedevelopmentofasetofmodelsthatcapturetheoverallappearanceofanindividualandcaneffectivelybeusedforinformationretrievalarereviewed.Someofthemprovideaholisticdescriptionofaperson,andsomeothersrequireanintermediatestepwherespecificbodypartsneedtobeidentified.Somearedesignedtoextractappearancefeaturesovertime,andsomeotherscanoperatereliablyalsoonsingleimages.Thepaperdiscussesalgorithmsforspeedingupthecomputationofsignatures.Inparticularitdescribesveryfastproceduresforcomputingco-occurrencematricesbyleveragingageneralizationoftheintegralrepresentationofimages.Thealgorithmsaredeployedandtestedinacameranetworkcomprisingofthreecameraswithnon-overlappingfieldofviews,whereamulti-cameramulti-targettrackerlinksthetracksindifferentcamerasbyreidentifyingthesamepeopleappearingindifferentviews.KeywordsRe-identificationSurveillanceTrackingAppearancematchingIntegralimageCo-occurrenceIntegralrepresentation1IntroductionInstallationsofcameranetworksnowadaysarewidespreadinvariousdomainsthatrangefromhomesurveillanceapplications,tosmallbusinessandlargeretailapplications,tofacilityaccess,sportsvenue,mass-transit,andenviron-mentmonitoring,toopenborderssurveillance,justtomentionafew.Thisraisestheneedforautomatedmethodsabletoextract,andaccesshigh-levelsemanticinformationcarriedbytheextremelyhighvolumeofrecordedvideodata.Inmanysurveillancetasksknowingwhetherinagivenscene,atagivenpositionandtime,agivenpersonwaspresentisofparamountimportance,andjustifiestheeffortsdevotedtothedevelopmentofsystemsthatcanperformdetectionandtrackingofpeople(Tuetal.2007).Intrinsictotheideaoftrackingapersonistheconceptofmaintaininghis/heridentity.Infact,trackingfromonevideoframetothenextmeansbeingabletotellthatthepersonthatispointedtoisthesamethatwaspointedinthepreviousframe.Whenonlyonevideofeedisprocessed,theseidentitymanagementissuesareaddressedbyso-calleddataassociationtechniques,suchasgeneralizednearestneighbor(BlackmanandPopoli1999),jointprobabilisticG.Doretto(&)WestVirginiaUniversity,P.O.Box6901,Morgantown,WV26506,USAe-mail:[e-mail](&)P.TuJ.RittscherGEGlobalResearch,Niskayuna,NY12309,USAe-mail:[e-mail]-mail:[e-mail]-mail:[e-mail]123JAmbientIntellHumanComput(2011)2:127–151DOI10.1007/s12652-010-0034-ydataassociationfiltering(RasmussenandHager1998),multiplehypothesistracking(CoxandHingorani1994),orBayesianmulti-targettracking(IsardandMacCormick2001).Growingfromonevideofeedtomultiplevideofeeds,recordedsimultaneouslybymultiplecameras(i.e.acameranetwork)withoverlappingfieldofviews,farthercompli-catesthetrackingproblem.Here,besidespointingtothesamepersonfromoneframetothenext,trackingmeansbeingabletopointtothesamepersonfromonecameratothenext,whenhe/shedisappearsfromtheformerandappearsinthelatter.Mostofcurrentapproachesleveragecameracalibration,togetherwiththespatiotemporalinfor-mationofthetargettomaintaintheidentityduringcamerahand-off(Krahnstoeveretal.2006;KhanandShah2006).Asthedimensionsofasitegrow,itquicklybecomesunsustainabletobeabletodeployacameranetworkwherethereareenoughoverlappingfieldofviewstonotleaveuncoveredanyareaofinterest.Intheseconditionstrackingacrosssuch‘‘blindgaps’’cannottakedirectadvantageofthespace-timeproximityofapersonbetweentwocon-secutiveframes,orofthejointcameracalibrationandkinematichistoryoftheperson.Indeed,thereisuncertaintyinthebehaviorofapersoninablindgap,whichisveryhardtopredict,nottomentionthatknowingtheintrinsicandextrinsiccalibration(Maetal.2004)wouldbeacostlytediousprocess.Passivebiometriccuessuchasface(Senioretal.2002),orgait(Wangetal.2003)immediatelystandoutasinformationthatmightbevaluabletoaddresstheidentitymanagementproblemacrossblindgaps.Signaturesdescribingthefaceorgaitofanindividualcouldbeacquired‘‘on-the-fly,’’whilethepersonisbeingreliablytracked.Whenthesamepersonreappearsinafieldofview(fromablindgap),thesametypeofsignatureisextractedandmatchedagainsttheoriginalone.Thisreidentificationprocesswouldallowtoreassigntothatindividualthesameidentity,andtrackinghistory,thatwaspreviouslyassoci-atedtohim.Inotherwords,reidentificationextendstrackingbeyondblindg