a comparsion of affine region detectors

InternationalJournalofComputerVision65(1/2),43–72,2005c
2005SpringerScience+BusinessMedia,Inc.ManufacturedinTheNetherlands.DOI:10.1007/s11263-005-3848-xAComparisonofAffineRegionDetectorsK.MIKOLAJCZYKUniversityofOxford,OX13PJ,Oxford,[e-mail]uven,KasteelparkArenberg10,3001Leuven,[e-mail],GRAVIR-CNRS,655,av.del’Europe,38330,Montbonnot,[e-mail]ord,OX13PJ,Oxford,[e-mail]rsity,KarlovoNamesti13,12135,Prague,[e-mail],OX13PJ,Oxford,[e-mail]@robots.ox.ac.ukL.VANGOOLUniversityofLeuven,KasteelparkArenberg10,3001Leuven,[e-mail]20,2004;RevisedMay3,2005;AcceptedMay11,2005FirstonlineversionpublishedinJanuary,2006Abstract.Thepapergivesasnapshotofthestateoftheartinaffinecovariantregiondetectors,andcomparestheirperformanceonasetoftestimagesundervaryingimagingconditions.Sixtypesofdetectorsareincluded:detectorsbasedonaffinenormalizationaroundHarris(MikolajczykandSchmid,2002;SchaffalitzkyandZisserman,2002)andHessianpoints(MikolajczykandSchmid,2002),adetectorof‘maximallystableextremalregions’,proposedbyMatasetal.(2002);anedge-basedregiondetector(TuytelaarsandVanGool,1999)andadetectorbasedonintensityextrema(TuytelaarsandVanGool,2000),andadetectorof‘salientregions’,44Mikolajczyketal.proposedbyKadir,ZissermanandBrady(2004).Theperformanceismeasuredagainstchangesinviewpoint,scale,illumination,defocusandimagecompression.Theobjectiveofthispaperisalsotoestablishareferencetestsetofimagesandperformancesoftware,sothatfuturedetectorscanbeevaluatedinthesameframework.Keywords:affineregiondetectors,invariantimagedescription,localfeatures,performanceevaluation1.IntroductionDetectingregionscovariantwithaclassoftransforma-tionshasnowreachedsomematurityinthecomputervisionliterature.Theseregionshavebeenusedinquitevariedapplicationsincluding:widebaselinematchingforstereopairs(Baumberg,2000;Matasetal.,2002;PritchettandZisserman,1998;TuytelaarsandVanGool,2000),reconstructingcamerasforsetsofdisparateviews(SchaffalitzkyandZisserman,2002),imageretrievalfromlargedatabases(SchmidandMohr,1997;TuytelaarsandVanGool,1999),modelbasedrecognition(Ferrarietal.,2004;Lowe,1999;Obdrˇz´alekandMatas,2002;Rothgangeretal.,2003),objectretrievalinvideo(SivicandZisserman,2003;Sivicetal.,2004),visualdatamining(SivicandZisserman,2004),texturerecognition(Lazebniketal.,2003a,b),shotlocation(SchaffalitzkyandZisserman,2003),robotlocalization(Seetal.,2002)andservoing(Tuytelaarsetal.,1999),buildingpanoramas(BrownandLowe,2003),symmetrydetection(Turinaetal.,2001),andobjectcategoriza-tion(Csurkaetal.,2004;DorkoandSchmid,2003;Fergusetal.,2003;Opeltetal.,2004).Therequirementfortheseregionsisthattheyshouldcorrespondtothesamepre-imagefordif-ferentviewpoints,i.e.,theirshapeisnotfixedbutautomaticallyadapts,basedontheunderlyingimageintensities,sothattheyaretheprojectionofthesame3Dsurfacepatch.Inparticular,considerimagesfromtwoviewpointsandthegeometrictransformationbetweentheimagesinducedbytheviewpointchange.Regionsdetectedaftertheviewpointchangeshouldbethesame,modulonoise,asthetransformedversionsoftheregionsdetectedintheoriginalimage–imagetransformationandregiondetectioncommute.Assuch,eventhoughtheyhaveoftenbeencalledinvariantregionsintheliterature(e.g.,DorkoandSchmid,2003;Lazebniketal.,2003a;SivicandZisserman,2004;TuytelaarsandVanGool,1999),inprincipletheyshouldbetermedcovariantregionssincetheychangecovariantlywiththetransformation.Theconfusionprobablyarisesfromthefactthat,eventhoughtheregionsthemselvesarecovariant,thenor-malizedimagepatterntheycoverandthefeaturede-scriptorsderivedfromthemaretypicallyinvariant.Note,ouruseoftheterm‘region’simplyreferstoasetofpixels,i.e.anysubsetoftheimage.Thisdiffersfromclassicalsegmentationsincetheregionbound-ariesdonothavetocorrespondtochangesinimageappearancesuchascolourortexture.Allthedetectorspresentedhereproducesimplyconnectedregions,butingeneralthisneednotbethecase.Forviewpointchanges,thetransformationofmostinterestisanaffinity.ThisisillustratedinFig.1.Clearly,aregionwithfixedshape(acircularexam-pleisshowninFig.1(a)and(b))cannotcopewiththegeometricdeformationscausedbythechangeinview-point.Wecanobservethatthecircledoesnotcoverthesameimagecontent,i.e.,thesamephysicalsurface.Instead,theshapeoftheregionhastobeadaptive,orcovariantwithrespecttoaffinities(Fig.1(c)–close-upsshowninFig.1(d)–(f)).Indeed,anaffinityissuffi-cienttolocallymodelimagedistortionsarisingfromviewpointchanges,providedthat(1)thescenesur-facecanbelocallyapproximatedbyaplaneorincaseofarotatingcamera,and(2)perspectiveeffectsareignored,whicharetypicallysmallonalocalscaleany-way.Asidefromthegeometricdeformations,alsopho-tometricdeformationsneedtobetakenintoaccount.Thesecanbemodeledbyalineartransformationoftheintensities.Tofurtherillustratetheseissues,andhowaffinecovariantregionscanbeexploitedtocopewi