Content-based image retrieval color and edges

Content-basedImageRetrieval:ColorandEdgesRobertS.GrayDepartmentofComputerScienceDartmouthCollegeHanover,NH03755E-mail:[e-mail]thatwillbeessentialforfutureelectronicpublishingisapowerfulimageretrievalsystem.Theauthorshouldbeabletosearchanimagedatabaseforimagesthatconveythedesiredinformationormood;areadershouldbeabletosearchacorpusofpublishedworkforimagesthatarerelevanttohisorherneeds.Mostcommercialimageretrievalsystemsassociatekeywordsortextwitheachimageandrequiretheusertoenterakeywordortextualdescriptionofthedesiredimage.Thistext-basedapproachhasnumerousdrawbacks{associatingkeywordsortextwitheachimageisatedioustask;someimagefeaturesmaynotbementionedinthetextualdescription;somefeaturesare\nearlyimpossibletodescribewithtext;andsomefeaturescanbedescribedinwidelydierentways[Na93a].Inaneorttoovercometheseproblemsandimproveretrievalperformance,researchershavefocusedmoreandmoreoncontent-basedimageretrievalinwhichretrievalisaccomplishedbycomparingimagefeaturesdirectlyratherthantextualdescriptionsoftheimagefeatures.Featuresthatarecommonlyusedincontent-basedretrievalincludecolor,shape,textureandedges.Inthispaperwedescribeasimplecontent-basedsystemthatretrievescolorimagesonthebasisoftheircolordistributionsandedgecharacteristics.Thesystemusestworetrievaltechniquesthathavebeendescribedintheliterature{i.e.histogramintersectiontocomparecolordistributionsandsketchcomparisontocompareedgecharacteristics.Theperformanceofthesystemisevaluatedandvariousextensionstotheexistingtechniquesareproposed.1IntroductionOneofthetoolsthatwillbeessentialforfutureelectronicpublishingisapowerfulimageretrievalsystem.Theauthorshouldbeabletosearchanimagedatabaseforimagesthatconveythedesiredinformationormood;thereadershouldbeabletosearchacorpusofpublishedworkforimagesthatarerelevanttohisorherneeds.Mostcommercialimageretrievalsystemsassociatekeywordsortextwitheachimageinthecorpusandrequiretheusertoenterakeywordortextualdescriptionofthedesiredimage.Standardtextretrievaltechniquesarethenusedtoidentifytherelevantimages.Unfortunatelythistext-basedapproachtoimageretrievalhasnumerousdrawbacks[Na93a].Associatingkeywordsortextwitheachimageisatediousandtime-consumingtasksinceitmustbedonemanuallyoratbestsemi-automatically;imageprocessingtechnologyisnotadvancedenoughtoallowtheautomaticconstructionoftextualimagedescriptionsexceptinwell-denedandtightlyfocuseddomains.Someimagefeaturesmaynotbementionedinthetextualdescriptionduetodesigndecisionorindexererror;theseimagefeaturesdonotexistfromthestandpointoftheretrievalsystemandanyquerythatmentionsthemwillfail.Somefeaturesare\nearlyimpossibletodescribewithtext[Na93a];forexamplemanytexturesandshapesdefyeasydescription.Finallydierentindexers{oreventhesameindexer{maydescribethesamefeaturewithdierenttermsordierentfeatureswiththesameterm;thesearethestandardtextretrievalproblemsofsynonymyandpolysemy.Inaneorttoovercometheproblemsofthetext-basedapproachandimproveretrievalperformance,re-searchershavefocusedmoreandmoreoncontent-basedimageretrievalinwhichretrievalisaccomplishedbycomparingimagefeaturesdirectlyratherthantextualdescriptionsoftheimagefeatures.Itishopedthatcontent-basedtechniquescanprovidethebasisforpowerful\querybyexampleretrievalsystems.Forexampletheusermightprovideasamplepictureandrequestsimilarpictures,apictureofanobjectandrequestpicturesthatcontaintheobject,asetofcolorsandrequestimagesthatcontainthosecolors,andsoon.Featuresthatarecommonlyusedincontent-basedimageretrievalincludecolor,shape,textureandedges.1PartiallysupportedbyAFOSRcontractF49620-93-1-0266andAFOSR/DARPA89-05361ImagesColorextractionQueryprocessingformulationGUIforqueryedgemapsEdgeextractionHistogramsandFigure1:Thearchitectureoftheimageretrievalsystem{themainmodulesareedgeextraction,colorextraction,queryprocessinganduserinterface.Inthispaperwedescribeasimplecontent-basedsystemthatretrievescolorimagesonthebasisoftheircolordistributionsandedgecharacteristics.Thesystemisfullyautomaticasnomanualinterventionisrequiredduringtheindexingprocess.Fullautomation{alongwitheectiveretrievalperformance{istheprimarygoalofthesystemsinceamanualorsemi-automaticindexingprocessiserror-proneandtime-intensive.Thesystemdoesnotdevelopanynovelretrievaltechniquesbutinsteadusesexistingtechniquesthathavebeendescribedintheliterature{i.ehistogramintersection[SB91,Swa93]isusedtocomparecolordistributionsandsketchcomparison[HK92,KKOH92]isusedtocompareedgecharacteristics.Itishopedthatthesystemwillhighlightpotentialavenuesofresearchandserveasatestbedforfuturework.Tothisend,theperformanceofthesystemisevaluatedandvariousextensionstotheexistingretrievaltechniquesareproposed.Thenextsectiondescribestheimplementationofthesystem.Theremainingsectionsdiscusstheweaknessesofthecurrentimplementationandmethodsforaddressingtheseweaknesses.2ImplementationThesystemisimplementedasfourmodules{edgeextraction,colorextraction,queryprocessinganduserinterface.Thecolorandedgeextractionmodulesconstructasetofhistogramsandanedgemapforeachimage.Nomanualinterventio