Multiple motion analysis in spatial or in spectral

Multiplemotionanalysis:inspatialorinspectraldomain?WeichuanYu,a,*GeraldSommer,bandKostasDaniilidiscaDepartmentofDiagnosticRadiology,YaleUniversity,BML332,P.O.Box208042,NewHaven,CT06520-8042,USAbInstituteofComputerScience,ChristianAlbrechtsUniversity,Preusserstrasse1-9,D-24105Kiel,GermanycGRASPLab,UniversityofPennsylvania,3401WalnutStreet,Suite336C,Philadelphia,PA19104-6228,USAReceived28June2001;accepted16January2003AbstractInthispaper,wecomparetheeffectsofmultiplemotionsinspatialandspectralrepresen-tationsofanimagesequence.Wedescribemultiplemotionsinbothdomainsandestablishacomparisonregardingtheirinherentpropertieswhendiscretized.Thoughthespectralmodelprovidesuswithanexplicitdescriptionofbothocclusionandtransparency,itturnsoutthatitsresolutionisverylimited.Weshowthatthespatialdomainrepresentedbythespatio-tem-poralderivativeshassuperiorresolutionpropertiesandisthusmoreappropriateforthetreat-mentofocclusion.Wepresentanalgorithmwhichbasedonaninitialestimateofthenumberofmotionsusestheshift-and-subtracttechniquetolocalizeocclusionboundariesandtotracktheirmovementinocclusionsequences.Thesametechniqueisusedtodistinguishocclusionfromtransparencyandtodecomposetransparencyscenesintomulti-layers.
2003ElsevierScience(USA).Allrightsreserved.Keywords:Motionanalysis;Multiplemotionmodel;Imagesegmentation;Occlusion;Transparency1.IntroductionThedetectionandestimationofmultiplemotionsarechallengingproblemsinthestudyofopticalflow.SinglemotionestimationapproachesarebasedontheComputerVisionandImageUnderstanding90(2003)129–152*Correspondingauthor.Fax:[phone].E-mailaddresses:[e-mail](W.Yu),[e-mail](G.Sommer),[e-mail](K.Daniilidis).1077-3142/03/$-seefrontmatter
2003ElsevierScience(USA).Allrightsreserved.doi:10.1016/S1077-3142(03)00011-0conservationofaresponsecharacterizingthelocalstructureofimages.Forexample,thewell-knownbrightnesschangeconstraintequation(BCCE)[1]IxuþIyvþIt¼0ð1Þisbasedontheconservationofintensity.Opticalflowisdenotedbyðu;vÞandthespatio-temporalderivativesbythetriple(Ix,Iy,It).Itisclearthatsuchaconstancyconstraintmodelsasinglemotioninsidetheneighborhoodinvolvedinestimationorinsidethesupportofthefiltersproducingtheresponsetobeconserved.Thedetectionofmultiplemotionscanbeaddressedassegmentationproblem.However,theopticalflow-fieldsegmentationproblemiscoupledwiththeestimationoftheflowitselfwhichisawell-knownill-posedproblem.Suchcoupledestimation–segmentationproblemshavethechicken-and-eggdilemmainherent.Iftheflowwereaccuratelygiveneverywherethenwewouldbeabletofindthemotionboundaries.Butflowcanbeaccuratelyestimatedonlyifweknowthemotionboundariesandhencetheexactneighborhoodwherethesinglemotionassumptionholds.Asolidapproachwouldbetoformulateaglobalvariationoptimizationproblemwhereunknownsareboththeflowandtheregionboundaries.Indeed,manyapproachespresentedseveralvariantsofdiscontinuitypreservingoranisotropicdiffusionapproaches[1–4].Inthispaper,weareinterestedinthelocaldetectionandestimationofmultiplemotions.Localapproachescanbedividedintwogroupsdependingonthedomaintheywork:spatio-temporalandfrequencydomain.1.1.SpatialapproachesMostofthespatialapproachesarebasedonparametricmodelsappliedinlargeneighborhoods[5].Thesefaceagaintheabove-mentioneddilemmawhichtheytrytosolvewithsuccessiveiterationsbetweensegmentationandestimation.Themostprofoundparadigmisexpectation–maximization(EM),e.g.[6].Bergenetal.[7]pro-posedaniterativemethodbasedontheshift-and-subtractstrategytoestimatetwomotions.Theysubtractedpixelsconnectedtoonemotionduringtherefinementoftheparametersoftheothermotionandviceversa.Iranietal.[8]appliedthisitera-tivemethodinthetemporalintegrationtobluroutuninterestedregionsandtotrackobjectsevenwithnon-consistentspeeds.Inpresentingexplicitmultiplemotionmodelsmanyresearchershavemadecontri-butions.WangandAdelson[9]representedmultiplemotionswithamulti-layermod-el.ThelocalmotionestimationtechniquetheyusedforestimationisstillbasedontheBCCE.Fleetetal.[10]explicitlymodeledanocclusionboundaryinthespatialdomainwithastepfunctioninbothcomponentsoftheopticalflowfieldandusedthesteerabilitytheorytodetecttheboundary.BlackandFleetfurtherproposedtousetheBayesianframeworktodeterminewhichpixelsbelongtothemotionbound-aryregionsandtochooseanappropriatemotionmodel(i.e.,translationalmotionvs.occlusionmotion)[11].BlackandAnandan[12]treatedocclusionregionsasoutliersofthemotioncon-straintandsetlowerweightstotheseregionsintheestimation.Theconceptofanoutlierrepresentsexactlytherelationshipbetweenthepixelsnearocclusionbound-130W.Yuetal./ComputerVisionandImageUnderstanding90(2003)129–152ariesandthepixelswithasinglemotion:Thespatio-temporalpartialderivativesofthepixelswithasinglemotionformaplaneinthederivativespaceandthederiva-tivesofthepixelsnearocclusionboundariesdeviatefromthisplaneduetomotiondiscontinuities.Basedonthisconcept,probabilisticmethodswereproposedtomod-elocclusionboundaries[13]andtoestimatemotionsnearocclusionboundaries[7,14].Outlierswereconsideredasnoiseinstatisticmethod