Accurate-Depth-Map-Estimation-from-a-Lenslet-Light

AccurateDepthMapEstimationfromaLensletLightFieldCameraHae-GonJeonJaesikParkGyeongminChoeJinsunParkYunsuBokYu-WingTaiInSoKweonKoreaAdvancedInstituteofScienceandTechnology(KAIST),RepublicofKorea[hgjeon,jspark,gmchoe,ysbok]@rcv.kaist.ac.kr[zzangjinsun,yuwing]@gmail.com,[e-mail]ducesanalgorithmthataccuratelyesti-matesdepthmapsusingalensletlightfieldcamera.Theproposedalgorithmestimatesthemulti-viewstereocor-respondenceswithsub-pixelaccuracyusingthecostvol-ume.Thefoundationforconstructingaccuratecostsisthreefold.First,thesub-apertureimagesaredisplacedus-ingthephaseshifttheorem.Second,thegradientcostsareadaptivelyaggregatedusingtheangularcoordinatesofthelightfield.Third,thefeaturecorrespondencesbetweenthesub-apertureimagesareusedasadditionalconstraints.Withthecostvolume,themulti-labeloptimizationpropa-gatesandcorrectsthedepthmapintheweaktexturere-gions.Finally,thelocaldepthmapisiterativelyrefinedthroughfittingthelocalquadraticfunctiontoestimateanon-discretedepthmap.Becausemicro-lensimagescon-tainunexpecteddistortions,amethodisalsoproposedthatcorrectsthiserror.Theeffectivenessoftheproposedalgo-rithmisdemonstratedthroughchallengingrealworldex-amplesandincludingcomparisonswiththeperformanceofadvanceddepthestimationalgorithms.1.IntroductionTheproblemofestimatinganaccuratedepthmapfromalensletlightfieldcamera,e.g.LytroTM[1]andRaytrixTM[19],isinvestigated.Differenttoconventionalcameras,alightfieldcameracapturesnotonlya2Dimage,butalsothedirectionsoftheincominglightrays.Theaddi-tionallightdirectionsallowtheimagetobere-focusedandthedepthmapofascenetobeestimatedasdemonstratedin[12,17,19,23,26,29].Becausethebaselinebetweensub-apertureimagesfromalensletlightfieldcameraisverynarrow,directlyapplyingtheexistingstereomatchingalgorithmssuchas[20]can-notproducesatisfyingresults,eveniftheappliedalgorithmisatoprankedmethodintheMiddleburystereomatchingbenchmark.AsreportedinYuetal.[29],thedisparityrangeLytrosoftware[1]OursFigure1.SynthesizedviewsofthetwodepthmapsacquiredfromLytrosoftware[1]andourapproach.ofadjacentsub-apertureimagesinLytroisbetween1to1pixels.Consequently,itisverychallengingtoestimateanaccuratedepthmapbecausetheonepixeldisparityerrorisalreadyasignificanterrorinthisproblem.Inthispaper,analgorithmforstereomatchingbetweensub-apertureimageswithanextremelynarrowbaselineispresented.CentraltotheproposedalgorithmistheuseofthephaseshifttheoremintheFourierdomaintoestimatethesub-pixelshiftsofsub-apertureimages.Thisenablestheestimationofthestereocorrespondencesatsub-pixelaccu-racy,evenwithaverynarrowbaseline.Thecostvolumeiscomputedtoevaluatethematchingcostofdifferentdis-paritylabels,whichisdefinedusingthesimilaritymeasure-mentbetweenthesub-apertureimagesandthecenterviewsub-apertureimageshiftedatdifferentsub-pixellocations.Here,thegradientmatchingcostsareadaptivelyaggregatedbasedontheangularcoordinatesofthelightfieldcamera.Inordertoreducetheeffectsofimagenoise,theweightedmedianfilterwasadoptedtoremovethenoiseinthecostvolume,followedbyusingthemulti-labelop-timizationtopropagatereliabledisparitylabelstotheweaktextureregions.Inthemulti-labeloptimization,confidentmatchingcorrespondencesbetweenthecenterviewandotherviewsareusedasadditionalconstraints,whichas-sistinpreventingoversmoothingattheedgesandtextureregions.Finally,theestimateddepthmapisiterativelyre-finedusingquadraticpolynomialinterpolationtoenhancetheestimateddepthmapwithsub-labelprecision.1Intheexperiments,itwasfoundthatamicro-lensim-ageoflensletlightfieldcamerascontainsdepthdistor-tions.Therefore,amethodofcorrectingthiserrorisalsopresented.TheeffectivenessoftheproposedalgorithmisdemonstratedusingchallengingrealworldexamplesthatwerecapturedbyaLytrocamera,aRaytrixcamera,andalab-madelensletlightfieldcamera.Aperformancecompar-isonwithadvancedmethodsisalsopresented.AnexampleoftheresultsoftheproposedmethodarepresentedinFig.1.2.RelatedWorkPreviousworkrelatedtodepthmap(ordisparitymap1)estimationfromalightfieldimageisreviewed.Comparedwithconventionalapproachesinstereomatching,lensletlightfieldimageshaveverynarrowbaselines.Conse-quently,approachesbasedoncorrespondencematchingdonottypicallyworkwellbecausethesub-pixelshiftinthespatialdomainusuallyinvolvesinterpolationwithblurri-ness,andthematchingcostsofstereocorrespondencearehighlyambiguous.Therefore,insteadofusingcorrespon-dencematching,othercuesandconstraintswereusedtoes-timatethedepthmapsfromalensletlightfieldimage.GeorgievandLumsdaine[7]computedanormalizedcrosscorrelationbetweenmicrolensimagesinordertoes-timatethedisparitymap.BishopandFavaro[4]introducedaniterativemethodforamulti-viewstereoimageforalightfield.WannerandGoldluecke[26]usedastructuretensortocomputetheverticalandhorizontalslopesintheepipolarplaneofalightfieldimage,andtheyformulatedthedepthmapestimationproblemasaglobaloptimizationapproachthatwassubjecttotheepipolarconstraint.Yuetal.[29]analyzedthe3Dgeometryoflinesinalightfieldimageandcomputedthedisparitymapsthroughlinematchingbe-tweenthesub-apertureimages.Taoetal.[23]introducedafusionmethodthatusesthecorrespond