Image Inpainting with Structural Bootstrap

ImageInpaintingwithStructuralBootstrapPriorsDanielaCalvetti∗FiorellaSgallari†ErkkiSomersalo‡January11,2006AbstractInthisarticle,weconsiderthefollowinginpaintingproblemarisinginimagerestoration:partofanimagehasbeenremoved,andwewanttorestoretheimagefromtheremaining,possiblynoisy,portion.Weshowthatifthetrueimagecontainsnosharpedges,theinpaintingcanbedonerathersatisfacto-rilybymeansofanisotropicsmoothnesspriorassumption.If,ontheotherhand,wehaveinformationconcerningdiscontinuitiesintheimage,wecaninputthisinformationintotherestorationalgorithmusingananisotropicsmoothnessprior.Basedontheseobservations,weproposeaninpaintingmethodbasedonabootstrappingprocedurethatconsistsofthefollowingsteps:first,wesmoothouttheincompleteimageandcalculatethegradientfield.Sincethisfieldissmooth,itcanbeinpaintedsatisfactorily.Byusingtheinpaintedgradientfield,wethenconstructananisotropicsmoothnesspriorthatpilotstheinpaintingoftheoriginalnon-smoothimage.ThecalculationsarebasedontheBayesianinterpretationoftheinpaintingproblemasastatisticalinverseproblem.1IntroductionReconstructionofmissingordamagedportionsofimagesisanancientpracticeusedextensivelyinartworkrestoration.Thisactivity,alsoknownasinpaintingorretouching,consistsoffillinginthemissingareasormodifyingthedamagedonesinamannernondetectablebyanobservernotfamiliarwiththeoriginalimages.Thegoalofinpaintingalgorithmsvaries,dependingontheapplication,∗DepartmentofMathematics,CaseWesternReserveUniversity,10900EuclidAv.,Cleve-landOH44106,[e-mail]†DepartmentofMathematics/CIRAM,UniversityofBologna,ViaSaragozza8,40123,Bologna,[e-mail]‡InstituteofMathematics,HelsinkiUniversityofTechnology,POBox1100,FIN–02015TKK,[e-mail]1frommakingtheinpaintedpartslookconsistentwiththerestoftheimage,tomakingthemascloseaspossibletotheoriginalimage.Theapplicationsofimageinpaintingrangefromrestorationofphotographs,filmsandpaintings,toremovalofocclusions,suchastext,subtitles,stampsandadvertisementsfromimages.Inaddition,inpaintingcanalsobeusedtoproducespecialeffects.While,traditionally,skilledartistshaveperformedimageinpaintingmanually,currentlydigitaltechniquesareused,e.g.,forautomaticrestorationofscratchedfilms[16].Bertalmioetal.[2]firstintroducedthenotionofdigitalimageinpaintingandusedthirdorderpartialdifferentialequations(PDE)todiffusetheknownimageinformationintothemissingregions.Later,thisinpaintingapproachwasmodi-fiedtotakeintoaccountthedirectionofthelevellines,calledisophotes,[1]andtorelateittotheNavier-Stokesflow[3].Thisoperationpropagatesinformationintothemaskedregionwhilepreservingtheedges.In[19],anisotropicdiffusionisusedtopreserveedgesacrosstheinpaintedregion.Forfurtherdiscussionofvariousmethods,seetherecentsurveyarticles[4],[11][20],[22],[24].Thealgorithmsproposedintheliteraturedifferdependingontheassumptionsmadeaboutthepropertiesoftheimage.Forexample,thetotalvariation(TV)inpaintingmodelproposedin[9],basedontheEuler-Lagrangeequation,em-ploysanisotropicdiffusion[19]basedonthecontrastoftheisophotesinsidetheinpaintingdomain.Thismodel,designedforinpaintingsmallregions,doesagoodjobatremovingnoise,butdoesnotconnectbrokenedges(singlelinesem-beddedinauniformbackground)[9].TheCurvature-DrivenDiffusion(CDD)model[7],extendstheTValgorithmtoalsotakeintoaccountgeometricin-formationofisophoteswhendefiningthe“strength”ofthediffusionprocess,thusallowingtheinpaintingtoproceedoverlargerareas.AlthoughsomeofthebrokenedgesareconnectedbytheCDDapproach,theresultinginterpolatedsegmentsappearblurry.Sincetherearenolocalcriteriaforstoppingthein-painting,theprocessisconstantlyappliedtoallmaskedpixels,regardlessofthelocalsmoothnessoftheregion.Asaresult,computationallyexpensiveoper-ationsmightbeunnecessarilyperformed,resultinginlengthyprocessingtime.Thus,althoughnonlinearPDE-basedimagerestorationmethodshavethepo-tentialofsystematicallypreservingedges,fastnumericalimplementationsaredifficulttodesign[7].Anemergingviewpointamongimageprocessingresearchersisthatglobalfea-tures,suchasshape,shouldbetakenintoaccount.Theideaisthatbyin-corporatingsomeaprioribeliefabouttheshapeandregularityoftheobjectsofinterest,morerobustandefficientimageanalysisalgorithmscouldbede-signed.Novelalgorithms:see,e.g.,[17],[15],forextractingshapesofcontoursofpartiallyoccludedobjectsfromnoisyorlow-contrastimagesuseaBayesianapproachandadoptaregion-basedmodelincorporatingpriorknowledgeofspe-cificshapesofinterest.ThestatisticalpointofviewwasfirstproposedbyGemanandGemanin[12],wherewellknowntechniquesinstatisticalphysicswereintroducedinthecomputervisioncommunity.2InthispaperweapproachtheimageinpaintingproblemfromaBayesianstatis-ticalinversionperspective,withthedataguidingthecontinuationofthepriorprobabilitydistributionintheoccludedareas,leadingtowhatwerefertoasbootstrapprior.Moreprecisely,theinpaintingalgorithmconsistsofthefollow-ingsteps.First,thedamagedimageissmoothedoutandthegradientfieldoutsidetheocclusioniscalculated.Thegradientfieldistheninpainted,andfromthisfield,apilotimageisproduced.Thesmoothpilotimagedefinesastructuralsmoothnessprior,orboots