Augmenting panoramas with object movies by generat

AugmentingPanoramaswithObjectMoviesbyGeneratingNovelViewswithDisparity-BasedViewMorphingYi-PingHung1,2,Chu-SongChen1,Yu-PaoTsai1,Szu-WeiLin1,21InstituteofInformationScience,AcademiaSinica,Taiwan2DepartmentofComputerScienceandInformationEngineering,NationalTaiwanUniversity,TaiwanEmail:[e-mail]:[phone]ext.1718AbstractOurgoalistoaugmentapanoramawithobjectmoviesinavisually3D-consistentway.Noticethatapanoramaisrecordedasonesingle2Dimageandanobjectmovie(OM)iscomposedofasetof2Dimagestakenarounda3Dobject.Thechallengeishowtointegratetheabovetwosourcesof2Dimagesina3D-consistentwaysothattheusercaneasilymanipulateobjectmoviesinapanorama.Tosolvethisproblem,weadoptapurelyimage-basedapproachthatdoesnothavetoreconstructthegeometricmodelsofthe3Dobjectstobeinsertedinthepanorama.AcriticalissueofthismethodishowtogeneratethenovelviewsrequiredforshowinganOMindifferentplacesofapanorama,andwehaveproposedaviewmorphingtechnique,calledt-DBVM,tosolvethisproblem.Ourexperimentshaveshownthatthispurelyimage-basedapproachcaneffectivelygeneratevisuallyconvincingOM-augmentedpanoramas.Thismethodhasgreatpotentialformanyapplicationsthatrequireintegrationofpanoramasandobjectmovies,suchasvirtualmalls,virtualmuseum,andinteriordesign.1Keywords:panorama,objectmovie,augmentedreality,image-basedrendering,novelviewgeneration,disparity-basedviewmorphing.IntroductionThispaperisconcernedwiththeconstructionofaspecialclassofaugmentedreality,whichwillbereferredtoasaugmentedpanoramas.Inaugmentedpanoramas,someforegroundobjectsareinsertedintoabackgroundscenerecordedintheformofpanoramicimages.Thisbackgroundscenecanbeeitherstaticordynamic.Ifthebackgroundsceneisstatic,wecanfirstacquireasequenceofoverlappingimagesandthenproduceapanoramicimagebyusingimage-mosaickingtechniques1,2.Ifthebackgroundscenetoberecordedisadynamicone,wecancaptureitwithomni-directionalcameras3.Infact,thepanoramicbackgroundcanalsobeacomputer-generatedscene.Theadvantageofusingpanoramatorecordacomputed-generatedvirtualsceneisthattherenderingcanbereal-timenomatterhowcomplexthevirtualenvironmentis.Augmentedpanoramashavemanyusefulapplications,suchasvirtualmall,virtualmuseum,andinteriordesign.Forexample,forInternetshoppinginavirtualmall,differentcommoditiescanbeexhibited,indifferentseasonsorfordifferentcategoriesofcustomers,onthesamecounterofanexistingandwell-decorateddepartmentstore.Insertinga“3Dobject”intoa“3Dscene”wouldbeaneasytask,providedthatwehave2thegeometricmodelsofboththe3Dobjectandthe3Dscene.However,reconstructionof3Dgeometricmodelsisknowntobeatediousandexpensivejob,andonemaywanttoavoiditifpossible.Onthecontrary,panoramasandobjectmoviesarerelativelyeasytocreate,andcanyieldphoto-realisticresults.Examplesofpanoramasandobjectmoviescanbefoundin(OM)isinfactasetofimagestakenfromdifferentperspectivesarounda3Dobject.Hence,theissuebecomesaproblemofintegratingdifferentsourcesof2Dimages,inavisually3D-consistentway,withoutreconstructingtheir3Dgeometricmodels.Thisisnotatrivialproblem,ifwedonothave3Dinformationoftheimage-basedobjectsandthepanoramicscene.Asimilarproblem,whichinserts3Dgeometry-basedobjectsintoimagesorvideosisusuallyreferredtoasaugmentedrealityandhasreceivedmuchattentionsinrecentyears4,5,6,7.Toinsertvirtualobjectsintoa“real”video(i.e.,avideocapturingrealworldorrealevents),amajordifficultyistoestimatetheparametersoftherealcamera.Theparameterscanbeestimatedbyusingreferencepointsorpatterns4,5,orwithoutusinganyreferencepatterns6,7.Insteadofinserting3Dobjectsintovideos,asinmostaugmentedrealityapplicat9ions,Chenetal.8inserted3Dobjectsintoonesingleimagebyutilizingauser-specifiedcuboidstructure3Noticethatthereisnointer-frameinformationavailablewhentheaugmentationisdonewithonesingleimage.AnotherrelatedworkistheonedonebyKatayanaetal.9,10,whichdiscussedhowtomanipulatemultipleperspectiveimagesofanobjectwithina3Dgeometry-basedsceneusingsomepriorknowledgesuchasfeaturepoints,baseplanes,orboundingvolumes.However,toourbestknowledge,nosuccessfulworkhasbeenreportedpreviouslyonintegratingimage-basedobjectsintoimage-basedpanoramicscene.Thispaperpresentsaneasy-to-useapproachtoaugmentingapanoramawithOMs,insuchawaythattheinsertedforegroundobjects,recordedasOMs,alwaysremainvisually3D-consistentwiththepanoramicbackground–evenwhentheusersareinteractivelymanipulatingtheOMsinfrontofthebackground.Inthiswork,theplacesinthepanoramawheretheOMsaretobeinsertedintoarespecifiedbytheuser,ormoreprecisely,bytheauthoroftheOM-augmentedpanorama.ConsidertheexampleshowninFigure1.ThetwoplacesselectedforinsertingOMsareatabletopandacountertop.Figure1(a)showstheentirepanoramicimageofthepanoramatobeaugmented.Figures1(b)and1(c)showthetwoplacesselectedfortheinsertion.SupposeourtaskistoinsertaWinnieintoFigure1(b)andaKittyintoFigure1(c).Then,thetwosnapshotsshowninFigures2(a)and2(b)exempli