A-Neural-Algorithm-of-Artistic-Style

ANeuralAlgorithmofArtisticStyleLeonA.Gatys,1;2;3AlexanderS.Ecker,1;2;4;5MatthiasBethge1;2;41WernerReichardtCentreforIntegrativeNeuroscienceandInstituteofTheoreticalPhysics,UniversityofT¨ubingen,Germany2BernsteinCenterforComputationalNeuroscience,T¨ubingen,Germany3GraduateSchoolforNeuralInformationProcessing,T¨ubingen,Germany4MaxPlanckInstituteforBiologicalCybernetics,T¨ubingen,Germany5DepartmentofNeuroscience,BaylorCollegeofMedicine,Houston,TX,USATowhomcorrespondenceshouldbeaddressed;E-mail:[e-mail]fineart,especiallypainting,humanshavemasteredtheskilltocreateuniquevisualexperiencesthroughcomposingacomplexinterplaybetweenthecon-tentandstyleofanimage.Thusfarthealgorithmicbasisofthisprocessisunknownandthereexistsnoartificialsystemwithsimilarcapabilities.How-ever,inotherkeyareasofvisualperceptionsuchasobjectandfacerecognitionnear-humanperformancewasrecentlydemonstratedbyaclassofbiologicallyinspiredvisionmodelscalledDeepNeuralNetworks.1,2HereweintroduceanartificialsystembasedonaDeepNeuralNetworkthatcreatesartisticimagesofhighperceptualquality.Thesystemusesneuralrepresentationstosepa-rateandrecombinecontentandstyleofarbitraryimages,providinganeuralalgorithmforthecreationofartisticimages.Moreover,inlightofthestrik-ingsimilaritiesbetweenperformance-optimisedartificialneuralnetworksandbiologicalvision,3–7ourworkoffersapathforwardtoanalgorithmicunder-standingofhowhumanscreateandperceiveartisticimagery.1arXiv:1508.06576v1[cs.CV]26Aug2015TheclassofDeepNeuralNetworksthataremostpowerfulinimageprocessingtasksarecalledConvolutionalNeuralNetworks.ConvolutionalNeuralNetworksconsistoflayersofsmallcomputationalunitsthatprocessvisualinformationhierarchicallyinafeed-forwardman-ner(Fig1).Eachlayerofunitscanbeunderstoodasacollectionofimagefilters,eachofwhichextractsacertainfeaturefromtheinputimage.Thus,theoutputofagivenlayerconsistsofso-calledfeaturemaps:differentlyfilteredversionsoftheinputimage.WhenConvolutionalNeuralNetworksaretrainedonobjectrecognition,theydeveloparepresentationoftheimagethatmakesobjectinformationincreasinglyexplicitalongthepro-cessinghierarchy.8Therefore,alongtheprocessinghierarchyofthenetwork,theinputimageistransformedintorepresentationsthatincreasinglycareabouttheactualcontentoftheim-agecomparedtoitsdetailedpixelvalues.Wecandirectlyvisualisetheinformationeachlayercontainsabouttheinputimagebyreconstructingtheimageonlyfromthefeaturemapsinthatlayer9(Fig1,contentreconstructions,seeMethodsfordetailsonhowtoreconstructtheim-age).Higherlayersinthenetworkcapturethehigh-levelcontentintermsofobjectsandtheirarrangementintheinputimagebutdonotconstraintheexactpixelvaluesofthereconstruc-tion.(Fig1,contentreconstructionsd,e).Incontrast,reconstructionsfromthelowerlayerssimplyreproducetheexactpixelvaluesoftheoriginalimage(Fig1,contentreconstructionsa,b,c).Wethereforerefertothefeatureresponsesinhigherlayersofthenetworkasthecontentrepresentation.Toobtainarepresentationofthestyleofaninputimage,weuseafeaturespaceoriginallydesignedtocapturetextureinformation.8Thisfeaturespaceisbuiltontopofthefilterresponsesineachlayerofthenetwork.Itconsistsofthecorrelationsbetweenthedifferentfilterresponsesoverthespatialextentofthefeaturemaps(seeMethodsfordetails).Byincludingthefeaturecorrelationsofmultiplelayers,weobtainastationary,multi-scalerepresentationoftheinputimage,whichcapturesitstextureinformationbutnottheglobalarrangement.2Figure1:ConvolutionalNeuralNetwork(CNN).AgiveninputimageisrepresentedasasetoffilteredimagesateachprocessingstageintheCNN.Whilethenumberofdifferentfiltersincreasesalongtheprocessinghierarchy,thesizeofthefilteredimagesisreducedbysomedownsamplingmechanism(e.g.max-pooling)leadingtoadecreaseinthetotalnumberofunitsperlayerofthenetwork.ContentReconstructions.WecanvisualisetheinformationatdifferentprocessingstagesintheCNNbyreconstructingtheinputimagefromonlyknow-ingthenetwork’sresponsesinaparticularlayer.Wereconstructtheinputimagefromfromlayers‘conv11’(a),‘conv21’(b),‘conv31’(c),‘conv41’(d)and‘conv51’(e)oftheorig-inalVGG-Network.Wefindthatreconstructionfromlowerlayersisalmostperfect(a,b,c).Inhigherlayersofthenetwork,detailedpixelinformationislostwhilethehigh-levelcontentoftheimageispreserved(d,e).StyleReconstructions.OntopoftheoriginalCNNrepresentationswebuiltanewfeaturespacethatcapturesthestyleofaninputimage.ThestylerepresentationcomputescorrelationsbetweenthedifferentfeaturesindifferentlayersoftheCNN.Werecon-structthestyleoftheinputimagefromstylerepresentationsbuiltondifferentsubsetsofCNNlayers(‘conv11’(a),‘conv11’and‘conv21’(b),‘conv11’,‘conv21’and‘conv31’(c),‘conv11’,‘conv21’,‘conv31’and‘conv41’(d),‘conv11’,‘conv21’,‘conv31’,‘conv41’and‘conv51’(e)).Thiscreatesimagesthatmatchthestyleofagivenimageonanincreasingscalewhilediscardinginformationoftheglobalarrangementofthescene.3Again,wecanvisualisetheinformationcapturedbythesestylefeaturespacesbuiltondifferentlayersofthenetworkbyconstructinganimagethatmatchesthestylerepresentationofagiveninputimage(Fig1,stylereconstructions).10,11Indeedreconstructionsfromthestylefeatur