Deep-Sparse-Rectifier-Neural-Networks

315DeepSparseRectierNeuralNetworksXavierGlorotAntoineBordesYoshuaBengioDIRO,UniversitedeMontrealMontreal,QC,[e-mail],UMRCNRS6599UTC,Compiegne,FranceandDIRO,UniversitedeMontrealMontreal,QC,[e-mail],UniversitedeMontrealMontreal,QC,[e-mail]igmoidneuronsaremorebi-ologicallyplausiblethanhyperbolictangentneurons,thelatterworkbetterfortrain-ingmulti-layerneuralnetworks.Thispa-pershowsthatrectifyingneuronsareanevenbettermodelofbiologicalneuronsandyieldequalorbetterperformancethanhy-perbolictangentnetworksinspiteofthehardnon-linearityandnon-dierentiabilityatzero,creatingsparserepresentationswithtruezeros,whichseemremarkablysuitablefornaturallysparsedata.Eventhoughtheycantakeadvantageofsemi-supervisedsetupswithextra-unlabeleddata,deeprectiernet-workscanreachtheirbestperformancewith-outrequiringanyunsupervisedpre-trainingonpurelysupervisedtaskswithlargelabeleddatasets.Hence,theseresultscanbeseenasanewmilestoneintheattemptsatunder-standingthedicultyintrainingdeepbutpurelysupervisedneuralnetworks,andclos-ingtheperformancegapbetweenneuralnet-workslearntwithandwithoutunsupervisedpre-training.1IntroductionManydierencesexistbetweentheneuralnetworkmodelsusedbymachinelearningresearchersandthoseusedbycomputationalneuroscientists.ThisisinpartAppearinginProceedingsofthe14thInternationalCon-ferenceonArticialIntelligenceandStatistics(AISTATS)2011,FortLauderdale,FL,USA.Volume15ofJMLR:W&CP15.Copyright2011bytheauthors.becausetheobjectiveoftheformeristoobtaincom-putationallyecientlearners,thatgeneralizewelltonewexamples,whereastheobjectiveofthelatteristoabstractoutneuroscienticdatawhileobtainingex-planationsoftheprinciplesinvolved,providingpredic-tionsandguidanceforfuturebiologicalexperiments.Areaswherebothobjectivescoincidearethereforeparticularlyworthyofinvestigation,pointingtowardscomputationallymotivatedprinciplesofoperationinthebrainthatcanalsoenhanceresearchinarticialintelligence.Inthispaperweshowthattwocom-mongapsbetweencomputationalneurosciencemodelsandmachinelearningneuralnetworkmodelscanbebridgedbyusingthefollowinglinearbypartactiva-tion:max(0;x),calledtherectier(orhinge)activa-tionfunction.Experimentalresultswillshowengagingtrainingbehaviorofthisactivationfunction,especiallyfordeeparchitectures(seeBengio(2009)forareview),i.e.,wherethenumberofhiddenlayersintheneuralnetworkis3ormore.Recenttheoreticalandempiricalworkinstatisticalmachinelearninghasdemonstratedtheimportanceoflearningalgorithmsfordeeparchitectures.Thisisinpartinspiredbyobservationsofthemammalianvi-sualcortex,whichconsistsofachainofprocessingelements,eachofwhichisassociatedwithadierentrepresentationoftherawvisualinput.Thisispartic-ularlyclearintheprimatevisualsystem(Serreetal.,2007),withitssequenceofprocessingstages:detectionofedges,primitiveshapes,andmovinguptogradu-allymorecomplexvisualshapes.Interestingly,itwasfoundthatthefeatureslearnedindeeparchitecturesresemblethoseobservedinthersttwoofthesestages(inareasV1andV2ofvisualcortex)(Leeetal.,2008),andthattheybecomeincreasinglyinvarianttofactorsofvariation(suchascameramovement)inhigherlay-ers(Goodfellowetal.,2009).316DeepSparseRectierNeuralNetworksRegardingthetrainingofdeepnetworks,somethingthatcanbeconsideredabreakthroughhappenedin2006,withtheintroductionofDeepBeliefNet-works(Hintonetal.,2006),andmoregenerallytheideaofinitializingeachlayerbyunsupervisedlearn-ing(Bengioetal.,2007;Ranzatoetal.,2007).Someauthorshavetriedtounderstandwhythisunsuper-visedprocedurehelps(Erhanetal.,2010)whileoth-ersinvestigatedwhytheoriginaltrainingprocedurefordeepneuralnetworksfailed(BengioandGlorot,2010).Fromthemachinelearningpointofview,thispaperbringsadditionalresultsintheselinesofinvestigation.Weproposetoexploretheuseofrectifyingnon-linearitiesasalternativestothehyperbolictangentorsigmoidindeeparticialneuralnetworks,inad-ditiontousinganL1regularizerontheactivationval-uestopromotesparsityandpreventpotentialnumer-icalproblemswithunboundedactivation.NairandHinton(2010)presentpromisingresultsoftheinu-enceofsuchunitsinthecontextofRestrictedBoltz-mannMachinescomparedtologisticsigmoidactiva-tionsonimageclassicationtasks.Ourworkextendsthisforthecaseofpre-trainingusingdenoisingauto-encoders(Vincentetal.,2008)andprovidesanexten-siveempiricalcomparisonoftherectifyingactivationfunctionagainstthehyperbolictangentonimageclas-sicationbenchmarksaswellasanoriginalderivationforthetextapplicationofsentimentanalysis.Ourexperimentsonimageandtextdataindicatethattrainingproceedsbetterwhenthearticialneuronsareeitherooroperatingmostlyinalinearregime.Sur-prisingly,rectifyingactivationallowsdeepnetworkstoachievetheirbestperformancewithoutunsupervisedpre-training.Hence,ourworkproposesanewcontri-butiontothetrendofunderstandingandmergingtheperformancegapbetweendeepnetworkslearntwithandwithoutunsupervisedpre-training(Erhanetal.,2010;BengioandGlorot,2010).Still,rectiernet-workscanbenetfromunsupervisedpre-traininginthecontextofsemi-sup