Deep Forest Towards An Alternative to Deep Neural

DeepForest:TowardsAnAlternativetoDeepNeuralNetworksZhi-HuaZhouandJiFengNationalKeyLaboratoryforNovelSoftwareTechnologyNanjingUniversity,Nanjing210023,Chinafzhouzh,[e-mail],weproposegcForest,adecisiontreeensembleapproachwithperformancehighlycom-petitivetodeepneuralnetworks.Incontrasttodeepneuralnetworkswhichrequiregreateffortinhyper-parametertuning,gcForestismucheasiertotrain.Actually,evenwhengcForestisappliedtodiffer-entdatafromdifferentdomains,excellentperfor-mancecanbeachievedbyalmostsamesettingsofhyper-parameters.ThetrainingprocessofgcFor-estisefficientandscalable.InourexperimentsitstrainingtimerunningonaPCiscomparabletothatofdeepneuralnetworksrunningwithGPUfacili-ties,andtheefficiencyadvantagemaybemoreap-parentbecausegcForestisnaturallyapttoparallelimplementation.Furthermore,incontrasttodeepneuralnetworkswhichrequirelarge-scaletrainingdata,gcForestcanworkwellevenwhenthereareonlysmall-scaletrainingdata.Moreover,asatree-basedapproach,gcForestshouldbeeasierforthe-oreticalanalysisthandeepneuralnetworks.1IntroductionInrecentyears,deepneuralnetworkshaveachievedgreatsuccessinvariousapplications,particularlyintasksinvolv-ingvisualandspeechinformation[Krizhenvskyetal.,2012;Hintonetal.,2012],leadingtothehotwaveofdeeplearning[Goodfellowetal.,2016].Thoughdeepneuralnetworksarepowerful,theyhaveap-parentdeficiencies.First,itiswellknownthatahugeamountoftrainingdataareusuallyrequiredfortraining,disablingdeepneuralnetworkstobeappliedtotaskswithsmall-scaledata.Notethateveninthebigdataera,manyrealtasksstilllacksufficientamountoflabeleddataduetoexpensivelabelingcost,leadingtoinferiorperformanceofdeepneu-ralnetworksonthosetasks.Second,deepneuralnetworksareverycomplicatedmodelsandpowerfulcomputationalfa-cilitiesareusuallyrequiredforthetrainingprocess,encum-beringindividualsoutsidebigcompaniestofullyexploitthelearningability.Moreimportantly,deepneuralnetworksarewithtoomanyhyper-parameters,andthelearningperfor-mancedependsseriouslyoncarefultuningofthem.Forex-ample,evenwhenseveralauthorsalluseconvolutionalneu-ralnetworks[LeCunetal.,1998;Krizhenvskyetal.,2012;SimonyanandZisserman,2014],theyareactuallyusingdif-ferentlearningmodelsduetothemanydifferentoptionssuchastheconvolutionallayerstructures.Thisfactnotonlymakesthetrainingofdeepneuralnetworksverytricky,likeanartratherthanscience/engineering,butalsomakestheoreticalanalysisofdeepneuralnetworksextremelydifficultbecauseoftoomanyinterferingfactorswithalmostinfiniteconfigu-rationalcombinations.Itiswidelyrecognizedthattherepresentationlearningabilityiscrucialfordeepneuralnetworks.Itisalsonotewor-thythat,toexploitlargetrainingdata,thecapacityoflearningmodelsshouldbelarge;thispartiallyexplainswhythedeepneuralnetworksareverycomplicated,muchmorecomplexthanordinarylearningmodelssuchassupportvectorma-chines.Weconjecturethatifwecanendowthesepropertiestosomeothersuitableformoflearningmodels,wemaybeabletoachieveperformancecompetitivetodeepneuralnet-worksbutwithlessaforementioneddeficiencies.Inthispaper,weproposegcForest(multi-GrainedCascadeforest),anoveldecisiontreeensemblemethod.Thismethodgeneratesadeepforestensemble,withacascadestructurewhichenablesgcForesttodorepresentationlearning.Itsrepresentationallearningabilitycanbefurtherenhancedbymulti-grainedscanningwhentheinputsarewithhighdimen-sionality,potentiallyenablinggcForesttobecontextualorstructuralaware.Thenumberofcascadelevelscanbeadap-tivelydeterminedsuchthatthemodelcomplexitycanbeau-tomaticallyset,enablinggcForesttoperformexcellentlyevenonsmall-scaledata.ItisnoteworthythatgcForesthasmuchfewerhyper-parametersthandeepneuralnetworks;evenbet-ternewsisthatitsperformanceisquiterobusttohyper-parametersettings,suchthatinmostcases,evenacrossdif-ferentdatafromdifferentdomains,itisabletogetexcellentperformancebyusingthedefaultsetting.ThisnotonlymakesthetrainingofgcForestconvenient,butalsomakestheoreti-calanalysis,althoughbeyondthescopeofthispaper,easierthandeepneuralnetworks(needlesstosaythattreelearnersaretypicallyeasiertoanalyzethanneuralnetworks).Inourexperiments,gcForestachieveshighlycompetitiveorevenbetterperformancethandeepneuralnetworks,whereasthetrainingtimecostofgcForestrunningonaPCiscomparabletothatofdeepneuralnetworksrunningwithGPUfacilities.Notethattheefficiencyadvantagecanbemoreapparentbe-arXiv:1702.08835v1[cs.LG]28Feb2017Figure1:Illustrationofthecascadeforeststructure.Eachlevelofthecascadeconsistsoftworandomforests(blue)andtwocomplete-randomtreeforests(black).Supposetherearethreeclassestopredict;thus,eachforestwilloutputathree-dimensionalclassvector,whichisthenconcatenatedforre-representationoftheoriginalinput.causegcForestisnaturallyapttoparallelimplementation.Webelievethattotacklecomplicatedlearningtasks,itislikelythatlearningmodelshavetogodeep.Currentdeepmodels,however,arealwaysneuralnetworks.Thispaperil-lustrateshowtoconstructdeepforest,anditmayopenadoortowardsalternativetodeepneuralnetworksformanytasks.InthenextsectionswewillintroducegcForestandreportonexperiments,followedbyrelatedworkandconclusion.2TheP