Petuun从统计机器学习视角理解深度学习算法理论与可扩展计算英文20189286页

AStatisticalMachineLearningPerspectiveofDeepLearning:Algorithm,Theory,ScalableComputingMaruanAl-Shedivat,ZhitingHu,HaoZhang,andEricXingPetuumInc&CarnegieMellonUniversity•Networkswitches•Infiniband•StochasticGradientDescent/Backpropagation•GraphicalModels•RegularizedBayesianMethods•DeepLearning•SparseCoding•SparseStructuredI/ORegression•Large-Margin•Spectral/MatrixMethods•NonparametricBayesianModels•CoordinateDescent•L-BFGS•GibbsSampling•Metropolis-Hastings•Mahout(MapReduce)•Mllib(BSP)•CNTK•MxNet•Tensorflow(Async)…•Networkattachedstorage•Flashstorage•Servermachines•Desktops/Laptops•NUMAmachines•Mobiledevices•GPUs,CPUs,FPGA,TPU•ARM-powereddevices•RAM•Flash•SSD•Cloudcompute(e.g.AmazonEC2)•IoTnetworks•Datacenters•VirtualmachinesHadoopSparkMPIRPCGraphLab…TaskModelAlgorithmImplementationSystemPlatformandHardwareElementofAI/MachineLearning©Petuum,Inc.1MLvsDL©Petuum,Inc.2Plan•StatisticalAndAlgorithmicFoundationandInsightofDeepLearning•OnUnifiedFrameworkofDeepGenerativeModels•ComputationalMechanisms:DistributedDeepLearningArchitectures©Petuum,Inc.3Part-IBasicsOutline•ProbabilisticGraphicalModels:Basics•AnoverviewofDLcomponents•Historicalremarks:earlydaysofneuralnetworks•Modernbuildingblocks:units,layers,activationsfunctions,lossfunctions,etc.•Reverse-modeautomaticdifferentiation(akabackpropagation)•SimilaritiesanddifferencesbetweenGMsandNNs•Graphicalmodelsvs.computationalgraphs•SigmoidBeliefNetworksasgraphicalmodels•DeepBeliefNetworksandBoltzmannMachines•CombiningDLmethodsandGMs•UsingoutputsofNNsasinputstoGMs•GMswithpotentialfunctionsrepresentedbyNNs•NNswithstructuredoutputs•BayesianLearningofNNs•BayesianlearningofNNparameters•Deepkernellearning©Petuum,Inc.5Outline•ProbabilisticGraphicalModels:Basics•AnoverviewofDLcomponents•Historicalremarks:earlydaysofneuralnetworks•Modernbuildingblocks:units,layers,activationsfunctions,lossfunctions,etc.•Reverse-modeautomaticdifferentiation(akabackpropagation)•SimilaritiesanddifferencesbetweenGMsandNNs•Graphicalmodelsvs.computationalgraphs•SigmoidBeliefNetworksasgraphicalmodels•DeepBeliefNetworksandBoltzmannMachines•CombiningDLmethodsandGMs•UsingoutputsofNNsasinputstoGMs•GMswithpotentialfunctionsrepresentedbyNNs•NNswithstructuredoutputs•BayesianLearningofNNs•BayesianlearningofNNparameters•Deepkernellearning©Petuum,Inc.6Fundamentalquestionsofprobabilisticmodeling•Representation:whatisthejointprobabilitydistr.onmultiplevariables?!(#$,#&,#',…,#))•Howmanystateconfigurationsarethere?•Dotheyallneedtoberepresented?•Canweincorporateanydomain-specificinsightsintotherepresentation?•Learning:wheredowegettheprobabilitiesfrom?•Maximumlikelihoodestimation?Howmuchdatadoweneed?•Arethereanyotherestablishedprinciples?•Inference:ifnotallvariablesareobservable,howtocomputetheconditionaldistributionoflatentvariablesgivenevidence?•Computing!(+|-)wouldrequiresummingover2/configurationsoftheunobservedvariables©Petuum,Inc.7Whatisagraphicalmodel?•Apossibleworldofcellularsignaltransduction©Petuum,Inc.8GM:structuresimplifiesrepresentation•Apossibleworldofcellularsignaltransduction©Petuum,Inc.9ProbabilisticGraphicalModels•If#0’sareconditionallyindependent(asdescribedbyaPGM),thenthejointcanbefactoredintoaproductofsimplerterms•WhywemayfavoraPGM?•Easytoincorporatedomainknowledgeandcausal(logical)structures•Significantreductioninrepresentationcost(21reduceddownto18)!#$,#&,#',#2,#3,#/,#4,#1=!#$!#&!#'#$!#2#&!#3#&!(#/|#',#2)!(#4|#/)!(#1|#3,#/)©Petuum,Inc.10ThetwotypesofGMs•Directededgesassigncausalmeaningtotherelationships(BayesianNetworksorDirectedGraphicalModels)•Undirectededgesrepresentcorrelationsbetweenthevariables(MarkovRandomFieldorUndirectedGraphicalModels)!#$,#&,#',#2,#3,#/,#4,#1=!#$!#&!#'#$!#2#&!#3#&!(#/|#',#2)!(#4|#/)!(#1|#3,#/)!#$,#&,#',#2,#3,#/,#4,#1=17exp {=#$+=#&+=#$,#'+=#&,#2+=#3,#&+ =#',#2,#/+=#/,#4 +=#3,#/,#1}!(+|@)q=argmaxq !q(@)©Petuum,Inc.11Outline•ProbabilisticGraphicalModels:Basics•AnoverviewofDLcomponents•Historicalremarks:earlydaysofneuralnetworks•Modernbuildingblocks:units,layers,activationsfunctions,lossfunctions,etc.•Reverse-modeautomaticdifferentiation(akabackpropagation)•SimilaritiesanddifferencesbetweenGMsandNNs•Graphicalmodelsvs.computationalgraphs•SigmoidBeliefNetworksasgraphicalmodels•DeepBeliefNetworksandBoltzmannMachines•CombiningDLmethodsandGMs•UsingoutputsofNNsasinputstoGMs•GMswithpotentialfunctionsrepresentedbyNNs•NNswithstructuredoutputs•BayesianLearningofNNs•BayesianlearningofNNparameters•Deepkernellearning©Petuum,Inc.12PerceptronandNeuralNets•Frombiologicalneurontoartificialneuron(perceptron)•FrombiologicalneuronnetworktoartificialneuronnetworksThresholdInputsx1x2OutputYåHardLimiterw2w1LinearCombinerqSomaSomaSynapseSynapseDendritesAxonSynapseDendritesAxonInputLayerOutputLayerMiddleLayerInputSignalsOutputSignalsMcCulloch&Pitts(1943)©Petuum,Inc.13Theperceptronlearningalgorithm•Recallthenicepropertyofsigmoidfunction•Considerregressionproblemf:XàY,forscalarY:•Weusedtomaximizetheconditionaldatalikelihood•Here…©Petuum,Inc.14xd=inputtd=targetoutputod=observedoutputwi=weig