Deep Learning简单介绍

DeepLearning简单介绍吴伟新浪微博：tornadomeet博客地址：的定义及历史:DL定义，DL历史，BP算法,pre-training,fine-tuning2.无监督学习模块:1.Auto-encoder(sparseauto-encoder,denoisingauto-encoder,contractiveauto-encoder)2.RBM(loss函数表达式，GRBM,CD算法,PCD算法等)3.SparseCoding(projectgradientdescent,blockcoordinatedescent,ISTA,PSD)3.BuildingaDeepNetwork:1.DNN,DeepAuto-encoder2.DBN3.DBM4.DSN4.ConvolutionalNeuralNetwork(CNN)：1.BPinCNN2.预训练：PCANet,ConvolutionalAuto-encoder,ConvolutionalRBM,ConvolutionalSparseCoding(DeconvolutionNN),ConvolutionalPSD3.CNN改进思路#2Outline5.RecurrentNeuralNetwork(RNN)：基本结构图和BPTT算法。6.SomeTricksinDL:1.trick来源2.dropout,NoiseinActivations,maxout,dropout,dropconnect,ReLU,tiledCNN7.特征的可视化：最大化激活值法、采样法、上层权值线性组合法、Deconvolution法8.DL的应用：AcousticModel,ObjectRecognition(有监督，无监督),NLP9.DL的未来：1.Whyneeddeep?2.What’swrongwithback-propagation?3.ATheoretician'sNightmare?4.What’sneedtobesolvedinDL?5.What’sthehottopicnextinDL?10.DL的一些资料一、DeepLearning定义及历史•DL的定义•DL的历史•BP算法•Pre-training•Fine-tuningDeepLearning的定义•Deeplearningisasetofalgorithmsinmachinelearningthatattempttomodelhigh-levelabstractionsindatabyusingarchitecturescomposedofmultiplenon-lineartransformations.———wikipedia的最新定义1.DL是机器学习算法的一种2.DL是用来提取数据的高层、抽象的特征3.DL是由多重非线性变换复合而成(意味着DL并不一定是DeepNeuralNetwork)ViewMachineLearningasOptimizationProblems#1神经网络发展历史第一阶段：1959年人工智能的起始典型代表：Rosenblatt的感知机1969年Minsky和Papert合写的“感知机”#2神经网络发展历史第二阶段：BP算法感知机的研究被扩展为人工神经网络网络越深，越容易出现梯度弥散问题BackupPropagationDeepLearningofRepresentations-bengio-aaai2013-tutorial#3神经网络发展历史第三阶段：2006年开始ShallowDeepDeeper主要引入的两个思想•layer-wiseunsupervisedlearning•supervisedfine-turningDeeplearning.netDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorial关于DeepLearning的综述•Bengio,Yoshua.LearningdeeparchitecturesforAI.,2009•Bengio,Yoshua.Representationlearning:Areviewandnewperspectives.,2013Bengio的综述侧重于理清各个DL模型之间的内在联系，给出了一定的理论解释，偏科研，比较”高大上”，需要有一定DL理论基础我这里关于DL的介绍侧重表明常见的DL模型是怎样的，偏工程，比较”俗”，需要一些ML基础二、无监督学习模块•Auto-encoder•RestrictedBoltzmannMachine•SparseCoding•Others#1Auto-encoderLarochelle‘sdeeplearningcourse#2Auto-encoderLarochelle‘sdeeplearningcourse#3Auto-encoderLarochelle‘sdeeplearningcourseSparseAutoEncoderDenoisingAuto-encoderLarochelle‘sdeeplearningcourse#1ContractiveAuto-encoderLarochelle‘sdeeplearningcourse#2ContractiveAuto-encoderLarochelle‘sdeeplearningcourse#2ContractiveAuto-encoderLarochelle‘sdeeplearningcourseAuto-encoder小结•如果隐含层采用overcomplete的表示，则不能够extractmeaningfulstructure(因为可以用copydifferentinputcomponent的方法实现)。•如果采用undercomplete的表示，则只对那些符合训练样本分布的样本有效。•因此需要对一些参数加入规则项，这就形成了各种RegularizedAutoencoders.#1RBMwithbinary-valuedinputdataDeepLearningMethodsforVision_HonglakLee-CVPR2012-tutorial#2RBMwithbinary-valuedinputdataDeepLearningMethodsforVision_HonglakLee-CVPR2012-tutorial#1RBMwithreal-valuedinputdataDeepLearningMethodsforVision_HonglakLee-CVPR2012-tutorial#2RBMwithreal-valuedinputdataDeepLearningMethodsforVision_HonglakLee-CVPR2012-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorial#1TrainingRBMsDeepLearningMethodsforVision_HonglakLee-CVPR2012-tutorial#2TrainingRBMsDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialRBM小结•RBM也可以看做是一种Auto-encoder,只不过其权值是tiled的，且loss函数用的是NLL形式。•RBM是无向图，不存在explainaway现象，inference过程很容易。•同样可以对RBM加一些规则项，比如SparseRBM.•RBM理论不太好理解，实际使用时也不方便。SparseCoding-KaiYu-CVPR2012-tutorialSparseCoding-KaiYu-CVPR2012-tutorialDictionaryupdatealgorithmILarochelle‘sdeeplearningcourseDictionaryupdatealgorithmIILarochelle‘sdeeplearningcourseSparseCoding-KaiYu-CVPR2012-tutorialSparsecodinginferencealgorithmLarochelle‘sdeeplearningcourseSparseCoding-KaiYu-CVPR2012-tutorialPSD(PredictiveSparseDecomposition)在普通的SparseCoding上，加入了一个前向网络，目的是解决Sparsecodinginference速度慢的问题，因为在PSD的inference，过程中，可以直接使用前向网络Sparsecoding小结•Sparsecoding也可以看做是Auto-encoder的一种。•SparseCoding学到的特征比较接近人脑视觉皮层的底层特征。•SparseCoding用于提取特征时速度非常慢，PSD算法虽有加速，但训练过程也较慢。三、BuildingaDeepNetwork•DNN,DeepAuto-encoder•DBN•DBM•DSNDNN(=DeepMLP)DeepLearningofRepresentations-bengio-aaai2013-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDeepAuto-encoderDeepBeliefNetwork(DBN)LearningHierarchicalGenera.veModels-Salakhutdinov-CVPR2012-tutorialDeepLearningofRepresentations-bengio-aaai2013-tutorialDBNvsDNN龙星计划2012深度学习课程_邓力DeepBoltzmannNetwork(DBM)LearningHierarchicalGenera.veModels-Salakhutdinov-CVPR2012-tutorialDBMpre