国科大中科院-人工智能与机器学习-12-DL

深度学习：快速推进中的机器学习与人工智能前沿山世光中科院计算所InstituteofComputingTechnology,ChineseAcademyofSciences提纲深度学习(DL)及其应用前沿DL在CV领域应用的启示关键算法介绍Perceptron及学习算法MLP及其BP算法Auto-EncoderCNN及其主要变种关于DL的思考与讨论2InstituteofComputingTechnology,ChineseAcademyofSciences机器学习的基本任务3)(xFxyClasslabel(Classification)Vector(Estimation){dog,cat,horse,,…}ObjectrecognitionSuperresolutionLow-resolutionimageHigh-resolutionimageInstituteofComputingTechnology,ChineseAcademyofSciences源起——生物神经系统的启示神经元之间通过突触(synapse)连接层级感受野，学习使突触连接增强或变弱甚至消失4Hubel,D.H.&Wiesel,T.N.(1962)InstituteofComputingTechnology,ChineseAcademyofSciences第一代神经网络感知机(Perceptrons)模型[Rosenblatt,1957]二类分类，单个神经元的功能(输入输出关系)f为激活函数，其中：=−=5FrankRosenblatt(1957),ThePerceptron--aperceivingandrecognizingautomaton.Report85-460-1,CornellAeronauticalLaboratory.InstituteofComputingTechnology,ChineseAcademyofSciences第一代神经网络单层感知机(Perceptrons)模型的局限性Minsky&Papert的专著Perceptron(1969)只能对线性可分的模式进行分类解决不了异或问题几乎宣判了这类模型的死刑，导致了随后多年NN研究的低潮6InstituteofComputingTechnology,ChineseAcademyofSciences2ndGenerationNeuralNetworks多层感知机(Multi-layerPerceptron,MLP)超过1层的hiddenlayers（正确输出未知的层）BP算法[Rumelhartetal.,1986]Computeerrorsignal;Then,back-propagateerrorsignaltogetderivativesforlearning7DavidE.Rumelhart,,GeoffreyE.Hinton,andRonaldJ.Williams.(Oct.1986).Learningrepresentationsbyback-propagatingerrors.Nature323(6088):533–536ƩƩƩƩƩƩInstituteofComputingTechnology,ChineseAcademyofSciencesErrorBackpropagationWistheparameterofthenetwork;JistheobjectivefunctionFeedforwardoperationBackerrorpropagationDavidE.Rumelhart,,GeoffreyE.Hinton,andRonaldJ.Williams.(Oct.1986).Learningrepresentationsbyback-propagatingerrors.Nature323(6088):533–536OutputlayerHiddenlayersInputlayerTargetvaluesInstituteofComputingTechnology,ChineseAcademyofSciences2ndGenerationNeuralNetworks理论上多层好两层权重即可逼近任何连续函数映射遗憾的是，训练困难ItrequireslabeledtrainingdataAlmostalldataisunlabeled.ThelearningtimedoesnotscalewellItisveryslowinnetworkswithmultiplehiddenlayers.ItcangetstuckinpoorlocaloptimaTheseareoftenquitegood,butfordeepnetstheyarefarfromoptimal.9InstituteofComputingTechnology,ChineseAcademyofSciences1990-2006更流行…SpecificmethodsforspecifictasksHand-craftedfeatures(SIFT,LBP,HOG)MLmethodsSVMKerneltricksBoostingAdaBoostkNNDecisiontree10Krugeretal.TPAMI’13InstituteofComputingTechnology,ChineseAcademyofSciencesABreakthroughBackto20062006年，通过分层的、无监督预训练，终于获得了训练深层网络结构的能力11InstituteofComputingTechnology,ChineseAcademyofSciencesABreakthroughBackto2006Hinton,G.E.,Osindero,S.andTeh,Y.,Afastlearningalgorithmfordeepbeliefnets.NeuralComputation18:1527-1554,2006Hinton,G.E.andSalakhutdinov,R.R.(2006)Reducingthedimensionalityofdatawithneuralnetworks.Science,Vol.313.no.5786,pp.504-507,28July2006YoshuaBengio,PascalLamblin,DanPopoviciandHugoLarochelle,GreedyLayer-WiseTrainingofDeepNetworks,AdvancesinNeuralInformationProcessingSystems19(NIPS2006)Marc’AurelioRanzato,ChristopherPoultney,SumitChopraandYannLeCun.EfficientLearningofSparseRepresentationswithanEnergy-BasedModel,AdvancesinNeuralInformationProcessingSystems(NIPS2006)12InstituteofComputingTechnology,ChineseAcademyofSciences其实是有例外的——CNN卷积神经网络CNNK.Fukushima,“Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition,”BiologicalCybernetics,vol.36,pp.193–202,1980Y.LeCun,B.Boser,J.S.Denker,D.Henderson,R.E.Howard,W.Hubbard,andL.D.Jackel,“Backpropagationappliedtohandwrittenzipcoderecognition,”NeuralComputation,vol.1,no.4,pp.541–551,1989Y.LeCun,L.Bottou,Y.Bengio,andP.Haffner,“Gradient-basedlearningappliedtodocumentrecognition,”ProceedingsoftheIEEE,vol.86,no.11,pp.2278–2324,199813InstituteofComputingTechnology,ChineseAcademyofSciences其实是有例外的——CNNNeocognitron198014K.Fukushima,“Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition,”BiologicalCybernetics,vol.36,pp.193–202,1980LocalConnectionInstituteofComputingTechnology,ChineseAcademyofSciences例外：CNN用于数字识别15InstituteofComputingTechnology,ChineseAcademyofSciences例外：CNN用于目标检测与识别16InstituteofComputingTechnology,ChineseAcademyofSciences而且，东风同样重要大数据大数据大数据语音图像视频计算能力并行计算平台GPU大量部署开放的社区开源，开放数据17InstituteofComputingTechnology,ChineseAcademyofSciences语音识别(2011)1819862006DBNScienceSpeech2011BPInstituteofComputingTechnology,ChineseAcademyofSciences2012年计算机视觉的巨大进步ImageNet物体分类任务上物体分类任务：1000类，1,431,167幅图像1919862006DBNScienceSpeech20112012RankNameErrorrates(TOP5)Description1U.Toronto0.153Deeplearning2U.Tokyo0.261Hand-craftedfeaturesandlearningmodels.Bottleneck.3U.Oxford0.2704Xerox/INRIA0.271BPInstituteofComputingTechnology,ChineseAcademyofSciencesImageNetwithDeepCNN方法：大规模CNN网络20A.Krizhevsky,L.Sutskever,andG.E.Hinton,“ImageNetClassificationwithDeepConvolutionalNeuralNetworks,”NIPS,2012.InstituteofComputingTechnology,ChineseAcademyofSciencesImageNetwithDeepCNN方法：大规模CNN网络650K神经元,60M参数TrainedwithBPonGPU使用了各种技巧+dropoutReLU,Dataaugment,contrastnormalization,...被Google收编(Jan2013)Google+PhotoTagging(2013.5)21A.Krizhevsky,L.Sutskever,andG.E.Hinton,“ImageNetClassifi