CNN基础与Caffe实践

深度学习一线实战暑期研讨班CNN基础与Caffe实践刘昕博士研究生中科院计算所VIPL研究组vipl.ict.ac.cnInstituteofComputingTechnology,ChineseAcademyofSciences深度学习轻松一刻2InstituteofComputingTechnology,ChineseAcademyofSciencesOutlineCNN结构演化Caffe源码与用法浅析CNN实战技巧：以Caffe为例开放话题讨论3InstituteofComputingTechnology,ChineseAcademyofSciencesOutlineCNN结构演化Caffe源码与用法浅析CNN实战技巧：以Caffe为例开放话题讨论4InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化演化脉络5NeocognitronLeCun1989LeNetAlexNetVGG16VGG19NINGoogLeNetInceptionV3InceptionV4ResNetFastR-CNNSTNetSPP-NetR-CNNFCNFCN+CRFFasterR-CNNInceptionResNetInceptionV2(BN)MSRANet网络加深增强卷积模块功能从分类任务到检测任务增加新的功能单元两条路线的集成，训练更深的网络结构，加速收敛早期尝试历史突破ReLUDropoutCNN+RNN/LSTMHubel&WieselInstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试Hubel&Wiesel1962Hubel&Wiesel对视觉皮层（VisualCortex）的功能划分：从简单特征提取神经元（简单细胞）到渐进复杂的特征提取神经元（复杂细胞，超复杂细胞等）的层级连接结构6Receptivefields,binocularinteractionandfunctionalarchitectureinthecat'svisualcortex模型：功能划分：InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试Hubel&Wiesel研究后续：GaborJonesandPalmer(1987)concludedthattheGaborfunctionprovidesausefulandreasonablyaccuratedescriptionofcellsincat’sV17JudsonandPalmer,Thetwo-Dimensionalspatialstructureofsimplereceptivefieldsincatstriatecortex,JournalofNeurophysiology,1987猫V1细胞感受野Gabor小波两者差异InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试1980-Neocognitron1980(1/2)Hubel&Wiesel模型的计算实现8K.Fukushima:Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition,BiologicalCybernetics,36[4](April1980).KunihikoFukushimaInstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试1980-Neocognitron(2/2)无监督训练并未采用误差反向传播方法进行监督训练，因此并不是CNN9(a)Handwrittendigitrecognitionbyaneocognitron.InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试1980-Neocognitron(2/2)无监督训练并未采用误差反向传播方法进行监督训练，因此并不是CNN10(a)Handwrittendigitrecognitionbyaneocognitron.InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试FirstBaby:LeCun1989(1/3)11Highlights:1)3个隐层，已经可以称为是deepnetwork2)Responsemap之间也共享卷积核权重Y.LeCun,B.Boser,J.S.Denker,D.Henderson,R.E.Howard,W.HubbardandL.D.Jackel:BackpropagationAppliedtoHandwrittenZipCodeRecognition,NeuralComputation,1(4):541-551,Winter1989InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试FirstBaby:LeCun1989(2/3)LeCun埋下的四枚彩蛋12Highlights:1)正切激活函数收敛更快！2)Sigmoid归一+欧氏损失！3)网络参数初始化！神秘的2.4是怎么回事？4)SGD比GD收敛快！InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试FirstBaby:LeCun1989(3/3)为什么tanh收敛更快：tanh(x)的梯度消失问题比sigmoid要轻，神秘的2.4Xavier初始化对应的均匀分布是：LeCun的公式分子2.4对应sqrt(6)，但是分母对不上。但神秘公式与XavierInit的出发点一致：keepingtherange欧氏损失前的sigmoid欧氏损失的梯度：通过归一化限制欧式损失梯度的范围，防止梯度溢出132tanh'()1tanh()(0,1)xx66~[,]outoutininWUnnnn11ˆ()ˆNnnnnLyyyN1'()()(1())(0,)4sxsxsxInstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试1998-LeNet（1/2）14Y.LeCun,L.Bottou,Y.Bengio,andP.Haffner.Gradient-basedlearningappliedtodocumentrecognition.ProceedingsoftheIEEE,november1998选择性连接！InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—早期尝试1998-LeNet（2/2）成功应用于美国邮政手写数字识别系统成长不顺，很快风头被SIFT、LBP、HOG等手工描述子盖过，LeCun自己事后回忆“havinghadcountlessConvNetpapersrejected,publishedandignored,andoccasionallypaidattentionto,forover15years”15截图来自视频(1993年拍摄）：=FwFduRA_L6QInstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—历史突破2012-吾家有女初长成：AlexNet非线性激活函数：ReLU防止过拟合：Dropout,数据增广大数据训练：百万级ImageNet图像数据其他：分Group实现双GPU并行，LRN归一化层16Krizhevsky,Alex,IlyaSutskever,andGeoffreyE.Hinton.Imagenetclassificationwithdeepconvolutionalneuralnetworks.Advancesinneuralinformationprocessingsystems.2012.InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—网络加深2014-VGGNetSlowertobedeeper卷积核大小3x3卷积采样间隔1x1MaxPooling间隔2x2多尺度融合17Simonyan,Karen,andAndrewZisserman.Verydeepconvolutionalnetworksforlarge-scaleimagerecognition.arXivpreprintarXiv:1409.1556(2014).InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—网络加深2015-MSRA-Net(1/2)5x5拆成两个3x3层可以降低计算量，深度增加从而提高性能18He,Kaiming,etal,ConvolutionalNeuralNetworksatConstrainedTimeCost.CVPR2015InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—网络加深2015-MSRA-Net(2/2)19InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—增强卷积模块功能2013-NetworkinNetwork(1/2)MLP网络替换线性卷积去掉全连接层，使用GlobalAveragePooling20MinLinetal,NetworkinNetwork,Arxiv2013.InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—增强卷积模块功能2013-NetworkinNetwork(2/2)21InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—增强卷积模块功能2014-GoogLeNet(1/2)24层网络Inception结构1x1convolutions3x3convolutions5x5convolutionsFilterconcatenationPreviouslayer3x3maxpooling1x1convolutions1x1convolutions1x1convolutions22InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—增强卷积模块功能2014-GoogLeNet(2/2)超大规模24层网络多个Inception结构串联两个辅助损失层23InstituteofComputingTechnology,ChineseAcademyofSciencesCNN结构演化—增强卷积模块功能2015-InceptionV3(1/2)更深的Inception结构5x5卷积核拆成两层3x3卷积核245x5-2x3x3Szegedy,Christian,etal.Ret