李宏毅深度学习

DeepLearningTutorial李宏毅Hung-yiLeeDeeplearningattractslotsofattention.•Ibelieveyouhaveseenlotsofexcitingresultsbefore.Thistalkfocusesonthebasictechniques.DeeplearningtrendsatGoogle.Source:SIGMOD/JeffDeanOutlineLectureIII:BeyondSupervisedLearningLectureII:VariantsofNeuralNetworkLectureI:IntroductionofDeepLearningLectureI:IntroductionofDeepLearningOutlineIntroductionofDeepLearning“HelloWorld”forDeepLearningTipsforDeepLearningMachineLearning≈LookingforaFunction•SpeechRecognition•ImageRecognition•PlayingGo•DialogueSystemffff“Cat”“Howareyou”“5-5”“Hello”“Hi”(whattheusersaid)(systemresponse)(nextmove)FrameworkAsetoffunction21,ff1f“cat”1f“dog”2f“money”2f“snake”Modelf“cat”ImageRecognition:FrameworkAsetoffunction21,fff“cat”ImageRecognition:ModelTrainingDataGoodnessoffunctionfBetter!“monkey”“cat”“dog”functioninput:functionoutput:SupervisedLearningFrameworkAsetoffunction21,fff“cat”ImageRecognition:ModelTrainingDataGoodnessoffunctionf“monkey”“cat”“dog”*fPickthe“Best”FunctionUsingf“cat”TrainingTestingStep1Step2Step3ThreeStepsforDeepLearningStep3:pickthebestfunctionStep2:goodnessoffunctionStep1:defineasetoffunctionNeuralNetworkNeuronbwawawazKKkk11NeuralNetworkz1wkwKw…1akaKabzbiasaweightsNeuron………AsimplefunctionActivationfunctionNeuralNetworkzbiasActivationfunctionweightsNeuron1-2-112-114zzzez11SigmoidFunction0.98NeuralNetworkzzzzDifferentconnectionsleadtodifferentnetworkstructuresWeightsandbiasesarenetworkparameters𝜃Theneuronshavedifferentvaluesofweightsandbiases.FullyConnectFeedforwardNetworkzzzez11SigmoidFunction1-11-21-1104-20.980.12FullyConnectFeedforwardNetwork1-21-1104-20.980.122-1-1-23-14-10.860.110.620.8300-221-1FullyConnectFeedforwardNetwork1-21-1100.730.52-1-1-23-14-10.720.120.510.8500-22𝑓00=0.510.85Givenparameters𝜃,defineafunction𝑓1−1=0.620.8300Thisisafunction.Inputvector,outputvectorGivennetworkstructure,defineafunctionsetOutputLayerHiddenLayersInputLayerFullyConnectFeedforwardNetworkInputOutput1x2xLayer1……Nx……Layer2……LayerL…………………………y1y2yMDeepmeansmanyhiddenlayersneuronWhyDeep?UniversalityTheoremReferenceforthereason::RRfNCanberealizedbyanetworkwithonehiddenlayer(givenenoughhiddenneurons)Why“Deep”neuralnetworknot“Fat”neuralnetwork?•Logiccircuitsconsistsofgates•AtwolayersoflogicgatescanrepresentanyBooleanfunction.•Usingmultiplelayersoflogicgatestobuildsomefunctionsaremuchsimpler•Neuralnetworkconsistsofneurons•Ahiddenlayernetworkcanrepresentanycontinuousfunction.•UsingmultiplelayersofneuronstorepresentsomefunctionsaremuchsimplerlessgatesneededLogiccircuitsNeuralnetworklessparameterslessdata?Morereason:=XsC9byQkUH8&list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49&index=13WhyDeep?Analogy8layers19layers22layersAlexNet(2012)VGG(2014)GoogleNet(2014)16.4%7.3%6.7%=ManyhiddenlayersAlexNet(2012)VGG(2014)GoogleNet(2014)152layers3.57%ResidualNet(2015)Taipei101101layers16.4%7.3%6.7%Deep=ManyhiddenlayersSpecialstructureOutputLayer•SoftmaxlayerastheoutputlayerOrdinaryLayer11zy22zy33zy1z2z3zIngeneral,theoutputofnetworkcanbeanyvalue.MaynotbeeasytointerpretOutputLayer•Softmaxlayerastheoutputlayer1z2z3zSoftmaxLayereee1ze2ze3ze3111jzzjeey31jzje3-312.7200.050.880.12≈0Probability:1𝑦𝑖0𝑦𝑖=1𝑖3122jzzjeey3133jzzjeeyExampleApplicationInputOutput16x16=2561x2x256x……Ink→1Noink→0……y1y2y10Eachdimensionrepresentstheconfidenceofadigit.is1is2is0……0.10.70.2Theimageis“2”ExampleApplication•HandwritingDigitRecognitionMachine“2”1x2x256x…………y1y2y10is1is2is0……Whatisneededisafunction……Input:256-dimvectoroutput:10-dimvectorNeuralNetworkOutputLayerHiddenLayersInputLayerExampleApplicationInputOutput1x2xLayer1……Nx……Layer2……LayerL……………………“2”……y1y2y10is1is2is0……AfunctionsetcontainingthecandidatesforHandwritingDigitRecognitionYouneedtodecidethenetworkstructuretoletagoodfunctioninyourfunctionset.FAQ•Q:Howmanylayers?Howmanyneuronsforeachlayer?•Q:Canwedesignthenetworkstructure?•Q:Canthestructurebeautomaticallydetermined?•Yes,butnotwidelystudiedyet.TrialandErrorIntuition+ConvolutionalNeuralNetwork(CNN)inthenextlectureHighwayNetwork•ResidualNetwork•HighwayNetworkDeepResidualLearningforImageRecognition://arxiv.org/pdf/1507.06228v2.pdf+copycopyGatecontrollerInputlayeroutputlayerInputlayeroutputlayerInputlayeroutputlayerHighwayNetworkautomaticallydeterminesthelayersneeded!ThreeStepsforDeepLearningStep3:pickthebestfunctionStep2:goodnessoffunctionStep1:defineasetoffunctionTrainingData•Preparingtrainingdata:imagesandtheirlabelsThelearningtargetisdefinedonthetrainingdata.“5”“0”“4”“1”“3”“1”“2”“9”LearningTarget16x16=2561x2x……256x……………………Ink→1Noink→0……y1y2y10y1hasthemaximumvalueThelearningtargetis……Input:y2hasthemaximumvalueInput:is1is2is0SoftmaxLoss1x2x……256x…………………………y1y2y10Loss𝑙“1”……100……Losscanbesquareerrororcrossentropybetweenth