深度学习教程 李宏毅

DeepLearningTutorial李宏毅Hung-yiLeeDeeplearningattractslotsofattention.•Ibelieveyouhaveseenlotsofexcitingresultsbefore.Thistalkfocusesonthebasictechniques.DeeplearningtrendsatGoogle.Source:SIGMOD/JeffDeanOutlineLectureIV:NextWaveLectureIII:VariantsofNeuralNetworkLectureII:TipsforTrainingDeepNeuralNetworkLectureI:IntroductionofDeepLearningLectureI:IntroductionofDeepLearningOutlineofLectureIIntroductionofDeepLearningWhyDeep?“HelloWorld”forDeepLearningLet’sstartwithgeneralmachinelearning.MachineLearning≈LookingforaFunction•SpeechRecognition•PlayingGo•DialogueSystemf•ImageRecognitionfff“Howareyou”“Cat”“Hello”“Hi”(whattheusersaid)(systemresponse)“5-5”(nextmove)f1f1“cat”“dog”f2f2“money”“snake”FrameworkModelAsetoffunctionf1,f2f“cat”ImageRecognition:“cat”ImageRecognition:FrameworkModelAsetoffunctionf1,f2TrainingDatafBetter!“cat”“dog”functioninput:functionoutput:“monkey”GoodnessoffunctionfSupervisedLearningFrameworkAsetoffunctionf1,f2f“cat”ImageRecognition:ModelTrainingData“monkey”“cat”“dog”Usingf“cat”TrainingTestingStep1GoodnessoffunctionfStep2Pickthe“Best”Functionf*Step3Step1:defineasetoffunctionStep2:goodnessoffunctionStep3:pickthebestfunctionThreeStepsforDeepLearningDeepLearningissosimple……NeuralofNetworkStep1:defineasetfunctionStep2:goodnessoffunctionStep3:pickthebestfunctionThreeStepsforDeepLearningDeepLearningissosimple……HumanBrains…………w1a1akaKbawkwKweightsNeuralNetworkNeuronza1w1akwkaKwKbAsimplefunctionzzActivationfunctionbiasNeuralNetworkzActivationfunctionbiasNeuron1-2-1weights12-114zzz11ezSigmoidFunction0.98zzzzNeuralNetworkDifferentconnectionsleadstodifferentnetworkstructureEachneuronscanhavedifferentvaluesofweightsandbiases.Weightsandbiasesarenetworkparameters𝜃FullyConnectFeedforwardNetworkzzz11ezSigmoidFunction1-11-2-1114-200.980.12FullyConnectFeedforwardNetwork1-21-1104-20.120.982-1-1-2-14-10.8630.110.620.8300-221-1FullyConnectFeedforwardNetwork1-2-11100.50.732-1-2-13-1-140.720.120.510.8500-22𝑓00=0.510.85𝑓1−1=0.620.8300Thisisafunction.Inputvector,outputvectorGivenparameters𝜃,defineafunctionGivennetworkstructure,defineafunctionset…………………………OutputLayerHiddenLayersInputLayerFullyConnectFeedforwardNetworkLayer1Inputx1x2xNLayer2LayerL………………Outputy1y2yMDeepmeansmanyhiddenlayersneuronOutputLayer(Option)•SoftmaxlayerastheoutputlayerOrdinaryLayery1z1y2z2y3z3z1z2z3Ingeneral,theoutputofnetworkcanbeanyvalue.Maynotbeeasytointerprety1eez2z22.70.05eeeez1•SoftmaxlayerastheoutputlayerSoftmaxLayereeez1ee3j1z1zjz33j1zj31z3-3200.88OutputLayer(Option)Probability:1𝑦𝑖0𝑖𝑦𝑖=13j13j1z2z3zjzj0.12y2e≈0y3e………………ExampleApplicationInputOutput16x16=256x1x2x256Ink→1Noink→0yy2y10Eachdimensionrepresentstheconfidenceofadigit.is1is2is00.10.70.2Theimageis“2”………………MachineExampleApplication•HandwritingDigitRecognitionx1x2x256y1y2“2”y10is1is2is0function……Input:256-dimvectoroutput:10-dimvectorNeuralNetworkWhatisneededisa………………………………ExampleApplicationInputOutputLayer1x1x2xNInputLayerLayer2LayerL……y1y2“2”y10is1is2is0……AfunctionsetcontainingthecandidatesforHandwritingDigitRecognition……OutputHiddenLayersLayerYouneedtodecidethenetworkstructuretoletagoodfunctioninyourfunctionset.FAQ•Q:Howmanylayers?Howmanyneuronsforeachlayer?•Q:Canthestructurebeautomaticallydetermined?TrialandErrorIntuition+NeuralofNetworkStep1:defineasetfunctionStep2:goodnessoffunctionStep3:pickthebestfunctionThreeStepsforDeepLearningDeepLearningissosimple……TrainingData•Preparingtrainingdata:imagesandtheirlabelsThelearningtargetisdefinedonthetrainingdata.“1”“3”“4”“1”“0”“2”“5”“9”Softmax………………LearningTarget16x16=256x1x2x256………………Ink→1Noink→0y1y2y10Thelearningtargetis……y1hasthemaximumvaluey2hasthemaximumvalueInput:Input:is1is2is0…………………………GivenasetofLossx1x2xN………………y2y10Loss𝑙00Losscanbethedistancebetweenthenetworkoutputandtargettargety1Ascloseas1possibleAgoodfunctionshouldmakethelossofallexamplesassmallaspossible.“1”parameters……………………TotalLossxRNNyR𝑦𝑅x1x2x3NNNNNNy1y2y3𝑦1𝑦2𝑦3Foralltrainingdata…𝐿=𝑟=1𝑅𝑙𝑟Findthenetworkparameters𝜽∗thatminimizetotallossLTotalLoss:𝑙1𝑙2𝑙3𝑙𝑅AssmallaspossibleFindafunctioninfunctionsetthatminimizestotallossLNeuralofNetworkStep1:defineasetfunctionStep2:goodnessoffunctionStep3:pickthebestfunctionThreeStepsforDeepLearningDeepLearningissosimple………………HowtopickthebestfunctionFindnetworkparameters𝜽∗thatminimizetotallossLLayerl1000neuronsLayerl+1106weights1000neuronsEnumerateallpossiblevaluesNetworkparameters𝜃=𝑤1,𝑤2,𝑤3,⋯,𝑏1,𝑏2,𝑏3,⋯MillionsofparametersE.g.speechrecognition:8layersand1000neuronseachlayerGradientDescentTotalLoss𝐿Random,RBMpre-trainUsuallygoodenoughNetworkparameters𝜃=𝑤1,𝑤2,⋯,𝑏1,𝑏2,⋯wFindnetworkparameters𝜽∗thatminimizetotallossLPickaninitialvalueforwGradientDescentTotalLoss𝐿Networkparameters𝜃=𝑤1,𝑤2,⋯,𝑏1,𝑏2,⋯Compute𝜕𝐿𝜕𝑤IncreasewDecreasewwNegativePositive𝜽∗thatminimizetotallossLPickaninitialvalueforwGradientDescentTotalLo