Gradient-based-learning-applied-to-document-recogn

Gradient-BasedLearningAppliedtoDocumentRecognitionYANNLECUN,MEMBER,IEEE,L´EONBOTTOU,YOSHUABENGIO,ANDPATRICKHAFFNERInvitedPaperMultilayerneuralnetworkstrainedwiththeback-propagationalgorithmconstitutethebestexampleofasuccessfulgradient-basedlearningtechnique.Givenanappropriatenetworkarchitecture,gradient-basedlearningalgorithmscanbeusedtosynthesizeacomplexdecisionsurfacethatcanclassifyhigh-dimensionalpatterns,suchashandwrittencharacters,withminimalpreprocessing.Thispaperreviewsvariousmethodsappliedtohandwrittencharacterrecognitionandcomparesthemonastandardhandwrittendigitrecognitiontask.Convolutionalneuralnetworks,whicharespecificallydesignedtodealwiththevariabilityoftwodimensional(2-D)shapes,areshowntooutperformallothertechniques.Real-lifedocumentrecognitionsystemsarecomposedofmultiplemodulesincludingfieldextraction,segmentation,recognition,andlanguagemodeling.Anewlearningparadigm,calledgraphtransformernetworks(GTN’s),allowssuchmultimodulesystemstobetrainedgloballyusinggradient-basedmethodssoastominimizeanoverallperformancemeasure.Twosystemsforonlinehandwritingrecognitionaredescribed.Experimentsdemonstratetheadvantageofglobaltraining,andtheflexibilityofgraphtransformernetworks.Agraphtransformernetworkforreadingabankcheckisalsodescribed.Itusesconvolutionalneuralnetworkcharacterrecognizerscombinedwithglobaltrainingtechniquestoproviderecordaccuracyonbusinessandpersonalchecks.Itisdeployedcommerciallyandreadsseveralmillionchecksperday.Keywords—Convolutionalneuralnetworks,documentrecog-nition,finitestatetransducers,gradient-basedlearning,graphtransformernetworks,machinelearning,neuralnetworks,opticalcharacterrecognition(OCR).NOMENCLATUREGTGraphtransformer.GTNGraphtransformernetwork.HMMHiddenMarkovmodel.HOSHeuristicoversegmentation.K-NNK-nearestneighbor.ManuscriptreceivedNovember1,1997;revisedApril17,1998.Y.LeCun,L.Bottou,andP.HaffnerarewiththeSpeechandImageProcessingServicesResearchLaboratory,AT&TLabs-Research,RedBank,NJ07701USA.Y.BengioiswiththeD´epartementd’InformatiqueetdeRechercheOp´erationelle,Universit´edeMontr´eal,Montr´eal,Qu´ebecH3C3J7Canada.PublisherItemIdentifierS0018-9219(98)07863-3.NNNeuralnetwork.OCROpticalcharacterrecognition.PCAPrincipalcomponentanalysis.RBFRadialbasisfunction.RS-SVMReduced-setsupportvectormethod.SDNNSpacedisplacementneuralnetwork.SVMSupportvectormethod.TDNNTimedelayneuralnetwork.V-SVMVirtualsupportvectormethod.I.INTRODUCTIONOverthelastseveralyears,machinelearningtechniques,particularlywhenappliedtoNN’s,haveplayedanincreas-inglyimportantroleinthedesignofpatternrecognitionsystems.Infact,itcouldbearguedthattheavailabilityoflearningtechniqueshasbeenacrucialfactorintherecentsuccessofpatternrecognitionapplicationssuchascontinuousspeechrecognitionandhandwritingrecognition.Themainmessageofthispaperisthatbetterpatternrecognitionsystemscanbebuiltbyrelyingmoreonauto-maticlearningandlessonhand-designedheuristics.Thisismadepossiblebyrecentprogressinmachinelearningandcomputertechnology.Usingcharacterrecognitionasacasestudy,weshowthathand-craftedfeatureextractioncanbeadvantageouslyreplacedbycarefullydesignedlearningmachinesthatoperatedirectlyonpixelimages.Usingdocumentunderstandingasacasestudy,weshowthatthetraditionalwayofbuildingrecognitionsystemsbymanuallyintegratingindividuallydesignedmodulescanbereplacedbyaunifiedandwell-principleddesignparadigm,calledGTN’s,whichallowstrainingallthemodulestooptimizeaglobalperformancecriterion.Sincetheearlydaysofpatternrecognitionithasbeenknownthatthevariabilityandrichnessofnaturaldata,beitspeech,glyphs,orothertypesofpatterns,makeitalmostimpossibletobuildanaccuraterecognitionsystementirelybyhand.Consequently,mostpatternrecognitionsystemsarebuiltusingacombinationofautomaticlearningtechniquesandhand-craftedalgorithms.Theusualmethod0018–9219/98$10.001998IEEE2278PROCEEDINGSOFTHEIEEE,VOL.86,NO.11,NOVEMBER1998Fig.1.Traditionalpatternrecognitionisperformedwithtwomodules:afixedfeatureextractorandatrainableclassifier.ofrecognizingindividualpatternsconsistsindividingthesystemintotwomainmodulesshowninFig.1.Thefirstmodule,calledthefeatureextractor,transformstheinputpatternssothattheycanberepresentedbylow-dimensionalvectorsorshortstringsofsymbolsthat:1)canbeeasilymatchedorcomparedand2)arerelativelyinvariantwithrespecttotransformationsanddistortionsoftheinputpat-ternsthatdonotchangetheirnature.Thefeatureextractorcontainsmostofthepriorknowledgeandisratherspecifictothetask.Itisalsothefocusofmostofthedesigneffort,becauseitisoftenentirelyhandcrafted.Theclassifier,ontheotherhand,isoftengeneralpurposeandtrainable.Oneofthemainproblemswiththisapproachisthattherecognitionaccuracyislargelydeterminedbytheabilityofthedesignertocomeupwithanappropriatesetoffeatures.Thisturnsouttobeadauntingtaskwhich,unfortunately,mustberedoneforeachnewproblem.Alargeamountofthepatternrecognitionliteratureisdevotedtodescribingandcomparingtherelativemeritsofdifferentfeaturesetsforparticulartasks.Historically,theneedforappropriatefeatureextractorswasduetothefactthatthelearningtechniquesusedbytheclassifierswerelimited