神经网络概述

1、NeuralNetworkIntroduction1.ObjectivesAsyoureadthesewordsyouareusingacomplexbiologicalneuralnetwork.Youhaveahighlyinterconnectedsetofsome1011neuronstofacilitateyourreading,breathing,motionandthinking.Eachofyourbiologicalneurons，arichassemblyoftissueandchemistry,hasthecomplexity,ifnotthespeed,ofamicroprocessor.Someofyourneuralstructurewaswithyouatbirth.Otherpartshavebeenestablishedbyexperience.Scientistshaveonlyjustbeguntounderstandhowbiologicalneuralnetworksoperate.Itisgenerallyunderstoodthatall。

2、biologicalneuralfunctions,includingmemory,arestoredintheneuronsandintheconnectionsbetweenthem.Learningisviewedastheestablishmentofnewconnectionsbetweenneuronsorthemodificationofexistingconnections.Thisleadstothefollowingquestion:Althoughwehaveonlyarudimentaryunderstandingofbiologicalneuralnetworks,isitpossibletoconstructasmallsetofsimpleartificial“neurons”andperhapstrainthemtoserveausefulfunction?Theansweris“yes.”Thisbook,then,isaboutartificialneuralnetworks.Theneuronsthatweconsiderherearenotbio。

3、logical.Theyareextremelysimpleabstractionsofbiologicalneurons,realizedaselementsinaprogramorperhapsascircuitsmadeofsilicon.Networksoftheseartificialneuronsdonothaveafractionofthepowerofthehumanbrain,buttheycanbetrainedtoperformusefulfunctions.Thisbookisaboutsuchneurons,thenetworksthatcontainthemandtheirtraining.2.HistoryThehistoryofartificialneuralnetworksisfilledwithcolorful,creativeindividualsfrommanydifferentfields,manyofwhomstruggledfordecadestodevelopconceptsthatwenowtakeforgranted.Thishist。

4、oryhasbeendocumentedbyvariousauthors.OneparticularlyinterestingbookisNeurocomputing:FoundationsofResearchbyJohnAndersonandEdwardRosenfeld.Theyhavecollectedandeditedasetofsome43papersofspecialhistoricalinterest.Eachpaperisprecededbyanintroductionthatputsthepaperinhistoricalperspective.Historiesofsomeofthemainneuralnetworkcontributorsareincludedatthebeginningofvariouschaptersthroughoutthistextandwillnotberepeatedhere.However,itseemsappropriatetogiveabriefoverview,asampleofthemajordevelopments.Atle。

5、asttwoingredientsarenecessaryfortheadvancementofatechnology:conceptandimplementation.First,onemusthaveaconcept,awayofthinkingaboutatopic,someviewofitthatgivesclaritynottherebefore.Thismayinvolveasimpleidea,oritmaybemorespecificandincludeamathematicaldescription.Toillustratethispoint,considerthehistoryoftheheart.Itwasthoughttobe,atvarioustimes,thecenterofthesoulorasourceofheat.Inthe17thcenturymedicalpractitionersfinallybegantoviewtheheartasapump,andtheydesignedexperimentstostudyitspumpingaction.T。

6、heseexperimentsrevolutionizedourviewofthecirculatorysystem.Withoutthepumpconcept,anunderstandingoftheheartwasoutofgrasp.Conceptsandtheiraccompanyingmathematicsarenotsufficientforatechnologytomatureunlessthereissomewaytoimplementthesystem.Forinstance,themathematicsnecessaryforthereconstructionofimagesfromcomputer-aidedtopography(CAT)scanswasknownmanyyearsbeforetheavailabilityofhigh-speedcomputersandefficientalgorithmsfinallymadeitpracticaltoimplementausefulCATsystem.Thehistoryofneuralnetworkshasp。

7、rogressedthroughbothconceptualinnovationsandimplementationdevelopments.Theseadvancements,however,seemtohaveoccurredinfitsandstartsratherthanbysteadyevolution.Someofthebackgroundworkforthefieldofneuralnetworksoccurredinthelate19thandearly20thcenturies.Thisconsistedprimarilyofinterdisciplinaryworkinphysics,psychologyandneurophysiologybysuchscientistsasHermannvonHelmholtz,ErnstMuchandIvanPavlov.Thisearlyworkemphasizedgeneraltheoriesoflearning,vision,conditioning,etc.,anddidnotincludespecificmathema。

8、ticalmodelsofneuronoperation.Themodernviewofneuralnetworksbeganinthe1940swiththeworkofWarrenMcCullochandWalterPitts[McPi43],whoshowedthatnetworksofartificialneuronscould,inprinciple,computeanyarithmeticorlogicalfunction.Theirworkisoftenacknowledgedastheoriginoftheneuralnetworkfield.McCullochandPittswerefollowedbyDonaldHebb[Hebb49],whoproposedthatclassicalconditioning(asdiscoveredbyPavlov)ispresentbecauseofthepropertiesofindividualneurons.Heproposedamechanismforlearninginbiologicalneurons.Thefirs。

9、tpracticalapplicationofartificialneuralnetworkscameinthelate1950s,withtheinventionoftheperceptionnetworkandassociatedlearningrulebyFrankRosenblatt[Rose58].Rosenblattandhiscolleaguesbuiltaperceptionnetworkanddemonstrateditsabilitytoperformpatternrecognition.Thisearlysuccessgeneratedagreatdealofinterestinneuralnetworkresearch.Unfortunately,itwaslatershownthatthebasicperceptionnetworkcouldsolveonlyalimitedclassofproblems.(SeeChapter4formoreonRosenblattandtheperceptionlearningrule.)Ataboutthesametim。

10、e,BernardWidrowandTedHoff[WiHo60]introducedanewlearningalgorithmandusedittotrainadaptivelinearneuralnetworks,whichweresimilarinstructureandcapabilitytoRosenblatt’sperception.TheWidrowHofflearningruleisstillinusetoday.(SeeChapter10formoreonWidrow-Hofflearning.)Unfortunately,bothRosenblatt'sandWidrow'snetworkssufferedfromthesameinherentlimitations,whichwerewidelypublicizedinabookbyMarvinMinskyandSeymourPapert[MiPa69].RosenblattandWidrowwereawareoftheselimitationsandproposednewnetworksthatwouldoverc。