1989-Cybenko-Approximation-by-superpositions-of-a-

Math.ControlSignalsSystems(1989)2:303-314MathematicsofControl,Signals,andSystems91989Springer-VerlagNewYorkInc.ApproximationbySuperpositionsofaSigmoidalFunction*G.CybenkotAbstr,,ct.Inthispaperwedemonstratethatfinitelinearcombinationsofcom-positionsofafixed,univariatefunctionandasetofaffinefunctionalscanuniformlyapproximateanycontinuousfunctionofnrealvariableswithsupportintheunithypercube;onlymildconditionsareimposedontheunivariatefunction.Ourresultssettleanopenquestionaboutrepresentabilityintheclassofsinglebiddenlayerneuralnetworks.Inparticular,weshowthatarbitrarydecisionregionscanbearbitrarilywellapproximatedbycontinuousfeedforwardneuralnetworkswithonlyasingleinternal,hiddenlayerandanycontinuoussigmoidalnonlinearity.Thepaperdiscussesapproximationpropertiesofotherpossibletypesofnonlinearitiesthatmightbeimplementedbyartificialneuralnetworks.Keywords.Neuralnetworks,Approximation,Completeness.1.IntroductionAnumberofdiverseapplicationareasareconcernedwiththerepresentationofgeneralfunctionsofann-dimensionalrealvariable,x9R,byfinitelinearcombina-tionsoftheformN+or),(1)j=lwhereyj9Randctj,09I~arefixed.(yristhetransposeofysothatyrxistheinnerproductofyandx.)Heretheunivariatefunctiontrdependsheavilyonthecontextoftheapplication.Ourmajorconcerniswithso-calledsigmoidala's:a(t)__,flast--,+~,ast--*-~.Such.functionsarisenaturallyinneuralnetworktheoryastheactivationfunctionofaneuralnode(orunitasisbecomingthepreferredterm)ILl],IRHM].Themainresultofthispaperisademonstrationofthefactthatsumsoftheform(1)aredenseinthespaceofcontinuousfunctionsontheunitcubeiftrisanycontinuoussigmoidal*Datereceived:October21,1988.Daterevised:February17,1989.ThisresearchwassupportedinpartbyNSFGrantDCR-8619103,ONRContractN000-86-G-0202andDOEGrantDE-FG02-85ER25001.tCenterforSupercomputingResearchandDevelopmentandDepartmentofElectricalandComputerEngineering,UniversityofIllinois,Urbana,Illinois61801,U.S.A.303304G.Cybcnkofunction.Thiscaseisdiscussedinthemostdetail,butwestategeneralconditionsonotherpossiblea'sthatguaranteesimilarresults.Thepossibleuseofartificialneuralnetworksinsignalprocessingandcontrolapplicationshasgeneratedconsiderableattentionrecently[B],I'G].Looselyspeak-ing,anartificialneuralnetworkisformedfromcompositionsandsuperpositionsofasingle,simplenonlinearactivationorresponsefunction.Accordingly,theoutputofthenetworkisthevalueofthefunctionthatresultsfromthatparticularcompositionandsuperpositionofthenonlinearities.Inparticular,thesimplestnontrivialclassofnetworksarethosewithoneinternallayerandtheyimplementtheclassoffunctionsgivenby(1).Inapplicationssuchaspatternclassification[L1]andnonlinearpredictionoftimeseriesiLF],forexample,thegoalistoselectthecompositionsandsuperpositionsappropriatelysothatdesirednetworkresponses(meanttoimplementaclassifyingfunctionornonlinearpredictor,respectively)areachieved.Thisleadstotheproblemofidentifyingtheclassesoffunctionsthatcanbeeffectivelyrealizedbyartificialneuralnetworks.Similarproblemsarequitefamiliarandwellstudiedincircuittheoryandfilterdesignwheresimplenonlineardevicesareusedtosynthesizeorapproximatedesiredtransferfunctions.Thus,forexample,afundamentalresultindigitalsignalprocessingisthefactthatdigitalfiltersmadefromunitdelaysandconstantmultiplierscanapproximateanycontinuoustransferfunctionarbitrarilywell.Inthissense,themainresultofthispaperdemonstratesthatnetworkswithonlyoneinternallayerandanarbitrarycontinuoussigmoidalnonlinearityenjoythesamekindofuniversality.Requiringthatfinitelinearcombinationssuchas(1)exactlyrepresentagivencontinuousfunctionisaskingfortoomuch.Inawell-knownresolutionofHilbert's13thproblem,Kolmogorovshowedthatallcontinuousfunctionsofnvariableshaveanexactrepresentationintermsoffinitesuperpositionsandcompositionsofasmallnumberoffunctionsofonevariable[K-I,[L2].However,theKolmogorovrepresen-tationinvolvesdifferentnonlinearfunctions.Theissueofexactrepresentabilityhasbeenfurtherexploredin[DS]inthecontextofprojectionpursuitmethodsforstatisticaldataanalysisIH].Ourinterestisinfinitelinearcombinationsinvolvingthesameunivariatefunc-tion.Moreover,wesettleforapproximationsasopposedtoexactrepresentations.Itiseasytoseethatinthislight,(I)merelygeneralizesapproximationsbyfiniteFourierseries.Themathematicaltoolsfordemonstratingsuchcompletenessprop-ertiestypicallyfallintotwocategories:thosethatinvolvealgebrasoffunctions(leadingtoStone-Weierstrassargumentsl-A])andthosethatinvolvetranslationinvariantsubspaces(leadingtoTauberiantheorems[R2]).Wegiveexamplesofeachofthesecasesinthispaper.Ourmainresultsettlesalong-standingquestionabouttheexactclassofdecisionregionsthatcontinuousvalued,singlehiddenlayerneuralnetworkscanimplement.Somerecentdiscussionsofthisquestionarein[HL1],[HL2],[MSJ],and[WL]whileIN]containsoneoftheearlyrigorousanalyses.InIN]NilssonshowedthatanysetofMpointscanbepartitionedintotwoarbitrarysubsetsbyanetworkwithoneinternallayer.TherehasbeengrowingevidencethroughexamplesandspecialApproximationbySuperpositionsofaSigmoidalFunction305casesthatsuchnetworkscanimplementmoregeneraldecisionregionsbu