_信息论之父Shannon的经典之作：《通信的数学理论》

ReprintedwithcorrectionsfromTheBellSystemTechnicalJournal,Vol.27,pp.379–423,623–656,July,October,1948.AMathematicalTheoryofCommunicationByC.E.SHANNONINTRODUCTIONTHErecentdevelopmentofvariousmethodsofmodulationsuchasPCMandPPMwhichexchangebandwidthforsignal-to-noiseratiohasintensifiedtheinterestinageneraltheoryofcommunication.AbasisforsuchatheoryiscontainedintheimportantpapersofNyquist1andHartley2onthissubject.Inthepresentpaperwewillextendthetheorytoincludeanumberofnewfactors,inparticulartheeffectofnoiseinthechannel,andthesavingspossibleduetothestatisticalstructureoftheoriginalmessageandduetothenatureofthefinaldestinationoftheinformation.Thefundamentalproblemofcommunicationisthatofreproducingatonepointeitherexactlyorap-proximatelyamessageselectedatanotherpoint.Frequentlythemessageshavemeaning;thatistheyrefertoorarecorrelatedaccordingtosomesystemwithcertainphysicalorconceptualentities.Thesesemanticaspectsofcommunicationareirrelevanttotheengineeringproblem.Thesignificantaspectisthattheactualmessageisoneselectedfromasetofpossiblemessages.Thesystemmustbedesignedtooperateforeachpossibleselection,notjusttheonewhichwillactuallybechosensincethisisunknownatthetimeofdesign.Ifthenumberofmessagesinthesetisfinitethenthisnumberoranymonotonicfunctionofthisnumbercanberegardedasameasureoftheinformationproducedwhenonemessageischosenfromtheset,allchoicesbeingequallylikely.AswaspointedoutbyHartleythemostnaturalchoiceisthelogarithmicfunction.Althoughthisdefinitionmustbegeneralizedconsiderablywhenweconsidertheinfluenceofthestatisticsofthemessageandwhenwehaveacontinuousrangeofmessages,wewillinallcasesuseanessentiallylogarithmicmeasure.Thelogarithmicmeasureismoreconvenientforvariousreasons:1.Itispracticallymoreuseful.Parametersofengineeringimportancesuchastime,bandwidth,numberofrelays,etc.,tendtovarylinearlywiththelogarithmofthenumberofpossibilities.Forexample,addingonerelaytoagroupdoublesthenumberofpossiblestatesoftherelays.Itadds1tothebase2logarithmofthisnumber.Doublingthetimeroughlysquaresthenumberofpossiblemessages,ordoublesthelogarithm,etc.2.Itisnearertoourintuitivefeelingastothepropermeasure.Thisiscloselyrelatedto(1)sincewein-tuitivelymeasuresentitiesbylinearcomparisonwithcommonstandards.Onefeels,forexample,thattwopunchedcardsshouldhavetwicethecapacityofoneforinformationstorage,andtwoidenticalchannelstwicethecapacityofonefortransmittinginformation.3.Itismathematicallymoresuitable.Manyofthelimitingoperationsaresimpleintermsoftheloga-rithmbutwouldrequireclumsyrestatementintermsofthenumberofpossibilities.Thechoiceofalogarithmicbasecorrespondstothechoiceofaunitformeasuringinformation.Ifthebase2isusedtheresultingunitsmaybecalledbinarydigits,ormorebrieflybits,awordsuggestedbyJ.W.Tukey.Adevicewithtwostablepositions,suchasarelayoraflip-flopcircuit,canstoreonebitofinformation.NsuchdevicescanstoreNbits,sincethetotalnumberofpossiblestatesis2Nandlog22N=N.Ifthebase10isusedtheunitsmaybecalleddecimaldigits.Sincelog2M=log10M=log102=3:32log10M;1Nyquist,H.,“CertainFactorsAffectingTelegraphSpeed,”BellSystemTechnicalJournal,April1924,p.324;“CertainTopicsinTelegraphTransmissionTheory,”A.I.E.E.Trans.,v.47,April1928,p.617.2Hartley,R.V.L.,“TransmissionofInformation,”BellSystemTechnicalJournal,July1928,p.535.1INFORMATIONSOURCEMESSAGETRANSMITTERSIGNALRECEIVEDSIGNALRECEIVERMESSAGEDESTINATIONNOISESOURCEFig.1—Schematicdiagramofageneralcommunicationsystem.adecimaldigitisabout313bits.Adigitwheelonadeskcomputingmachinehastenstablepositionsandthereforehasastoragecapacityofonedecimaldigit.Inanalyticalworkwhereintegrationanddifferentiationareinvolvedthebaseeissometimesuseful.Theresultingunitsofinformationwillbecallednaturalunits.Changefromthebaseatobasebmerelyrequiresmultiplicationbylogba.ByacommunicationsystemwewillmeanasystemofthetypeindicatedschematicallyinFig.1.Itconsistsofessentiallyfiveparts:1.Aninformationsourcewhichproducesamessageorsequenceofmessagestobecommunicatedtothereceivingterminal.Themessagemaybeofvarioustypes:(a)Asequenceoflettersasinatelegraphofteletypesystem;(b)Asinglefunctionoftimef(t)asinradioortelephony;(c)Afunctionoftimeandothervariablesasinblackandwhitetelevision—herethemessagemaybethoughtofasafunctionf(x;y;t)oftwospacecoordinatesandtime,thelightintensityatpoint(x;y)andtimetonapickuptubeplate;(d)Twoormorefunctionsoftime,sayf(t),g(t),h(t)—thisisthecasein“three-dimensional”soundtransmissionorifthesystemisintendedtoserviceseveralindividualchannelsinmultiplex;(e)Severalfunctionsofseveralvariables—incolortelevisionthemessageconsistsofthreefunctionsf(x;y;t),g(x;y;t),h(x;y;t)definedinathree-dimensionalcontinuum—wemayalsothinkofthesethreefunctionsascomponentsofavectorfielddefinedintheregion—similarly,severalblackandwhitetelevisionsourceswouldproduce“messages”consistingofanumberoffunctionsofthreevariables;(f)Variouscombinationsalsooccur,forexampleintelevisionwithanassociatedaudiochannel.2.Atransmitterwhichoperatesonthemessageinsomewaytoproduceasignalsuitablefortrans-missionoverthechannel.Intelephonythisoperationconsistsmerelyofchangingsoundpressureintoapropor