您好,欢迎访问三七文档
当前位置:首页 > 电子/通信 > 数据通信与网络 > 第2章-熵和平均互信息
熵和平均互信息一、熵如果任何时刻信源发出的消息都是单一符号,而这些符号取值于有限或可数的集合,该信源为单符号离散信源——离散型随机变量1、单符号离散信源定义符号X取值于集合}x,,x,x{N21第2章熵和平均互信息熵和平均互信息)x(P)x(P)x(Pxxx)X(PXN21N211)x(PN,,2,1i,1)x(P0,N1iii且其中表示6/16/16/1621)X(PX例1熵和平均互信息例264/164/164/164/116/18/14/12/1xxxxxxxx)X(PX87654321例316/18/14/12/1xxxx)X(PX4321熵和平均互信息2、自信息消息xi的概率P(xi)对数的负值为该消息的自信息,用I(xi)表示。定义表示)x(Plog)x(Iiai熵和平均互信息单位由对数的底a决定——当a=2时为比特(bit,binaryunit);a=e时为奈特(nat,natureunit);a=10时为哈特(Hart,Hartley)。)x(Plog)x(Iii以比特(bit)为单位的自信息可记为3、熵信源各消息自信息的数学期望为该信源的熵,也叫无条件熵,用H(X)表示。定义熵和平均互信息表示N1iiiN1iiii)x(Plog)x(P)x(I)x(P)]x(I[E)X(HH(X)反映信源每发出一条消息所提供的平均信息,不反映信源发出某条特定消息的信息H(X)表示信源的平均不确定性一般情况下,H(X)不等于每接收一条消息所获得的平均信息。熵和平均互信息4、熵的主要性质和最大熵定理①非负性0)X(H0)x(Plog)x(I,1)x(P0iii0)x(I)x(P)X(HN1iii熵和平均互信息严格上凸性的描述——设函数f(x)对任一小于1的正数α及定义域中任意两个值x1、x2,如果)x(f)1()x(f]x)1(x[f2121则称函数f(x)为严格上凸函数。)x(Plog)x(P)1()x(Plog)x(P)]x(P)1()x(Plog[)]x(P)1()x(P[i2n1ii2i1n1ii1i2i1n1ii2i1②严格上凸性熵和平均互信息证明利用不等式1xxln)x(P)x(P)1()x(Plog)x(P)x(Plog)x(P)]x(P)1()x(Plog[)x(Pi1i2i1n1ii1i1n1ii1i2i1n1ii1n1ii1i2i1]1)x(P)x(P)1()[x(Pelog0])x(P)1()1)(x(P[elogn1in1ii2i1熵和平均互信息,等号不成立)x(P)x(Pi2i1)x(Plog)x(P)]x(P)1()x(Plog[)x(Pi1n1ii1i2i1n1ii1)x(Plog)x(P)]x(P)1()x(Plog[)x(Pi1n1ii1i2i1n1ii1即)]x(P)1()x(Plog[)x(P)1(i2i1n1ii2同理,)x(Plog)x(P)1(i2n1ii2熵和平均互信息)]x(P)1()x(Plog[)]x(P)1()]x(P)1()x(Plog[)x(P)]x(P)1()x(Plog[)]x(P)1()x(P[i2i1n1ii2i2i1n1ii1i2i1n1ii2i1)x(Plog)x(P)1()x(Plog)x(Pi2n1ii2i1n1ii1熵和平均互信息限制下的条件极值在1)x(P)X(Hn1ii③最大熵定理nlog)X(Hn,,2,1k0]}1)x(P[)X(H{)x(Pn1iik令等概率信源具有最大熵,最大熵H(X)max=logn熵和平均互信息n,,2,1k0)]x(Ploge[log]}1)x(P[)x(Plog)x(P{)x(Pkn1iin1iiikn,,2,1kelog)x(Plogkn,,2,1ke2)x(Pk1e2n)x(Pn1kkn,,2,1kn1e2)x(Pk熵和平均互信息nlogn1logn1)x(Plog)x(P)X(Hn1in1iiimaxnlog)X(H例1求信源的熵64/164/164/164/116/18/14/12/1xxxxxxxx)X(PX87654321熵和平均互信息641log6414161log16181log8141log4121log21)x(Plog)x(P)X(H81iii)bit(216516483422164log16116log1618log814log412log21熵和平均互信息例216/18/14/12/1xxxx)X(PX4321求信源的熵81log8141log4121log21)x(Plog)x(P)X(H1iiinn21log21161log161熵和平均互信息nn2log2116log1618log814log412log21n2n164834221)]X(H21)X(H[2)X(H)]2n164834221(21)2n164834221[(2nn熵和平均互信息]21161814121[2n)bit(2例3p1p10)X(PX求信源的熵及p-H(p)曲线熵和平均互信息)p(H)p1log()p1(plogp)i(Plog)i(P)X(H10i当p=0时,H(p)=0p=0.25时,H(p)=0.811(bit)p=0.75时,H(p)=0.811(bit)p=1时,H(p)=0p=0.5时,H(p)=1(bit)熵和平均互信息00.51H(p)1p熵和平均互信息如果任何时刻信源发出的消息都是有限或可数的符号序列,而这些符号都取值于同一个有限或可数的集合,则该信源为多符号离散信源。5、多符号离散信源定义多符号离散信源发出的符号序列记为N21XXX熵和平均互信息序列中任一符号都取值于同一集合N,,2,1k},x,,x,x{Xn21k)a(P)a(P)a(Paaa)XXX(PXXXNNn21n21N21N21表示)xxx(P)xxx(P)xxx(Pxxxxxxxxxnnn211111nnn211111熵和平均互信息n,,2,1i,,i,in,,2,1ixxxaN21NiiiiN21其中,1)xxx/x(P)x/x(P)x(P)xxx(P)a(P01N21N121N21iiiiiiiiiii1)xxx(P)a(Pn1in1in1iiiin1ii12NN21N且熵和平均互信息4/118/1018/13/118/1018/136/7xxxxxxxxxxxxxxxxxx)XX(PXX3323133222123121112121例16、联合自信息消息xi1xi2…xiN的概率P(xi1xi2…xiN)对数的负值为该消息的自信息,也称联合自信息,用I(xi1xi2…xiN)表示。定义熵和平均互信息表示)xxx(Plog)xxx(IN21N21iiiiii)xx/x(Plog)xx/x(I)x/x(Plog)x/x(I1N1N1N1N1212iiiiiiiiii记称为条件自信息)xx/x(Plog)x/x(Plog)x(Plog1N1N121iiiiii熵和平均互信息信息的链式法则)xx/x(I)x/x(I)x(I)xxx(I1N1N121N21iiiiiiiii7、联合熵信源各消息联合自信息的数学期望为该信源的熵,也叫联合熵,用H(X1X2…XN)表示。定义熵和平均互信息表示)]xxx(I[E)XXX(HN21iiiN21n1in1in1iiiiiiin1in1in1iiiiiii12NN21N2112NN21N21)xxx(Plog)xxx(P)xxx(I)xxx(Pn1in1in1iiiii12N1N21)x(Plog)xxx(P熵和平均互信息n1in1in1iiiiii12N12N21)x/x(Plog)xxx(Pn1in1in1iiiiiiii12N1N21NN21)xxx/x(Plog)xxx(Pn1in1iiiiin1iii121221111)x/x(Plog)xx(P)x(Plog)x(P熵和平均互信息n1in1in1iiiiiiii12N1N21NN21)xxx/x(Plog)xxx(Pn1in1in1iiiiiiii1N21Nn1in1iiiii1212N1N21NN21121221)xxx/x(Plog)xxx(P)XXX/X(H)x/x(Plog)xx(P)X/X(H记熵和平均互信息称为条件熵熵的链式法则)XXX/X(H)X/X(H)X(H)XXX(H1N21N121N21例1}36/11,9/4,4/1{)X(P1}x,x,x{X,X32121)X/X(P12熵和平均互信息x1x2x3x17/92/90x21/83/41/8x302/119/11xi2xi14/118/1018/13/118/1018/136/7xxxxxxxxxxxxxxxxxx)XX(PXX3323133222123121112121求信源的联合熵熵和平均互信息41log41181log181181log18131log31181log181181log181367log367)xx(Plog)xx(P)XX(H211221ii31i31iii21)bit(412.23611log361194log9441log41)x(Plog)x(P)X(H31iii1111熵和平均互信息)bit(542.131i31iiiiii31i31iiiii121212121121221)x/x(Plog)x/x(P)x(P)x/x(Plog)xx(P)X/X(H119log1193611112log112361181log819443log439481log819492log924197log9741熵和平均互信息)bit(870.0119log41112log18181log18143log3181log18192log18197log367)bit(412.2870.0542.1)X/X(H)X(H)XX(H12121熵和平均互信息8、熵的独立界N1kkN21)X(H)XXX(Hn1in1iiiiiin1in1iiiin1in1iiiii212121222112221121221)x/x(P)x(Plog)xx(P)x(Plog)xx(P)x/x(Plog)xx(P)X(H)X/X(H熵和平均互信息0)]xx(P)x(P)x(P[elog]1)x/x(P)x(P)[
本文标题:第2章-熵和平均互信息
链接地址:https://www.777doc.com/doc-2188969 .html