X, N1 xi X, Y, N2 yj Y. (xi yj). , N1N2 .
( ), , (xi yj) :
log2(N1N2)= log2 N1+ log2 N2
.
.
1. .
2. N :
log N = -log 1/N = -log p
p = 1/N .
3. , . , . N :
log N = - log 1/N= - log p
=1/N N .
, , . 20 .
{X,P(x)}. . , . , .. .
x1, x2, xi xn Σ Pi = 1
{X,P(x)} =
P1, P2,PiPn
( ), .
xi, Pi. , , I(xi). . , .
, , . , 1/P(xi) : .
.
1. xi X I(xi), :
I(xi)= log 1/P(xi)=-log P(xi) (1)
. :
2, .. log2P(xi) []
e, .. logeP(xi)=lnP(xi) [] natural digit
10, ... lg10P(xi) []
|
|
, .
, (1) {X,P(x)} , I(x1), I(x2),, I(xn).
, :
M [I(xi)]=Σ Pi I(xi)=-Σ P ilogPi (2)
(2) , . :
I[X] = -Σ Pi log2Pi []
, , . , , . Pi(xi)=1/N=1/2n, N=2n, :
-Σ Pi log2Pi = N1/Nlog2 1/(1/N) = log2N = n
, () , , .
, , , . , .
. H(X) I(xi), {X,P(x)} :
H(X) = Σ I(xi) P(xi) = -Σ P(xi) log2P(xi) /
. I(xi).
:
1. , , . :
0 ≤ P(xi) ≤ 1
2. .
:
H(X) = -P1 log2Pi - Σ Pi log2Pi = 0, P1log2Pi=0 (2)
P1=1, 0. (2) . , Pi→0.
lim(-Pi log2Pi)=lim P log21/Pi= lim log2ß/ ßi
Pi→0 Pi→0 ß → ∞
, :
lim (1/ ß) ln2 = 0
. , , , .
3. .
p q, q=(1-p), :
H=-p log2(p-q) log2q
p q. , q=(1-p), :
H=-p log2(p-(1-p)) log2(1-p)
p 0 1 . 2.
4. , .
, .
|
|
:
I(xi) = -log2P(xi)
:
H(x) = M [ I(xi) ] = -Σ Pi log2Pi
, :
0 ≤ H(X) ≤ log2N
, . :
H(x) = - Σ Pi log2Pi
, , .. (xjxk):
P(xj,xk) = P(xj/xk) P(xk) = P(xj) P(xk)
, . .
Hmax(x) H(x) :
H(x) ≤ Hmax(x)
.
, , :
H(x) = Σ P(xi) H(x/xi), /.
H(x/xi) = - Σ P(xj/xi) log2P(xj/xi)
, xi. :
H (x) ≤ H(x)
, .
, , .
, - , (K-1) . _ , .
, (1.4)
H(X) - ;
H(Y) - Y;
H(Y/X) - Y ;
H(X/Y) - X Y.
.
X Y
, (1.5)
P(xi/yj) - xi X , Y yj ( ).
X yj Y
. (1.6)