basic definitions and facts
play

Basic Definitions and Facts Iftach Haitner Tel Aviv University. - PowerPoint PPT Presentation

Nov 19, 2023 •161 likes •1.34k views

Application of Information Theory, Lecture 1 Basic Definitions and Facts Iftach Haitner Tel Aviv University. October 28, 2014 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 1 / 12 The entropy function X

  1. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  2. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  3. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  4. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  5. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  6. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  7. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 ◮ h ( p ) := H ( p , 1 − p ) is continuous Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  8. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 ◮ h ( p ) := H ( p , 1 − p ) is continuous Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  9. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 ◮ h ( p ) := H ( p , 1 − p ) is continuous Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  10. Axiomatic derivation of the entropy function Any other choices for defining entropy? Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  11. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  12. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  13. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Not hard to prove that Shannon’s entropy function satisfies above axioms, proving this is the only such function is more challenging. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  14. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Not hard to prove that Shannon’s entropy function satisfies above axioms, proving this is the only such function is more challenging. Let H be a symmetric function that satisfying the above axioms. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  15. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Not hard to prove that Shannon’s entropy function satisfies above axioms, proving this is the only such function is more challenging. Let H be a symmetric function that satisfying the above axioms. We prove (assuming additional axiom) that H is the Shannon function. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  16. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  17. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  18. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  19. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  20. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  21. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  22. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  23. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  24. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Hence, k − 1 H ( p 1 , . . . , p k ) = H ( S k − 1 , p k S i h ( p i / S k � ) + ) S k S k S k S k S k S i / S k i = 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  25. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Hence, k − 1 k H ( p 1 , . . . , p k ) = H ( S k − 1 , p k S i h ( p i / S k ) = 1 S i h ( p i � � ) + ) (2) S k S k S k S k S k S i / S k S k S i i = 2 i = 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  26. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Hence, k − 1 k H ( p 1 , . . . , p k ) = H ( S k − 1 , p k S i h ( p i / S k ) = 1 S i h ( p i � � ) + ) (2) S k S k S k S k S k S i / S k S k S i i = 2 i = 2 Claim follows by combining the above equations. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  27. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  28. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  29. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  30. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  31. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  32. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  33. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  34. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) ⇒ f ( 3 n ) = nf ( 3 ) . = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  35. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) ⇒ f ( 3 n ) = nf ( 3 ) . = ◮ f ( mn ) = f ( m ) + f ( n ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  36. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) ⇒ f ( 3 n ) = nf ( 3 ) . = ◮ f ( mn ) = f ( m ) + f ( n ) ⇒ f ( m k ) = kf ( m ) = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  37. f ( m ) = log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  38. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  39. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  40. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  41. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  42. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  43. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. ⌊ n log 3 ⌋ < f ( 3 ) < ⌊ n log 3 ⌋ + 1 = ⇒ for any n ∈ N n n Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  44. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. ⌊ n log 3 ⌋ < f ( 3 ) < ⌊ n log 3 ⌋ + 1 = ⇒ for any n ∈ N n n = ⇒ f ( 3 ) = log 3. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  45. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. ⌊ n log 3 ⌋ < f ( 3 ) < ⌊ n log 3 ⌋ + 1 = ⇒ for any n ∈ N n n = ⇒ f ( 3 ) = log 3. ◮ Proof extends to any integer (not only 3) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  46. H ( p , q ) = − p log p − q log q Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  47. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  48. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  49. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  50. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  51. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  52. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) = p ( logm − logk ) + q ( log m − log ( m − k )) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  53. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) = p ( logm − logk ) + q ( log m − log ( m − k )) − p log m k − q log m − k = = − p log p − q log q m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  54. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) = p ( logm − logk ) + q ( log m − log ( m − k )) − p log m k − q log m − k = = − p log p − q log q m ◮ By continuity axiom, holds for every p , q . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  55. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  56. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  57. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  58. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  59. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  60. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  61. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  62. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 − p 1 log k 1 m − p 2 log k 2 k 3 = m − p 3 m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  63. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 − p 1 log k 1 m − p 2 log k 2 k 3 = m − p 3 m = − p 1 log p 1 − p 2 log p 2 − p 3 log p 3 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  64. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 − p 1 log k 1 m − p 2 log k 2 k 3 = m − p 3 m = − p 1 log p 1 − p 2 log p 2 − p 3 log p 3 ◮ By continuity axiom, holds for every p 1 , p 2 , p 3 . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  65. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  66. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  67. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  68. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  69. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  70. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  71. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  72. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  73. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . ◮ log ( x ) is (strictly) concave for x > 0, since its second derivative ( − 1 x 2 ) is always negative. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  74. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . ◮ log ( x ) is (strictly) concave for x > 0, since its second derivative ( − 1 x 2 ) is always negative. ◮ Hence, H ( p 1 , . . . , p m ) = � p i ≤ log � i p i log 1 i p i 1 p i = log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  75. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . ◮ log ( x ) is (strictly) concave for x > 0, since its second derivative ( − 1 x 2 ) is always negative. ◮ Hence, H ( p 1 , . . . , p m ) = � p i ≤ log � i p i log 1 i p i 1 p i = log m ◮ Alternatively, for X over { 1 , . . . , m } , 1 1 H ( X ) = E X log P X ( X ) ≤ log E X P X ( X ) = log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  76. H ( g ( X )) ≤ H ( X ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  77. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  78. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . ◮ H ( Y = g ( X )) ≤ H ( X ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  79. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . ◮ H ( Y = g ( X )) ≤ H ( X ) . Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  80. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . ◮ H ( Y = g ( X )) ≤ H ( X ) . Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

Recommend

1 Basic Definitions Below are some basic definitions and terminology that will be used throughout

1 Basic Definitions Below are some basic definitions and terminology that will be used throughout

1 Basic Definitions Below are some basic definitions and terminology that will be used throughout the course. Additional terms and definitions will be introduced as we come across them. A graph G is a tuple consisting of a set V ( G ) of elements

450 views • 3 slides
Plan 2 Outline: Basic definitions and facts Relation between L p , interpolation of norms

Plan 2 Outline: Basic definitions and facts Relation between L p , interpolation of norms

Function spaces L p Updated April 22, 2020 Plan 2 Outline: Basic definitions and facts Relation between L p , interpolation of norms Uniform convexity of L p with 1 p 8 Projections on closed convex sets Peculiarities of L p -spaces with

724 views • 28 slides
Street foods basic definitions and facts L. Filippo DAntuono Department of Agri-food science

Street foods basic definitions and facts L. Filippo DAntuono Department of Agri-food science

Street foods basic definitions and facts L. Filippo DAntuono Department of Agri-food science and technology Campus of Food Science, Cesena Alma Mater Studiorum, University of Bologna, Italy Traditional Food International - TFI-2012 Street

280 views • 11 slides
Basic Definitions Basic Definitions Swelling Pressure Due to volumetric Expansion of a

Basic Definitions Basic Definitions Swelling Pressure Due to volumetric Expansion of a

Basic Definitions Basic Definitions Swelling Pressure Due to volumetric Expansion of a rock mass having swelling minerals when comes in contact with water Primitive stress Stress in the state of equilibrium Primitive stress Stress

703 views • 69 slides
network Complex Networks Complex Networks experience for professional or social purposes : a

network Complex Networks Complex Networks experience for professional or social purposes : a

Overview of Overview of Outline Complex Networks Complex Networks Complex Networks Principles of Complex Systems Basic definitions Basic definitions Basic definitions Examples of Examples of Course CSYS/MATH 300, Fall, 2009 Complex

754 views • 30 slides
Convex hull: basic facts Convex hull: basic facts CG Lecture 1 CG Lecture 1 Problem : give a set

Convex hull: basic facts Convex hull: basic facts CG Lecture 1 CG Lecture 1 Problem : give a set

Convex hull: basic facts Convex hull: basic facts CG Lecture 1 CG Lecture 1 Problem : give a set of n points P in the plane, compute Convex Hull Algorithms its convex hull CH ( P ). Basic facts : 2D CH ( P ) is a convex polygon with

297 views • 7 slides
Review of basic probability and statistics Probability: basic definitions A random variable is

Review of basic probability and statistics Probability: basic definitions A random variable is

Review of basic probability and statistics Probability: basic definitions A random variable is the outcome of a natural process that can not be predicted with certainty. Examples: the maximum temperature next Tuesday in Chicago, the

1.85k views • 153 slides
Basics of Traditional Reliability Where we are going N Basic Definitions N Life and times of a

Basics of Traditional Reliability Where we are going N Basic Definitions N Life and times of a

Basics of Traditional Reliability Where we are going N Basic Definitions N Life and times of a Fault N Reliability Models N N-Modular redundant systems Definitions N RELIABILITY: SURVIVAL PROBABILITY When repair is costly or function is

411 views • 15 slides
CMU-Q 15-381 Lecture 1: Introduction AI, basic definitions, problems, road map Teacher:

CMU-Q 15-381 Lecture 1: Introduction AI, basic definitions, problems, road map Teacher:

CMU-Q 15-381 Lecture 1: Introduction AI, basic definitions, problems, road map Teacher: Gianni A. Di Caro O UTLINE AI? Representation and Problem Solving? What this course is about Fundamental definitions and notions Road map

458 views • 43 slides
1 2 Outlines Probability Basic definitions: Randomization experiment Sample

1 2 Outlines Probability Basic definitions: Randomization experiment Sample

1 2 Outlines Probability Basic definitions: Randomization experiment Sample spaces Elementary outcomes Basic operationsconditional probability Bayes Theorem Randomness 3 Things may happen randomly, for examples

839 views • 48 slides
Rings Definitions and Basic Properties Abhijit Das Department of Computer Science and

Rings Definitions and Basic Properties Abhijit Das Department of Computer Science and

Rings Definitions and Basic Properties Abhijit Das Department of Computer Science and Engineering Indian Institute of Technology Kharagpur October 5, 2020 Discrete Structures, Autumn 2020 Abhijit Das Definitions A set R with two binary

589 views • 30 slides
Outline Overview of Complex Networks Complex Networks Complex Networks Basic definitions

Outline Overview of Complex Networks Complex Networks Complex Networks Basic definitions

Overview of Outline Overview of Complex Networks Complex Networks Complex Networks Basic definitions Basic definitions Basic definitions Principles of Complex Systems Books Books Course 300, Fall, 2008 Examples of Examples of Books

588 views • 23 slides
Review Probability Basic definitions: Randomization experiment Sample spaces

Review Probability Basic definitions: Randomization experiment Sample spaces

Review Probability Basic definitions: Randomization experiment Sample spaces Elementary outcomes Event Basic operationsconditional probability Bayes Theorem Objectives Random Variable Discrete random

712 views • 49 slides
network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun Basic

network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun Basic

Overview of Thesaurus deliciousness: Overview of Complex Networks Complex Networks Complex Networks Principles of Complex Systems Basic definitions Basic definitions Examples of Examples of CSYS/MATH 300, Fall, 2011 Complex Networks

563 views • 21 slides
Geek Speak Can you keep up with internet terminology? April 24, 2018 Agenda 1. Fun Facts 2.

Geek Speak Can you keep up with internet terminology? April 24, 2018 Agenda 1. Fun Facts 2.

Geek Speak Can you keep up with internet terminology? April 24, 2018 Agenda 1. Fun Facts 2. Definitions 3. Planning for your event 4. Questions to ask your venue 5. Calculating your networking needs Fun Facts Fun Facts the iPad

522 views • 35 slides
Artificial Intelligence Photo CC BY-SA 3.0 History Definitions Basic Concepts

Artificial Intelligence Photo CC BY-SA 3.0 History Definitions Basic Concepts

Self Service is Artificial Intelligence Photo CC BY-SA 3.0 History Definitions Basic Concepts Challenges Contents Getting Started Applications Evaluation Process Technology Partners History From game playing

598 views • 13 slides
Introduction to Testing and Measurement Testing: Basic Definitions Assessment - process of

Introduction to Testing and Measurement Testing: Basic Definitions Assessment - process of

Introduction to Testing and Measurement Testing: Basic Definitions Assessment - process of documenting knowledge, skills, attitudes, and/or beliefs Evaluation - the making of a judgment about the amount, number, or value

683 views • 46 slides
Relativism Relativism Is ethics in the eye of the beholder? Is ethics in the eye of the

Relativism Relativism Is ethics in the eye of the beholder? Is ethics in the eye of the

Relativism Relativism Is ethics in the eye of the beholder? Is ethics in the eye of the beholder? George Matthews 2020 1 / 25 A basic distinction A basic distinction facts facts 2 / 25 A basic distinction A basic distinction facts

836 views • 59 slides
I ndustrial Rela tions Statistics: Basic concepts an nd definitions and Classifications

I ndustrial Rela tions Statistics: Basic concepts an nd definitions and Classifications

I ndustrial Rela tions Statistics: Basic concepts an nd definitions and Classifications Classifications Reynold Simons, Senior Specialist Employment and Labour Market Policies, ILO Decent Work Team and Office for the Caribbean 1

223 views • 18 slides
Developing a Whole School Approach to Mental Computation Day 2 Basic Facts Milestones for

Developing a Whole School Approach to Mental Computation Day 2 Basic Facts Milestones for

Developing a Whole School Approach to Mental Computation Day 2 Basic Facts Milestones for Multiplication and Division Dr Paul Swan and David Dunstan www.drpaulswan.com.au | Key Ideas Strategies versus recall Understandings

1.14k views • 74 slides
Srujan Kumar Enaganti Terms and Definitions Etymological Origin The basic texts The

Srujan Kumar Enaganti Terms and Definitions Etymological Origin The basic texts The

Srujan Kumar Enaganti Terms and Definitions Etymological Origin The basic texts The Gods and Incarnations Reverence Holy Cow Vedic chants Movie clip Asatoma Sadgamaya - ThanQ Sanskrit Language of

183 views • 16 slides
Tests, Facts and Backtracking Contents Tests Returning results Facts More on

Tests, Facts and Backtracking Contents Tests Returning results Facts More on

Lecture 2 Tests, Facts and Backtracking Contents Tests Returning results Facts More on backtracking The story so far... A Prolog program may succeed or fail . Prolog programs consist of predicate definitions Predicate

576 views • 19 slides
BASIC DEFINITIONS IN ADVOCACY, LOBYING/ INFLUENCING AND ACTIVISM The constitution of Uganda

BASIC DEFINITIONS IN ADVOCACY, LOBYING/ INFLUENCING AND ACTIVISM The constitution of Uganda

BASIC DEFINITIONS IN ADVOCACY, LOBYING/ INFLUENCING AND ACTIVISM The constitution of Uganda Article 38 of : Civic rights and activities 1. Every Uganda citizen has the right to participate in the affairs of government, individually or through

250 views • 23 slides
Convex Hull Algorithms 2D Basic facts Algorithms: Nave, Gift wrapping, Graham

Convex Hull Algorithms 2D Basic facts Algorithms: Nave, Gift wrapping, Graham

CG Lecture 1 CG Lecture 1 Convex Hull Algorithms 2D Basic facts Algorithms: Nave, Gift wrapping, Graham scan, Quick hull, Divide-and-conquer Lower bound 3D Basic facts Algorithms: Gift wrapping, Divide and

466 views • 21 slides

More recommend