kolmogorov complexity and other entropy measures
play

Kolmogorov Complexity and Other Entropy Measures Iftach Haitner - PowerPoint PPT Presentation

Apr 05, 2023 •387 likes •2.24k views

Application of Information Theory, Lecture 8 Kolmogorov Complexity and Other Entropy Measures Iftach Haitner Tel Aviv University. December 16, 2014 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 1 / 24

  1. Kolmogorov complexity ◮ For s string x ∈ { 0 , 1 } ∗ , let K ( x ) be the length of the shortest C ++ program (written in binary) that outputs x (on empty input) ◮ Now the term “described” is well defined. ◮ Why C ++ ? ◮ All (complete) programming language/computational model are essentially equivalent. ◮ Let K ′ ( x ) be the description length of x in another complete language, then | K ( x ) − k ′ ( x ) | ≤ const . ◮ What is K ( x ) for x = 0101010101 . . . 01 � �� � n pairs ◮ “For i = 1 : i ++ : n ; print 01” ◮ K ( x ) ≤ log n + const ◮ This is considered to be small complexity. We typically ignore log n factors. ◮ What is K ( x ) for x being the first n digits of π ? Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 4 / 24

  2. Kolmogorov complexity ◮ For s string x ∈ { 0 , 1 } ∗ , let K ( x ) be the length of the shortest C ++ program (written in binary) that outputs x (on empty input) ◮ Now the term “described” is well defined. ◮ Why C ++ ? ◮ All (complete) programming language/computational model are essentially equivalent. ◮ Let K ′ ( x ) be the description length of x in another complete language, then | K ( x ) − k ′ ( x ) | ≤ const . ◮ What is K ( x ) for x = 0101010101 . . . 01 � �� � n pairs ◮ “For i = 1 : i ++ : n ; print 01” ◮ K ( x ) ≤ log n + const ◮ This is considered to be small complexity. We typically ignore log n factors. ◮ What is K ( x ) for x being the first n digits of π ? ◮ K ( x ) = log n + const Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 4 / 24

  3. More examples Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 5 / 24

  4. More examples ◮ What is K ( x ) for x ∈ { 0 , 1 } n with k ones? Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 5 / 24

  5. More examples ◮ What is K ( x ) for x ∈ { 0 , 1 } n with k ones? � n � ◮ Recall that ≤ 2 nh ( k / n ) k Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 5 / 24

  6. More examples ◮ What is K ( x ) for x ∈ { 0 , 1 } n with k ones? � n � ◮ Recall that ≤ 2 nh ( k / n ) k ◮ Hence K ( x ) ≤ log n + nh ( k / n ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 5 / 24

  7. Bounds Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  8. Bounds ◮ K ( x ) ≤ | x | + const Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  9. Bounds ◮ K ( x ) ≤ | x | + const ◮ Proof : “output x ” Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  10. Bounds ◮ K ( x ) ≤ | x | + const ◮ Proof : “output x ” ◮ Most sequences have high Kolmogorov complexity: Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  11. Bounds ◮ K ( x ) ≤ | x | + const ◮ Proof : “output x ” ◮ Most sequences have high Kolmogorov complexity: ◮ At most 2 n − 1 ( C ++ ) programs of length ≤ n − 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  12. Bounds ◮ K ( x ) ≤ | x | + const ◮ Proof : “output x ” ◮ Most sequences have high Kolmogorov complexity: ◮ At most 2 n − 1 ( C ++ ) programs of length ≤ n − 2 ◮ 2 n strings of length n Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  13. Bounds ◮ K ( x ) ≤ | x | + const ◮ Proof : “output x ” ◮ Most sequences have high Kolmogorov complexity: ◮ At most 2 n − 1 ( C ++ ) programs of length ≤ n − 2 ◮ 2 n strings of length n ◮ Hence, at least 1 2 of n -bit strings have Kolmogorov complexity at least n − 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  14. Bounds ◮ K ( x ) ≤ | x | + const ◮ Proof : “output x ” ◮ Most sequences have high Kolmogorov complexity: ◮ At most 2 n − 1 ( C ++ ) programs of length ≤ n − 2 ◮ 2 n strings of length n ◮ Hence, at least 1 2 of n -bit strings have Kolmogorov complexity at least n − 1 ◮ In particular, a random sequence has Kolmogorov complexity ≈ n Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 6 / 24

  15. Conditional Kolmogorov complexity Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 7 / 24

  16. Conditional Kolmogorov complexity ◮ K ( x | y ) — Kolmogorov complexity of x given y . The length of the shortest partogram that outputd x on input y Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 7 / 24

  17. Conditional Kolmogorov complexity ◮ K ( x | y ) — Kolmogorov complexity of x given y . The length of the shortest partogram that outputd x on input y ◮ Chain rule K ( x , y ) ≈ k ( y ) + k ( x | y ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 7 / 24

  18. H vs. K Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  19. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  20. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but ◮ Both quantities measure the amount of uncertainty or randomness in an object Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  21. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but ◮ Both quantities measure the amount of uncertainty or randomness in an object ◮ Both measure the number of bits it takes to describe an object Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  22. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but ◮ Both quantities measure the amount of uncertainty or randomness in an object ◮ Both measure the number of bits it takes to describe an object ◮ Another property: Let X 1 , . . . , X n be iid, then whp K ( X 1 , . . . , X n ) ≈ H ( X 1 , . . . , X n ) = nH ( X 1 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  23. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but ◮ Both quantities measure the amount of uncertainty or randomness in an object ◮ Both measure the number of bits it takes to describe an object ◮ Another property: Let X 1 , . . . , X n be iid, then whp K ( X 1 , . . . , X n ) ≈ H ( X 1 , . . . , X n ) = nH ( X 1 ) ◮ Proof : ? Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  24. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but ◮ Both quantities measure the amount of uncertainty or randomness in an object ◮ Both measure the number of bits it takes to describe an object ◮ Another property: Let X 1 , . . . , X n be iid, then whp K ( X 1 , . . . , X n ) ≈ H ( X 1 , . . . , X n ) = nH ( X 1 ) ◮ Proof : ? Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  25. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but ◮ Both quantities measure the amount of uncertainty or randomness in an object ◮ Both measure the number of bits it takes to describe an object ◮ Another property: Let X 1 , . . . , X n be iid, then whp K ( X 1 , . . . , X n ) ≈ H ( X 1 , . . . , X n ) = nH ( X 1 ) ◮ Proof : ? AEP Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  26. H vs. K H ( X ) speaks about a random variable X and K ( x ) of a string x , but ◮ Both quantities measure the amount of uncertainty or randomness in an object ◮ Both measure the number of bits it takes to describe an object ◮ Another property: Let X 1 , . . . , X n be iid, then whp K ( X 1 , . . . , X n ) ≈ H ( X 1 , . . . , X n ) = nH ( X 1 ) ◮ Proof : ? AEP ◮ Example: coin flip ( 0 . 7 , 0 . 3 ) then whp we get a string with K ( x ) ≈ n · h ( 0 . 3 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 8 / 24

  27. Universal compression Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 9 / 24

  28. Universal compression ◮ A program of length K ( x ) that outputs x , compresses x into k ( x ) bit of information. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 9 / 24

  29. Universal compression ◮ A program of length K ( x ) that outputs x , compresses x into k ( x ) bit of information. ◮ Example: length of the human genome: 6 · 10 9 bits Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 9 / 24

  30. Universal compression ◮ A program of length K ( x ) that outputs x , compresses x into k ( x ) bit of information. ◮ Example: length of the human genome: 6 · 10 9 bits ◮ But the code is redundant Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 9 / 24

  31. Universal compression ◮ A program of length K ( x ) that outputs x , compresses x into k ( x ) bit of information. ◮ Example: length of the human genome: 6 · 10 9 bits ◮ But the code is redundant ◮ The relevant number to measure the number of possible values is the Kolmogorov complexity of the code. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 9 / 24

  32. Universal compression ◮ A program of length K ( x ) that outputs x , compresses x into k ( x ) bit of information. ◮ Example: length of the human genome: 6 · 10 9 bits ◮ But the code is redundant ◮ The relevant number to measure the number of possible values is the Kolmogorov complexity of the code. ◮ No-one knows its value... Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 9 / 24

  33. Universal probability Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  34. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  35. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  36. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  37. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  38. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. ◮ Interpretation: P U ( x ) is the the probability that you observe x in nature. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  39. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. ◮ Interpretation: P U ( x ) is the the probability that you observe x in nature. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  40. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. ◮ Interpretation: P U ( x ) is the the probability that you observe x in nature. ◮ Computer as an intelligent amplifier Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  41. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. ◮ Interpretation: P U ( x ) is the the probability that you observe x in nature. ◮ Computer as an intelligent amplifier Theorem 2 ∃ c > 0 such that 2 − K ( x ) ≤ P U ( x ) ≤ c · 2 − K ( x ) for every x ∈ { 0 , 1 } ∗ . Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  42. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. ◮ Interpretation: P U ( x ) is the the probability that you observe x in nature. ◮ Computer as an intelligent amplifier Theorem 2 ∃ c > 0 such that 2 − K ( x ) ≤ P U ( x ) ≤ c · 2 − K ( x ) for every x ∈ { 0 , 1 } ∗ . Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  43. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. ◮ Interpretation: P U ( x ) is the the probability that you observe x in nature. ◮ Computer as an intelligent amplifier Theorem 2 ∃ c > 0 such that 2 − K ( x ) ≤ P U ( x ) ≤ c · 2 − K ( x ) for every x ∈ { 0 , 1 } ∗ . ◮ The interesting part is P U ( x ) ≤ c · 2 − K ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  44. Universal probability K ( x ) = min p : p ()= x | p | , where p () is the output of C ++ program defined by p . Definition 1 The universal probability of a string x is P U ( x ) = � p : p ()= x 2 −| p | = Pr p ←{ 0 , 1 } ∞ [ p () = x ] ◮ Namely, the probability that if one picks a program at random, it prints x . ◮ Insensitive (up o constant factor) to the computation model. ◮ Interpretation: P U ( x ) is the the probability that you observe x in nature. ◮ Computer as an intelligent amplifier Theorem 2 ∃ c > 0 such that 2 − K ( x ) ≤ P U ( x ) ≤ c · 2 − K ( x ) for every x ∈ { 0 , 1 } ∗ . ◮ The interesting part is P U ( x ) ≤ c · 2 − K ( x ) � � � ≤ c ◮ Hence, for X ∼ P U , it holds that � E K ( X ) [ − ] H ( X ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 10 / 24

  45. Proving Theorem 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 11 / 24

  46. Proving Theorem 2 ◮ We need to find c > 0 such that k ( x ) ≤ log P U ( x ) + c for every x ∈ { 0 , 1 } ∗ 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 11 / 24

  47. Proving Theorem 2 ◮ We need to find c > 0 such that k ( x ) ≤ log P U ( x ) + c for every x ∈ { 0 , 1 } ∗ 1 1 ◮ In other words, find a program to output x whose length is log P U ( x ) + c Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 11 / 24

  48. Proving Theorem 2 ◮ We need to find c > 0 such that k ( x ) ≤ log P U ( x ) + c for every x ∈ { 0 , 1 } ∗ 1 1 ◮ In other words, find a program to output x whose length is log P U ( x ) + c ◮ Idea, program chooses a leaf on the Shannon code for P U (in which x is � � 1 of depth log ) P U ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 11 / 24

  49. Proving Theorem 2 ◮ We need to find c > 0 such that k ( x ) ≤ log P U ( x ) + c for every x ∈ { 0 , 1 } ∗ 1 1 ◮ In other words, find a program to output x whose length is log P U ( x ) + c ◮ Idea, program chooses a leaf on the Shannon code for P U (in which x is � � 1 of depth log ) P U ( x ) ◮ Problem: P U is not computable Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 11 / 24

  50. Proving Theorem 2 ◮ We need to find c > 0 such that k ( x ) ≤ log P U ( x ) + c for every x ∈ { 0 , 1 } ∗ 1 1 ◮ In other words, find a program to output x whose length is log P U ( x ) + c ◮ Idea, program chooses a leaf on the Shannon code for P U (in which x is � � 1 of depth log ) P U ( x ) ◮ Problem: P U is not computable ◮ Solution: compute a better and better estimate for the tree of P U along with the “mapping" from the tree nodes back to codewords. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 11 / 24

  51. Proving Theorem 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  52. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  53. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  54. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) ◮ The program never gets stack (can always add the node). Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  55. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) ◮ The program never gets stack (can always add the node). Proof : Let x ∈ { 0 , 1 } ∗ . At each point through the execution of M, � ( p , x , · ) ∈ T 2 −| p | ≤ 2 − K ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  56. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) ◮ The program never gets stack (can always add the node). Proof : Let x ∈ { 0 , 1 } ∗ . At each point through the execution of M, � ( p , x , · ) ∈ T 2 −| p | ≤ 2 − K ( x ) Since � x 2 − K ( x ) ≤ 1, the proof follows by Kraft inequality. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  57. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) ◮ The program never gets stack (can always add the node). Proof : Let x ∈ { 0 , 1 } ∗ . At each point through the execution of M, � ( p , x , · ) ∈ T 2 −| p | ≤ 2 − K ( x ) Since � x 2 − K ( x ) ≤ 1, the proof follows by Kraft inequality. � � ◮ ∀ x ∈ { 0 , 1 } ∗ : M adds a node ( · , x , · ) to T at depth 2 + 1 log P U ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  58. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) ◮ The program never gets stack (can always add the node). Proof : Let x ∈ { 0 , 1 } ∗ . At each point through the execution of M, � ( p , x , · ) ∈ T 2 −| p | ≤ 2 − K ( x ) Since � x 2 − K ( x ) ≤ 1, the proof follows by Kraft inequality. � � ◮ ∀ x ∈ { 0 , 1 } ∗ : M adds a node ( · , x , · ) to T at depth 2 + 1 log P U ( x ) Proof : ˆ P U ( x ) converges to P U ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  59. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) ◮ The program never gets stack (can always add the node). Proof : Let x ∈ { 0 , 1 } ∗ . At each point through the execution of M, � ( p , x , · ) ∈ T 2 −| p | ≤ 2 − K ( x ) Since � x 2 − K ( x ) ≤ 1, the proof follows by Kraft inequality. � � ◮ ∀ x ∈ { 0 , 1 } ∗ : M adds a node ( · , x , · ) to T at depth 2 + 1 log P U ( x ) Proof : ˆ P U ( x ) converges to P U ( x ) � � 1 ◮ For x ∈ { 0 , 1 } ∗ , let ℓ ( x ) be the location its ( 2 + log ) -depth node P U ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  60. Proving Theorem 2 ◮ Initial T to be the infinite Binary tree. Program 3 ( M ) Enumerate over all programs in { 0 , 1 } ∗ : at round i emulate the first i programs (one after the other), for i steps, and do: If program p outputs a string x and ( ∗ , x , n ( x )) / ∈ T , place ( p , x , n ( x )) at unused n ( x ) -depth node of � � P U ( x ) = � p ′ : emulated p ′ has output x 2 − | p ′ | + 1 and ˆ 1 T , for n ( x ) = log ˆ P U ( x ) ◮ The program never gets stack (can always add the node). Proof : Let x ∈ { 0 , 1 } ∗ . At each point through the execution of M, � ( p , x , · ) ∈ T 2 −| p | ≤ 2 − K ( x ) Since � x 2 − K ( x ) ≤ 1, the proof follows by Kraft inequality. � � ◮ ∀ x ∈ { 0 , 1 } ∗ : M adds a node ( · , x , · ) to T at depth 2 + 1 log P U ( x ) Proof : ˆ P U ( x ) converges to P U ( x ) � � 1 ◮ For x ∈ { 0 , 1 } ∗ , let ℓ ( x ) be the location its ( 2 + log ) -depth node P U ( x ) ◮ Program for printing x . Run M till it assigns the node at the location of ℓ ( x ) Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 12 / 24

  61. Applications Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 13 / 24

  62. Applications ◮ (another) Proof that there are infinity many primes. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 13 / 24

  63. Applications ◮ (another) Proof that there are infinity many primes. ◮ Assume there are finitely many primes p 1 , . . . , p m Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 13 / 24

  64. Applications ◮ (another) Proof that there are infinity many primes. ◮ Assume there are finitely many primes p 1 , . . . , p m ◮ Any length n integer x can be written as x = � m i = 1 p d i i Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 13 / 24

  65. Applications ◮ (another) Proof that there are infinity many primes. ◮ Assume there are finitely many primes p 1 , . . . , p m ◮ Any length n integer x can be written as x = � m i = 1 p d i i ◮ d i ≤ n , hence length d i ≤ log n Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 13 / 24

  66. Applications ◮ (another) Proof that there are infinity many primes. ◮ Assume there are finitely many primes p 1 , . . . , p m ◮ Any length n integer x can be written as x = � m i = 1 p d i i ◮ d i ≤ n , hence length d i ≤ log n ◮ Hence, K ( x ) ≤ m · log n + const Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 13 / 24

  67. Applications ◮ (another) Proof that there are infinity many primes. ◮ Assume there are finitely many primes p 1 , . . . , p m ◮ Any length n integer x can be written as x = � m i = 1 p d i i ◮ d i ≤ n , hence length d i ≤ log n ◮ Hence, K ( x ) ≤ m · log n + const ◮ But for most numbers k ( x ) ≥ n − 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 13 / 24

  68. Computability of K Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  69. Computability of K ◮ Can we compute K ( x ) ? Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  70. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  71. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  72. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C ◮ Let s be the smallest positive integer s.t. K ( s ) > 2 C + 10 , 000 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  73. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C ◮ Let s be the smallest positive integer s.t. K ( s ) > 2 C + 10 , 000 ◮ s can be computed by the following program: Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  74. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C ◮ Let s be the smallest positive integer s.t. K ( s ) > 2 C + 10 , 000 ◮ s can be computed by the following program: 1. x = 0 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  75. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C ◮ Let s be the smallest positive integer s.t. K ( s ) > 2 C + 10 , 000 ◮ s can be computed by the following program: 1. x = 0 2. While ( K ( x ) < 2 C + 10 , 000 ) : x ++ Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  76. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C ◮ Let s be the smallest positive integer s.t. K ( s ) > 2 C + 10 , 000 ◮ s can be computed by the following program: 1. x = 0 2. While ( K ( x ) < 2 C + 10 , 000 ) : x ++ 3. Output x Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  77. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C ◮ Let s be the smallest positive integer s.t. K ( s ) > 2 C + 10 , 000 ◮ s can be computed by the following program: 1. x = 0 2. While ( K ( x ) < 2 C + 10 , 000 ) : x ++ 3. Output x ◮ Thus K ( s ) < C + log C + log 10 , 000 + const < 2 C + 10 , 000 Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

  78. Computability of K ◮ Can we compute K ( x ) ? ◮ Answer, No. ◮ Proof : Assume K is computable by a program of length C ◮ Let s be the smallest positive integer s.t. K ( s ) > 2 C + 10 , 000 ◮ s can be computed by the following program: 1. x = 0 2. While ( K ( x ) < 2 C + 10 , 000 ) : x ++ 3. Output x ◮ Thus K ( s ) < C + log C + log 10 , 000 + const < 2 C + 10 , 000 ◮ Bergg’s Paradox, revisited: Iftach Haitner (TAU) Application of Information Theory, Lecture 8 December 16, 2014 14 / 24

Recommend

Kolmogorov complexity of 2D sequences Bruno Durand Laboratoire dInformatique Fondamentale de

Kolmogorov complexity of 2D sequences Bruno Durand Laboratoire dInformatique Fondamentale de

Kolmogorov complexity of 2D sequences Bruno Durand Laboratoire dInformatique Fondamentale de Marseille Kolmogorov complexity Goal: to measure the complexity of an individual object (Shannon) theory of information: measures the complexity

1.04k views • 41 slides
preliminaries: Kolmogorov complexity U(p) = T i (p) K(x) = min p { p : U(p) = x }

preliminaries: Kolmogorov complexity U(p) = T i (p) K(x) = min p { p : U(p) = x }

A safe &amp; computable approximation to Kolmogorov complexity Peter Bloem, Francisco Mota, Steven de Rooij, Lus Antunes &amp; Pieter Adriaans preliminaries: Kolmogorov complexity U(p) = T i (p) K(x) = min p { p : U(p) = x }

563 views • 30 slides
Kolmogorov-Loveland stochasticity and Kolmogorov complexity Laurent Bienvenu Laboratoire

Kolmogorov-Loveland stochasticity and Kolmogorov complexity Laurent Bienvenu Laboratoire

Stochasticity Randomness and Kolmogorov complexity Our result Kolmogorov-Loveland stochasticity and Kolmogorov complexity Laurent Bienvenu Laboratoire dInformatique Fondamentale Marseille, France Symposium on Theoretical Aspects of

530 views • 48 slides
Kolmogorov Complexity of Categories Complexity Programing Language Kolmogorov Noson S.

Kolmogorov Complexity of Categories Complexity Programing Language Kolmogorov Noson S.

Kolmogorov Complexity of Categories Noson S. Yanofsky Kolmogorov Kolmogorov Complexity of Categories Complexity Programing Language Kolmogorov Noson S. Yanofsky Complexity of Categories Complexity Brooklyn College, CUNY Computability

952 views • 42 slides
E N T R O P Y S T R U C T U R E Entropy versus resolution Tomasz Downarowicz MOTIVATION

E N T R O P Y S T R U C T U R E Entropy versus resolution Tomasz Downarowicz MOTIVATION

E N T R O P Y S T R U C T U R E Entropy versus resolution Tomasz Downarowicz MOTIVATION Entropy measures exponential complexity in a topological dynamical system: topological entropy very crude, entropy function on invariant

556 views • 23 slides
On the Kolmogorov Complexity of Continuous Real Functions Amin Farjudian Division of Computer

On the Kolmogorov Complexity of Continuous Real Functions Amin Farjudian Division of Computer

Kolmogorov Complexity: Finite Objects Non-Finite Objects Main Results Summary and Future Work On the Kolmogorov Complexity of Continuous Real Functions Amin Farjudian Division of Computer Science University of Nottingham Ningbo, China

497 views • 22 slides
CISC 876: Kolmogorov Complexity Neil Conway March 27, 2007 Neil Conway CISC 876: Kolmogorov

CISC 876: Kolmogorov Complexity Neil Conway March 27, 2007 Neil Conway CISC 876: Kolmogorov

Introduction Basic Properties Variants of K-Complexity Applications Summary CISC 876: Kolmogorov Complexity Neil Conway March 27, 2007 Neil Conway CISC 876: Kolmogorov Complexity Introduction Basic Properties Variants of K-Complexity

1.12k views • 69 slides
Model Uncertainty and Robustness: Entropy Coherent and Entropy Convex Measures of Risk Roger J.

Model Uncertainty and Robustness: Entropy Coherent and Entropy Convex Measures of Risk Roger J.

Model Uncertainty and Robustness: Entropy Coherent and Entropy Convex Measures of Risk Roger J. A. Laeven Dept. of Econometrics and OR Tilburg University, CentER and Eurandom based on joint work with Mitja Stadje August 30, 2011 Entropy

442 views • 40 slides
CDM Program Size Complexity Klaus Sutner Carnegie Mellon University kolmogorov 2018/2/8 22:58

CDM Program Size Complexity Klaus Sutner Carnegie Mellon University kolmogorov 2018/2/8 22:58

CDM Program Size Complexity Klaus Sutner Carnegie Mellon University kolmogorov 2018/2/8 22:58 Wolfram Prize 1 Program-Size Complexity Prefix Complexity Incompleteness Small Universal 3 A Prize Question 4 In May 2007,

1.14k views • 89 slides
Kolmogorov complexity as a language Alexander Shen LIF CNRS, Marseille; on leave from

Kolmogorov complexity as a language Alexander Shen LIF CNRS, Marseille; on leave from

CSR-2011 Kolmogorov complexity as a language Alexander Shen LIF CNRS, Marseille; on leave from , . . . . . . A powerful tool Just a way to reformulate arguments three languages:

1.85k views • 165 slides
Topological arguments and Kolmogorov complexity Andrei Romashenko (joint work with Alexander

Topological arguments and Kolmogorov complexity Andrei Romashenko (joint work with Alexander

JAC 2012 Topological arguments and Kolmogorov complexity Andrei Romashenko (joint work with Alexander Shen) LIRMM, CNRS &amp; UM2, Montpellier; on leave from , Supported by ANR NAFIT grant . . . . . .

1.38k views • 101 slides
Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of a discrete random variable. Properties: H(x) &gt;= 0 Entropy Entropy Lesser the probability for an event, larger the entropy.

473 views • 21 slides
Randomness and Intractability in Kolmogorov Complexity Igor Carboni Oliveira University of Oxford

Randomness and Intractability in Kolmogorov Complexity Igor Carboni Oliveira University of Oxford

Randomness and Intractability in Kolmogorov Complexity Igor Carboni Oliveira University of Oxford ICALP 2019 1 Background and motivation 2 Structure versus Randomness Given a string x { 0 , 1 } n , is it structured or

649 views • 45 slides
Monotone complexity of a pair Pavel Karpovich (Moscow State University) CSR2010 Kolmogorov

Monotone complexity of a pair Pavel Karpovich (Moscow State University) CSR2010 Kolmogorov

Monotone complexity of a pair Pavel Karpovich (Moscow State University) CSR2010 Kolmogorov complexity Decompressor D - a program that reads input binary word and prints output binary word Binary word p is a description of q with respect

524 views • 19 slides
Kolmogorov-Chaitin Complexity of Linear Digital Controllers Implemented using Fixed-point

Kolmogorov-Chaitin Complexity of Linear Digital Controllers Implemented using Fixed-point

Kolmogorov-Chaitin Complexity of Linear Digital Controllers Implemented using Fixed-point Arithmetic James F. Whidborne , John Mckernan , Da-Wei Gu dag@leicester.ac.uk Department of Engineering, University of Leicester

421 views • 29 slides
Polynomial time algorithms in Kolmogorov complexity theory Marius Zimand Towson University CCR

Polynomial time algorithms in Kolmogorov complexity theory Marius Zimand Towson University CCR

Polynomial time algorithms in Kolmogorov complexity theory Marius Zimand Towson University CCR 2015 Marius Zimand (Towson University) Poly.Time Kolmogorov 2015 1 / 40 What is this talk about It will challenge the common perception that

1.32k views • 100 slides
Strong Reducibilities, Scattered Linear Orders, Ranked Sets, and Kolmogorov Complexity Jennifer

Strong Reducibilities, Scattered Linear Orders, Ranked Sets, and Kolmogorov Complexity Jennifer

Strong Reducibilities, Scattered Linear Orders, Ranked Sets, and Kolmogorov Complexity Jennifer Chubb George Washington University Washington, DC Second New York Graduate Student Logic Conference March 17, 2007 Slides available at

442 views • 13 slides
I-Complexity and Discrete Towards Precise . . . Derivative of Logarithms: Our Result Proof A

I-Complexity and Discrete Towards Precise . . . Derivative of Logarithms: Our Result Proof A

Kolmogorov Complexity Need for Approximate . . . I-Complexity Good Properties of I- . . . I-Complexity and Discrete Towards Precise . . . Derivative of Logarithms: Our Result Proof A Group-Theoretic Acknowledgments References Explanation

481 views • 12 slides
Dimension of stationary measures with infinite entropy Adam Spiewak University of Warsaw

Dimension of stationary measures with infinite entropy Adam Spiewak University of Warsaw

Dimension of stationary measures with infinite entropy Adam Spiewak University of Warsaw Student/PhD Dynamical Systems seminar June 05, 2020 Adam Spiewak Dimension of stationary measures with infinite entropy Based on a preprint

720 views • 37 slides
Complexity Measures for Parallel Computation Complexity Measures for Parallel Computation

Complexity Measures for Parallel Computation Complexity Measures for Parallel Computation

Complexity Measures for Parallel Computation Complexity Measures for Parallel Computation Problem parameters: n index of problem size p number of processors Algorithm parameters: t p running time on p processors t 1 time on 1

311 views • 16 slides
Computational Complexity and Information Asymmetry in Election Audits with Low-Entropy Randomness

Computational Complexity and Information Asymmetry in Election Audits with Low-Entropy Randomness

Computational Complexity and Information Asymmetry in Election Audits with Low-Entropy Randomness Nadia Heninger Princeton University August 10, 2010 Computational complexity and information asymmetry in financial products [Arora,

649 views • 25 slides
Brownian Motion and Kolmogorov Complexity Bjrn Kjos-Hanssen University of Hawaii at Manoa

Brownian Motion and Kolmogorov Complexity Bjrn Kjos-Hanssen University of Hawaii at Manoa

Brownian Motion and Kolmogorov Complexity Bjrn Kjos-Hanssen University of Hawaii at Manoa Logic Colloquium 2007 The Church-Turing thesis (1930s) The Church-Turing thesis (1930s) A function f : N N is computable by an algorithm f

756 views • 75 slides
Quantum Turing machines Hiddensee meeting on BSS machines and computability Andr e Nies

Quantum Turing machines Hiddensee meeting on BSS machines and computability Andr e Nies

Quantum Turing machines Hiddensee meeting on BSS machines and computability Andr e Nies August 15, 2016 August 15, 2016 1 / 13 Kolmogorov complexity We survey attempts to introduce an analog of Kolmogorov complexity in the setting of

458 views • 14 slides
Entropy and The Second Law of Thermodynamics Entropy (S)

Entropy and The Second Law of Thermodynamics Entropy (S)

Entropy and The Second Law of Thermodynamics Entropy (S) Entropy is o8en related to disorder but not strictly correct Entropy is a measure

352 views • 8 slides

More recommend