Entropy and Shannons Theorem Lecture 28 December 10, 2013 Sariel - PowerPoint PPT Presentation

CS 573: Algorithms, Fall 2013 Entropy and Shannon’s Theorem Lecture 28 December 10, 2013 Sariel (UIUC) CS573 1 Fall 2013 1 / 24

Part I . Entropy . Sariel (UIUC) CS573 2 Fall 2013 2 / 24

Part II . Extracting randomness . Sariel (UIUC) CS573 3 Fall 2013 3 / 24

Storing all strings of length n and j bits on . . S n , j : set of all strings of length n with j ones in them. 1 . . T n , j : prefix tree storing all T n , j . 2 T 0 , 0 T 1 , 0 T 1 , 1 Sariel (UIUC) CS573 4 Fall 2013 4 / 24

Prefix tree ∀ binary strings of length n with j ones T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

Prefix tree ∀ binary strings of length n with j ones T n,j # of leafs: 0 1 | T n , j | = | T n − 1 , j | + | T n − 1 , j − 1 | T n − 1 ,j T n − 1 ,j − 1 T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

Prefix tree ∀ binary strings of length n with j ones T n,j # of leafs: | T n , j | = | T n − 1 , j | + | T n − 1 , j − 1 | 0 1 ( n ) ( n − 1 ) ( n − 1 ) = + j j j − 1 T n − 1 ,j T n − 1 ,j − 1 T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

Prefix tree ∀ binary strings of length n with j ones T n,j # of leafs: | T n , j | = | T n − 1 , j | + | T n − 1 , j − 1 | 0 1 ( n ) ( n − 1 ) ( n − 1 ) = + j − 1 j j ( n ) = ⇒ | T n , j | = . T n − 1 ,j T n − 1 ,j − 1 j T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

Encoding/decoding strings of S n , j . Lemma . S n , j : Set of binary strings of length n with j ones, sorted lexicographically. . . EncodeBinomCoeff ( α ): Input is string α ∈ S n , j , compute 1 index x of α in S n , j in polynomial time in n . . . { ( n )} DecodeBinomCoeff ( x ): Input index x ∈ 1 , . . . , . 2 j Output x th string α in S n , j , in time O (polylog n + n ) . . Sariel (UIUC) CS573 8 Fall 2013 8 / 24

Extracting randomness . Theorem . Consider a coin that comes up heads with probability p > 1 / 2 . For any constant δ > 0 and for n sufficiently large: (A) One can extract, from an input of a sequence of n flips, an output sequence of (1 − δ ) n H ( p ) (unbiased) independent random bits. (B) One can not extract more than n H ( p ) bits from such a sequence. . Sariel (UIUC) CS573 9 Fall 2013 9 / 24

Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24

Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24

Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24

Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24