How to Estimate Amount Need to Distinguish . . . of Useful - PowerPoint PPT Presentation

How to Gauge the . . . Finite Case Finite Case with . . . How to Gauge . . . How to Estimate Amount Need to Distinguish . . . of Useful Information, Such Distinction Is . . . Such Distinction Is . . . in Particular Under How to Estimate the . . . What If We Only Have . . . Imprecise Probability Home Page Title Page e 1 , Olga Kosheleva 2 , and Luc Longpr´ Vladik Kreinovich 1 ◭◭ ◮◮ 1 Department of Computer Science ◭ ◮ 2 Department of Teacher Education University of Texas at El Paso Page 1 of 22 El Paso, TX 79968, USA longpre@utep.edu, olgak@utep.edu, Go Back vladik@utep.edu Full Screen Close Quit

How to Gauge the . . . Finite Case 1. How to Gauge the Amount of Information: Finite Case with . . . General Idea How to Gauge . . . • Our ultimate goal is to gain a complete knowledge of Need to Distinguish . . . the world. Such Distinction Is . . . Such Distinction Is . . . • In practice, we usually have only partial information. How to Estimate the . . . • In other words, in practice, we have uncertainty . What If We Only Have . . . • Additional information allows us to decrease this un- Home Page certainty. Title Page • It is therefore reasonable to: ◭◭ ◮◮ – gauge the amount of information in the new knowl- ◭ ◮ edge Page 2 of 22 – by how much this information decreases the original Go Back uncertainty. Full Screen • Uncertainty means that for some questions, we do not have a definite answer. Close Quit

How to Gauge the . . . Finite Case 2. Gauging Amount of Information (cont-d) Finite Case with . . . • Once we learn the answers to these questions, we thus How to Gauge . . . decrease the original uncertainty. Need to Distinguish . . . Such Distinction Is . . . • It is therefore reasonable to: Such Distinction Is . . . – estimate the amount of uncertainty How to Estimate the . . . – by the number of questions needed to eliminate this What If We Only Have . . . uncertainty. Home Page • Of course, not all questions are created equal: Title Page ◭◭ ◮◮ – some can have a simple binary “yes”-“no” answer; – some look for a more detailed answer – e.g., we can ◭ ◮ ask what is the value of a certain quantity. Page 3 of 22 • No matter what is the answer, we can describe this Go Back answer inside the computer. Full Screen • Everything in a computer is represented as 0s and 1s. Close Quit

How to Gauge the . . . Finite Case 3. Gauging Amount of Information (cont-d) Finite Case with . . . • Everything in a computer is represented as 0s and 1s. How to Gauge . . . Need to Distinguish . . . • So, each answer is a sequence of 0s and 1s. Such Distinction Is . . . • Such a several-bits question can be represented as a Such Distinction Is . . . sequence of on-bit questions: How to Estimate the . . . – we can first ask what is the first bit of the answer, What If We Only Have . . . Home Page – we can then ask what is the second bit of the an- swer, etc. Title Page ◭◭ ◮◮ • So, every question can thus be represented as a se- quence of one-bit (“yes”-“no”) questions. ◭ ◮ • So, it is reasonable to: Page 4 of 22 – measure uncertainty Go Back – by the smaller number of such “yes”-“no” questions Full Screen which are needed to eliminate this uncertainty. Close Quit

How to Gauge the . . . Finite Case 4. Finite Case Finite Case with . . . • Let us first consider the situation when we have finitely How to Gauge . . . many N alternatives. Need to Distinguish . . . Such Distinction Is . . . • If we ask one binary question, then we can get two Such Distinction Is . . . possible answers (0 and 1). How to Estimate the . . . • Thus, we can uniquely determine one of the two differ- What If We Only Have . . . ent states. Home Page • If we ask 2 binary questions, then we can get four pos- Title Page sible combinations of answers (00, 01, 10, and 11). ◭◭ ◮◮ • In general, if we ask q binary questions, then we can get 2 q possible combinations of answers. ◭ ◮ • Thus, we can uniquely determine one of 2 q states. Page 5 of 22 Go Back • So, to identify one of n states, we need to ask q ques- tions, where 2 q ≥ N . Full Screen • The smallest such q is ⌈ log 2 ( N ) ⌉ . Close Quit

How to Gauge the . . . Finite Case 5. Finite Case with Known Probabilities Finite Case with . . . • So far, we considered the situation when we have n How to Gauge . . . alternatives about whose frequency we know nothing. Need to Distinguish . . . Such Distinction Is . . . • In practice, we often know the probabilities p 1 , . . . , p n Such Distinction Is . . . of different alternatives; in this case: How to Estimate the . . . – instead of considering the worst-case number of bi- What If We Only Have . . . nary questions needed to eliminate uncertainty, Home Page – it is reasonable to consider the average number of Title Page questions. ◭◭ ◮◮ • This value can be estimated as follows. ◭ ◮ • We have a large number N of similar situations with Page 6 of 22 n -uncertainty. Go Back • In N · p 1 of these situations, the actual state is State 1. Full Screen • In N · p 2 of them, the actual state is State 2, etc. Close Quit

How to Gauge the . . . Finite Case 6. Case of Known Probabilities (cont-d) Finite Case with . . . • The average number of binary questions can be ob- How to Gauge . . . tained if we divide: Need to Distinguish . . . Such Distinction Is . . . – the overall number of questions needed to deter- Such Distinction Is . . . mine the states in all N situations, How to Estimate the . . . – by N . � N What If We Only Have . . . � N ! • There are = ( N · p 1 )! · ( N − N · p 1 )! ways to Home Page N · p 1 select the situations in State 1. Title Page • Out of these, there are many ways to to select N · p 2 ◭◭ ◮◮ situations in State 2: ◭ ◮ � N − N · p 1 � ( N − N · p 1 )! = ( N · p 2 )! · ( N − N · p 1 − N · p 2 )! . Page 7 of 22 N · p 2 Go Back • So, the number A of possible arrangements is: Full Screen N ! ( N − N · p 1 )! ( N · p 1 )! · ( N − N · p 1 )! · ( N · p 2 )! · ( N − N · p 1 − N · p 2 )! · . . . Close Quit

How to Gauge the . . . Finite Case 7. Case of Known Probabilities (final) Finite Case with . . . N ! How to Gauge . . . • Thus, A = ( N · p 1 )! · ( N · p 2 )! · . . . · ( N · p n )! . Need to Distinguish . . . • To identify an arrangement, we need to ask the follow- Such Distinction Is . . . ing number of binary questions: Such Distinction Is . . . How to Estimate the . . . n � Q = log 2 ( A ) = log 2 ( N !) − log 2 (( N · p i )!) . What If We Only Have . . . Home Page i =1 � m � m Title Page • Here, m ! ∼ , so e ◭◭ ◮◮ log 2 ( m !) ∼ m · (log 2 ( m ) − log 2 ( e )) . ◭ ◮ • As a result, we get the usual Shannon’s formula: Page 8 of 22 n Go Back � q = − p i · log 2 ( p i ) . Full Screen i =1 Close Quit

How to Gauge the . . . Finite Case 8. How to Gauge Uncertainty: Continuous Case Finite Case with . . . • In the continuous case, when the unknown(s) can take How to Gauge . . . any of the infinitely many values from some interval. Need to Distinguish . . . Such Distinction Is . . . • So, we need infinitely many binary questions to Such Distinction Is . . . uniquely determine the exact value. How to Estimate the . . . • It thus makes sense to determine each value with a What If We Only Have . . . given accuracy ε > 0: Home Page – we divide the real line into intervals [ x i − ε, x i + ε ], Title Page where x i +1 = x i + 2 ε , and ◭◭ ◮◮ – we want to find out to which of these intervals the ◭ ◮ actual value x belongs. Page 9 of 22 • For small ε , the probability p i of belonging to the i -th interval is equal to p i ≈ ρ ( x i ) · (2 ε ). Go Back • Substituting this expression for p i into Shannon’s for- Full Screen mula, we get the following formula: Close Quit

How to Gauge the . . . Finite Case 9. Continuous Case (cont-d) Finite Case with . . . n n How to Gauge . . . � � q = − p i · log 2 ( p i ) = − ρ ( x i ) · (2 ε ) · log 2 ( ρ ( x i ) · (2 ε )) , i.e., Need to Distinguish . . . i =1 i =1 Such Distinction Is . . . n n � � Such Distinction Is . . . q = − ρ ( x i ) · (2 ε ) · log 2 ( ρ ( x i )) − ρ ( x i ) · (2 ε ) · log 2 (2 ε ) . How to Estimate the . . . i =1 i =1 • The first term in this sum has the form What If We Only Have . . . Home Page n n � � − ρ ( x i ) · log 2 ( ρ ( x i )) · (2 ε ) = − ρ ( x i ) · log 2 ( ρ ( x i )) · ∆ x i . Title Page i =1 i =1 ◭◭ ◮◮ • This term is an integral sum for the interval ◭ ◮ � − ρ ( x ) · log 2 ( ρ ( x )) dx. Page 10 of 22 Go Back • Thus, for small ε , it is practically equal to this interval. Full Screen Close Quit