Formulation of the . . . Our Main Idea Which Objective . . . Which Constraints Are . . . Why Some Families of Probability Which Constraints Are . . . Distributions Are Practically Efficient: Optimal Distributions: . . . A Symmetry-Based Explanation All Constraints Are . . . Constraints Are Shift- . . . Vladik Kreinovich 1 , Olga Kosheleva 1 , Conclusion Hung T. Nguyen 2 , 3 , and Songsak Sriboonchitta 3 Home Page 1 University of Texas at El Paso, Title Page El Paso, TX 79968, USA ◭◭ ◮◮ vladik@utep.edu, olgak@utep.edu 2 Department of Mathematical Sciences ◭ ◮ New Mexico State University Las Cruces, NM 88003, USA, hunguyen@nmsu.edu Page 1 of 32 3 Faculty of Economics, Chiang Mai University Chiang Mai, Thailand, Go Back songsakecon@gmail.com Full Screen Close Quit
Formulation of the . . . Our Main Idea 1. Formulation of the Problem Which Objective . . . • Theoretically, we can have infinite many different fam- Which Constraints Are . . . ilies of probability distributions. Which Constraints Are . . . Optimal Distributions: . . . • In practice, only a few families have been empirically All Constraints Are . . . successful. Constraints Are Shift- . . . • For some of these families, there is a good theoretical Conclusion explanation for their success. Home Page • For example, the Central Limit theorem explains the Title Page ubiquity of normal distributions. ◭◭ ◮◮ • However, for many other families, there is no theoreti- ◭ ◮ cal explanation for their empirical success. Page 2 of 32 • In this talk, we provide a theoretical explanation of Go Back their success. Full Screen Close Quit
Formulation of the . . . Our Main Idea 2. Our Main Idea Which Objective . . . • We are looking for a family which is the best among Which Constraints Are . . . all the families that satisfy appropriate constraints. Which Constraints Are . . . Optimal Distributions: . . . • So, we need to select: All Constraints Are . . . – objective functions and Constraints Are Shift- . . . – constraints. Conclusion • The numerical value of each quantity x depends: Home Page – on the starting point for measurement and Title Page – on the choice of the measuring unit. ◭◭ ◮◮ • If we change the starting point to the one x 0 units ◭ ◮ smaller, then all the values shift by x 0 : x → x + x 0 . Page 3 of 32 • Similarly, if we change the original measuring unit by Go Back a one λ times smaller, then x → λ · x : 2 m = 200 cm. Full Screen • Shifts and scaling do not change the physical quantities Close – just change the numbers. Quit
Formulation of the . . . Our Main Idea 3. Main Idea (cont-d) Which Objective . . . • Shifts and scaling do not change the physical quantities Which Constraints Are . . . – just change the numbers. Which Constraints Are . . . Optimal Distributions: . . . • It is therefore reasonable to require that objective func- All Constraints Are . . . tions and constraints are shift- and scale-invariant. Constraints Are Shift- . . . • We look for distributions which are optimal w.r.t. in- Conclusion variant objective functions under invariant constraints. Home Page • It turns out that the resulting optimal families indeed Title Page include many empirically successful families of distri- ◭◭ ◮◮ butions. ◭ ◮ • Thus, our approach explains the empirical success of Page 4 of 32 many such families. Go Back • This approach is in good accordance with modern physics, where symmetries are ubiquitous. Full Screen Close Quit
Formulation of the . . . Our Main Idea 4. Which Objective Functions Are Invariant? Which Objective . . . • According to decision theory, decisions of a rational Which Constraints Are . . . agent are equivalent to maximizing utility . Which Constraints Are . . . Optimal Distributions: . . . • It is reasonable to require that: All Constraints Are . . . – if have two distribution which differ only in some Constraints Are Shift- . . . local region, Conclusion – and the first distribution is better, then Home Page – if we replace a common distribution outside this Title Page region by another common distribution, ◭◭ ◮◮ – the first distribution will still be better. ◭ ◮ • It is known that each utility function with this property Page 5 of 32 is either a sum or a product of functions A ( ρ ( x ) , x ). Go Back • Maximizing the product is equivalent to maximizing Full Screen its logarithm: the sum of logarithms. Close Quit
Formulation of the . . . Our Main Idea 5. Invariant Objective Functions (cont-d) Which Objective . . . • Thus, the general expression of an objective function Which Constraints Are . . . � with the above “localness” property is A ( ρ ( x ) , x ) dx . Which Constraints Are . . . Optimal Distributions: . . . • Shift-invariance implies no explicit dependence on x : All Constraints Are . . . � u = A ( ρ ( x )) dx. Constraints Are Shift- . . . Conclusion • Scaling y = λ · x changes ρ ( x ) to λ − 1 · ρ ( λ − 1 · y ). Home Page Title Page � � A ( ρ ′ ( x )) dx , then • We require that if A ( ρ ( x )) dx = ◭◭ ◮◮ this equality remains after re-scaling. ◭ ◮ • This requirement leads to: Page 6 of 32 � – entropy S = − ρ ( x ) · ln( ρ ( x )) dx and ( ρ ( x )) α dx . Go Back � � – generalized entropy: ln( ρ ( x )) dx and Full Screen Close Quit
Formulation of the . . . Our Main Idea 6. Which Constraints Are Invariant: Definitions Which Objective . . . • Decision making is based on expected values of utility. Which Constraints Are . . . Which Constraints Are . . . • So, we consider constraints of the type Optimal Distributions: . . . � f i ( x ) · ρ ( x ) dx = c i . All Constraints Are . . . Constraints Are Shift- . . . • We says that constraints corr. to f i ( x ) are shift- Conclusion invariant if: Home Page � – the values of the corr. quantities f i ( x ) · ρ ( x ) dx Title Page – uniquely determine the values of these quantities ◭◭ ◮◮ for a shifted distribution. ◭ ◮ • We says that constraints corr. to f i ( x ) are scale- Page 7 of 32 invariant if: Go Back � – the values of the corr. quantities f i ( x ) · ρ ( x ) dx Full Screen – uniquely determine the values of these quantities for a scaled distribution. Close Quit
Formulation of the . . . Our Main Idea 7. Which Constraints Are Invariant: Results Which Objective . . . • Functions f i ( x ) corresponding to shift-invariant con- Which Constraints Are . . . straints are linear combinations of the functions Which Constraints Are . . . x k · exp( a · x ) · sin( ω · x + ϕ ) , Optimal Distributions: . . . k = 0 , 1 , 2 , . . . All Constraints Are . . . • Functions f i ( x ) corresponding to scale-invariant con- Constraints Are Shift- . . . straints are linear combinations of the functions Conclusion (ln( x − x 0 )) k · ( x − x 0 ) a · sin( ω · ln( x − x 0 ) + ϕ ) . Home Page Title Page • Only functions which are both shift- and scale- ◭◭ ◮◮ invariant are polynomials. ◭ ◮ • We optimize: Page 8 of 32 – an invariant objective function J ( ρ ) Go Back � – under the constraint ρ ( x ) dx = 1 and � – under the constraints f i ( x ) · ρ ( x ) dx = c i for in- Full Screen variant functions f i ( x ). Close Quit
Formulation of the . . . Our Main Idea 8. Optimal Distributions: General Formula Which Objective . . . • The Lagrange multiplier methods means optimizing Which Constraints Are . . . Which Constraints Are . . . �� � �� � � J ( ρ )+ λ · ρ ( x ) dx − 1 + f i ( x ) · ρ ( x ) dx − c . Optimal Distributions: . . . i All Constraints Are . . . Constraints Are Shift- . . . • Differentiating this expression with respect to ρ ( x ) and equating the resulting derivative to 0, we get: Conclusion Home Page � ln( ρ ( x )) = − 1+ λ + λ i · f i ( x ) for the usual entropy; Title Page i ◭◭ ◮◮ � − ( ρ ( x )) − 1 = λ + � λ i · f i ( x ) for J ( ρ ) = ln( ρ ( x )) dx ; and ◭ ◮ i Page 9 of 32 � ( − α ) · ( ρ ( x )) α − 1 = λ + ( ρ ( x )) α dx. � λ i · f i ( x ) for J ( ρ ) = Go Back i Full Screen • This is how we will explain all empirically successful distributions. Close Quit
Formulation of the . . . Our Main Idea 9. All Constraints Are Both Shift- and Scale- Which Objective . . . Invariant, Objective Function is Entropy Which Constraints Are . . . • In this case, f i ( x ) are polynomials P i ( x ), and equation Which Constraints Are . . . is Optimal Distributions: . . . � ln( ρ ( x )) = − 1 + λ + λ i · P i ( x ) . All Constraints Are . . . i Constraints Are Shift- . . . • The right-hand side of this formula is a polynomial Conclusion P ( x ), so ρ ( x ) = exp( P ( x )). Home Page • The most widely used distribution, the normal distri- Title Page bution, is exactly of this type: ◭◭ ◮◮ − ( x − µ ) 2 1 � � ◭ ◮ √ ρ ( x ) = 2 π · exp . σ 2 Page 10 of 32 • It is a known fact that it has the largest entropy among Go Back all the distributions with given mean and variance. Full Screen Close Quit
Recommend
More recommend
