Counting Words: Expectation = sample average Poisson sampling - PowerPoint PPT Presentation

Binomial sampling vs. Poisson sampling LNRE models Switch to Poisson sampling can be motivated in two ways: Baroni & Evert ◮ Philosophical: Computing ◮ Not as unreasonable as it seems: think of the frequency expectations distribution of nouns in text sample of 1 million running Expectation = sample average words (such as the Brown corpus) ➜ sample size N (= Poisson sampling Plugging in ZM number of noun tokens) will be different for each sample LNRE models ◮ Practical: Pooling types Type density ◮ When N is large and π small (as with word frequency LNRE models Zipf-Mandelbrot distributions), Poisson probabilities are a very good as LNRE model approximation to binomial probabilities The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Binomial sampling vs. Poisson sampling LNRE models Switch to Poisson sampling can be motivated in two ways: Baroni & Evert ◮ Philosophical: Computing ◮ Not as unreasonable as it seems: think of the frequency expectations distribution of nouns in text sample of 1 million running Expectation = sample average words (such as the Brown corpus) ➜ sample size N (= Poisson sampling Plugging in ZM number of noun tokens) will be different for each sample LNRE models ◮ Practical: Pooling types Type density ◮ When N is large and π small (as with word frequency LNRE models Zipf-Mandelbrot distributions), Poisson probabilities are a very good as LNRE model approximation to binomial probabilities The problem Type distribution ◮ In lexical statistics, word frequency distribution models Zipf-Mandelbrot The ZM & fZM LNRE models almost always use Poisson expectations Wrapping up

Poisson expectations for V m and V LNRE models ( N π k ) m Baroni & Evert � · e − N π k � � V m ( N ) = E m ! Computing k expectations Expectation = sample average � 1 − e − N π k � � � � V ( N ) = E Poisson sampling Plugging in ZM k LNRE models Pooling types Type density ◮ E [ V ] sums over probabilities that w k occurs at least once LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Poisson expectations for V m and V LNRE models ( N π k ) m Baroni & Evert � · e − N π k � � V m ( N ) = E m ! Computing k expectations Expectation = sample average � 1 − e − N π k � � � � V ( N ) = E Poisson sampling Plugging in ZM k LNRE models Pooling types Type density ◮ E [ V ] sums over probabilities that w k occurs at least once LNRE models Zipf-Mandelbrot ☞ Now we need to plug in population model for π k as LNRE model (we will use the Zipf-Mandelbrot model, of course) The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in the population model LNRE models C Baroni & Evert π k = Zipf-Mandelbrot: ( k + b ) a Computing expectations Expectation = sample average ( N π k ) m Poisson sampling � Plugging in ZM · e − N π k � � E V m ( N ) = m ! LNRE models k Pooling types Type density LNRE models Zipf-Mandelbrot � 1 − e − N π k � � � � V ( N ) = E as LNRE model The problem k Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in the population model LNRE models C Baroni & Evert π k = Zipf-Mandelbrot: ( k + b ) a Computing expectations Expectation = sample average ( NC ) m Poisson sampling NC � Plugging in ZM ( k + b ) a · m · m ! · e − � � ( k + b ) a E V m ( N ) = LNRE models k Pooling types Type density LNRE models Zipf-Mandelbrot � 1 − e − N π k � � � � V ( N ) = E as LNRE model The problem k Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in the population model LNRE models C Baroni & Evert π k = Zipf-Mandelbrot: ( k + b ) a Computing expectations Expectation = sample average ( NC ) m Poisson sampling NC � Plugging in ZM ( k + b ) a · m · m ! · e − � � ( k + b ) a E V m ( N ) = LNRE models k Pooling types Type density LNRE models Zipf-Mandelbrot NC � 1 − e − � � � ( k + b ) a � V m ( N ) = E as LNRE model The problem k Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in the population model LNRE models C Baroni & Evert π k = Zipf-Mandelbrot: ( k + b ) a Computing expectations Expectation = sample average ( NC ) m Poisson sampling NC � Plugging in ZM ( k + b ) a · m · m ! · e − � � ( k + b ) a E V m ( N ) = LNRE models k Pooling types Type density LNRE models Zipf-Mandelbrot NC � 1 − e − � � � ( k + b ) a � V m ( N ) = E as LNRE model The problem k Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up ☞ This looks ugly even to a mathematician . . . . . . and to a computer

Outline LNRE models Baroni & Evert Computing expectations from the population model Computing expectations The type density function and LNRE modeling Expectation = sample average Poisson sampling Plugging in ZM Zipf-Mandelbrot as LNRE model LNRE models Pooling types Type density LNRE models Wrapping up Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The bad news LNRE models ( NC ) m Baroni & Evert NC � ( k + b ) a · m · m ! · e − � � ( k + b ) a V m ( N ) = E Computing k expectations Expectation = sample average ◮ This looks ugly even to a mathematician Poisson sampling Plugging in ZM ◮ Are we stuck? LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

An idea. . . LNRE models ◮ Look back at the observed word frequency data Baroni & Evert ◮ Huge type frequency lists with many ties in the ranking Computing ◮ and unstable ordering across different samples expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

An idea. . . LNRE models ◮ Look back at the observed word frequency data Baroni & Evert ◮ Huge type frequency lists with many ties in the ranking Computing ◮ and unstable ordering across different samples expectations Expectation = ◮ More robust view on the data by pooling types with the sample average Poisson sampling same frequency ➜ frequency spectrum Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

An idea. . . LNRE models ◮ Look back at the observed word frequency data Baroni & Evert ◮ Huge type frequency lists with many ties in the ranking Computing ◮ and unstable ordering across different samples expectations Expectation = ◮ More robust view on the data by pooling types with the sample average Poisson sampling same frequency ➜ frequency spectrum Plugging in ZM LNRE models ◮ Perhaps we can use a similar approach for the Pooling types Type density probabilities of the population model? LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Pooling type probabilities LNRE models ◮ Different from frequency spectrum because ZM model Baroni & Evert stipulates different, unique probabiliy π k for each type k Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Pooling type probabilities LNRE models ◮ Different from frequency spectrum because ZM model Baroni & Evert stipulates different, unique probabiliy π k for each type k ◮ Pool types with similar probabilities into cells Computing expectations ◮ intuition: contribution to E [ V m ] should be similar Expectation = sample average ◮ e.g. for π k = . 02501 vs. π l = . 02504 Poisson sampling Plugging in ZM ☞ histogram for the distribution of type probabilities LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Pooling type probabilities LNRE models ◮ Different from frequency spectrum because ZM model Baroni & Evert stipulates different, unique probabiliy π k for each type k ◮ Pool types with similar probabilities into cells Computing expectations ◮ intuition: contribution to E [ V m ] should be similar Expectation = sample average ◮ e.g. for π k = . 02501 vs. π l = . 02504 Poisson sampling Plugging in ZM ☞ histogram for the distribution of type probabilities LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model ◮ L = 1000 cells The problem Type distribution ◮ cell j represents types Zipf-Mandelbrot The ZM & fZM with π k ≈ j / L LNRE models Wrapping up ◮ cell count c j = area of bar in histogram 0.010 0.012 0.014 0.016 0.018 0.020 π

Plugging in, 2nd attempt LNRE models ◮ Produce histogram with L cells (e.g., L = 1000) Baroni & Evert ◮ Cell number j contains types w k with π k ≈ j / L Computing ◮ The number of such types is the cell count c j expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in, 2nd attempt LNRE models ◮ Produce histogram with L cells (e.g., L = 1000) Baroni & Evert ◮ Cell number j contains types w k with π k ≈ j / L Computing ◮ The number of such types is the cell count c j expectations Expectation = sample average ◮ Now plug this into the Poisson expectation formula: Poisson sampling Plugging in ZM ( N π k ) m � LNRE models · e − N π k � � V m ( N ) = E Pooling types m ! Type density k LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in, 2nd attempt LNRE models ◮ Produce histogram with L cells (e.g., L = 1000) Baroni & Evert ◮ Cell number j contains types w k with π k ≈ j / L Computing ◮ The number of such types is the cell count c j expectations Expectation = sample average ◮ Now plug this into the Poisson expectation formula: Poisson sampling Plugging in ZM ( N π k ) m � LNRE models · e − N π k � � V m ( N ) = E Pooling types m ! Type density k LNRE models Zipf-Mandelbrot ⇓ as LNRE model The problem Type distribution L ( N · j ) m Zipf-Mandelbrot L m · m ! · e − N · j � � � V m ( N ) = L · c j The ZM & fZM E LNRE models Wrapping up j =1

Plugging in, 2nd attempt LNRE models ◮ Produce histogram with L cells (e.g., L = 1000) Baroni & Evert ◮ Cell number j contains types w k with π k ≈ j / L Computing ◮ The number of such types is the cell count c j expectations Expectation = sample average ◮ Now plug this into the Poisson expectation formula: Poisson sampling Plugging in ZM ( N π k ) m � LNRE models · e − N π k � � V m ( N ) = E Pooling types m ! Type density k LNRE models Zipf-Mandelbrot ⇓ as LNRE model The problem Type distribution L ( N · j ) m Zipf-Mandelbrot L m · m ! · e − N · j � � � V m ( N ) = L · c j The ZM & fZM E LNRE models Wrapping up j =1 ☞ This looks much better (to a mathematician . . . )

Plugging in, 2nd attempt LNRE models ◮ Shorter summation for small L ➜ easier to calculate Baroni & Evert Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in, 2nd attempt LNRE models ◮ Shorter summation for small L ➜ easier to calculate Baroni & Evert ◮ But then it is only a coarse approximation: Computing ◮ for L = 1000, we pool all types with π k < . 001 together expectations ◮ some occcur once in a milion words, some once in 100 Expectation = sample average million words, some only once in a billion words Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in, 2nd attempt LNRE models ◮ Shorter summation for small L ➜ easier to calculate Baroni & Evert ◮ But then it is only a coarse approximation: Computing ◮ for L = 1000, we pool all types with π k < . 001 together expectations ◮ some occcur once in a milion words, some once in 100 Expectation = sample average million words, some only once in a billion words Poisson sampling Plugging in ZM ◮ We can refine the histogram, i.e. increase number L of LNRE models Pooling types cells, but then the summation becomes expensive again Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Plugging in, 2nd attempt LNRE models ◮ Shorter summation for small L ➜ easier to calculate Baroni & Evert ◮ But then it is only a coarse approximation: Computing ◮ for L = 1000, we pool all types with π k < . 001 together expectations ◮ some occcur once in a milion words, some once in 100 Expectation = sample average million words, some only once in a billion words Poisson sampling Plugging in ZM ◮ We can refine the histogram, i.e. increase number L of LNRE models Pooling types cells, but then the summation becomes expensive again Type density LNRE models ◮ The real advantage: we have moved the population Zipf-Mandelbrot model equation from π k to c j , and thus out of the as LNRE model The problem exponential and power functions Type distribution Zipf-Mandelbrot ☞ this makes it much easier to plug in a population model The ZM & fZM LNRE models Wrapping up

Plugging in, 2nd attempt LNRE models ◮ Shorter summation for small L ➜ easier to calculate Baroni & Evert ◮ But then it is only a coarse approximation: Computing ◮ for L = 1000, we pool all types with π k < . 001 together expectations ◮ some occcur once in a milion words, some once in 100 Expectation = sample average million words, some only once in a billion words Poisson sampling Plugging in ZM ◮ We can refine the histogram, i.e. increase number L of LNRE models Pooling types cells, but then the summation becomes expensive again Type density LNRE models ◮ The real advantage: we have moved the population Zipf-Mandelbrot model equation from π k to c j , and thus out of the as LNRE model The problem exponential and power functions Type distribution Zipf-Mandelbrot ☞ this makes it much easier to plug in a population model The ZM & fZM LNRE models Wrapping up   L � m j m � N m ! e − N L j · c j � � � V m ( N ) = · E   L j =1

Refining the histogram LNRE models Baroni & Evert ◮ L = 1000 cells Computing expectations ◮ L = 2000 cells Expectation = sample average Poisson sampling ◮ L = 5000 cells Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model 0.010 0.012 0.014 0.016 0.018 0.020 The problem Type distribution π Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Refining the histogram LNRE models Baroni & Evert ◮ L = 1000 cells Computing expectations ◮ L = 2000 cells Expectation = sample average Poisson sampling ◮ L = 5000 cells Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model 0.010 0.012 0.014 0.016 0.018 0.020 The problem Type distribution π Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Refining the histogram LNRE models Baroni & Evert ◮ L = 1000 cells Computing expectations ◮ L = 2000 cells Expectation = sample average Poisson sampling ◮ L = 5000 cells Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model 0.010 0.012 0.014 0.016 0.018 0.020 The problem Type distribution π Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Refining the histogram LNRE models Baroni & Evert ◮ L = 1000 cells Computing expectations ◮ L = 2000 cells Expectation = sample average Poisson sampling ◮ L = 5000 cells Plugging in ZM LNRE models ◮ type density function Pooling types Type density g ( π ) ≥ 0 LNRE models Zipf-Mandelbrot as LNRE model 0.010 0.012 0.014 0.016 0.018 0.020 The problem Type distribution π Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The type density function LNRE models Baroni & Evert ◮ L = 1000 cells Computing expectations ◮ L = 2000 cells Expectation = sample average Poisson sampling ◮ L = 5000 cells Plugging in ZM LNRE models ◮ type density function Pooling types Type density g ( π ) ≥ 0 LNRE models Zipf-Mandelbrot as LNRE model 0.010 0.012 0.014 0.016 0.018 0.020 The problem Type distribution π Zipf-Mandelbrot The ZM & fZM ◮ Number of types w k with A ≤ π k ≤ B LNRE models Wrapping up = area under curve g ( π ) between A and B

The type density function LNRE models Baroni & Evert ◮ L = 1000 cells Computing expectations ◮ L = 2000 cells Expectation = sample average Poisson sampling ◮ L = 5000 cells Plugging in ZM LNRE models ◮ type density function Pooling types Type density g ( π ) ≥ 0 LNRE models Zipf-Mandelbrot as LNRE model 0.010 0.012 0.014 0.016 0.018 0.020 The problem Type distribution π Zipf-Mandelbrot The ZM & fZM ◮ Number of types w k with A ≤ π k ≤ B LNRE models Wrapping up = area under curve g ( π ) between A and B � B = g ( π ) d π A

The integral form of expectations LNRE models � m � Baroni & Evert N · j L L · e − N · j � � � L · c j E V m ( N ) = Computing m ! expectations j =1 Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The integral form of expectations LNRE models � m � Baroni & Evert N · j L L · e − N · j � � � L · c j E V m ( N ) = Computing m ! expectations j =1 Expectation = sample average Poisson sampling Plugging in ZM ◮ Mathematically, for L → ∞ this converges to an integral, LNRE models Pooling types with j / L ↔ π and c j ↔ g ( π ) d π : Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The integral form of expectations LNRE models � m � Baroni & Evert N · j L L · e − N · j � � � L · c j E V m ( N ) = Computing m ! expectations j =1 Expectation = sample average Poisson sampling Plugging in ZM ◮ Mathematically, for L → ∞ this converges to an integral, LNRE models Pooling types with j / L ↔ π and c j ↔ g ( π ) d π : Type density LNRE models Zipf-Mandelbrot � 1 ( N π ) m as LNRE model · e − N π · g ( π ) d π � � V m ( N ) = E The problem m ! Type distribution 0 Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The integral form of expectations LNRE models � m � Baroni & Evert N · j L L · e − N · j � � � L · c j E V m ( N ) = Computing m ! expectations j =1 Expectation = sample average Poisson sampling Plugging in ZM ◮ Mathematically, for L → ∞ this converges to an integral, LNRE models Pooling types with j / L ↔ π and c j ↔ g ( π ) d π : Type density LNRE models Zipf-Mandelbrot � 1 ( N π ) m as LNRE model · e − N π · g ( π ) d π � � V m ( N ) = E The problem m ! Type distribution 0 Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up ◮ Beautiful! :-)

Summary time What did we just do? LNRE models ◮ Initial formula was too complex Baroni & Evert ◮ Histogram approximation: simpler but coarse Computing ◮ Get nuances back by increasing number of cells expectations Expectation = sample average ◮ . . . but this time we end up with a convenient integral Poisson sampling Plugging in ZM that we can compute efficiently! LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

LNRE models LNRE models Baroni & Evert � 1 ( N π ) m · e − N π · g ( π ) d π Computing � � V m ( N ) = E expectations m ! 0 Expectation = � 1 sample average Poisson sampling 1 − e − N π � � � � V ( N ) = · g ( π ) d π E Plugging in ZM 0 LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

LNRE models LNRE models Baroni & Evert � 1 ( N π ) m · e − N π · g ( π ) d π Computing � � V m ( N ) = E expectations m ! 0 Expectation = � 1 sample average Poisson sampling 1 − e − N π � � � � V ( N ) = · g ( π ) d π E Plugging in ZM 0 LNRE models Pooling types Type density LNRE models ◮ We can plug in any function g defined on [0 , 1] Zipf-Mandelbrot as LNRE model ◮ Population model expressed in terms of a type density The problem Type distribution Zipf-Mandelbrot function g is what we call a LNRE model (for Large The ZM & fZM LNRE models Number of Rare Events, Baayen 2001) Wrapping up

LNRE models LNRE models ◮ You can’t just use any old function, of course – g must Baroni & Evert satisfy the following conditions: Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

LNRE models LNRE models ◮ You can’t just use any old function, of course – g must Baroni & Evert satisfy the following conditions: ◮ g ≥ 0 Computing � 1 expectations Expectation = π · g ( π ) d π = 1 ◮ sample average 0 Poisson sampling Plugging in ZM ☞ Do they look familiar to you? LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

LNRE models LNRE models ◮ You can’t just use any old function, of course – g must Baroni & Evert satisfy the following conditions: ◮ g ≥ 0 Computing � 1 expectations Expectation = π · g ( π ) d π = 1 ◮ sample average 0 Poisson sampling Plugging in ZM ☞ Do they look familiar to you? LNRE models ◮ Moreover, we want to use a function that can be derived Pooling types Type density from a plausible population model, e.g. Zipf-Mandelbrot LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Outline LNRE models Baroni & Evert Computing expectations from the population model Computing expectations The type density function and LNRE modeling Expectation = sample average Poisson sampling Plugging in ZM Zipf-Mandelbrot as LNRE model LNRE models Pooling types Type density LNRE models Wrapping up Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The Zipf-Mandelbrot law as a LNRE model LNRE models ◮ We need to reformulate the Zipf-Mandelbrot law in terms Baroni & Evert of a type density function (to calculate expectations) Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The Zipf-Mandelbrot law as a LNRE model LNRE models ◮ We need to reformulate the Zipf-Mandelbrot law in terms Baroni & Evert of a type density function (to calculate expectations) ◮ ZM has 2 parameters (and fZM has 3 parameters) Computing expectations ➜ type density function will also have parameters Expectation = sample average ◮ same number of parameters, but different interpretation Poisson sampling Plugging in ZM ◮ cannot use parameter values of the population model! LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The Zipf-Mandelbrot law as a LNRE model LNRE models ◮ We need to reformulate the Zipf-Mandelbrot law in terms Baroni & Evert of a type density function (to calculate expectations) ◮ ZM has 2 parameters (and fZM has 3 parameters) Computing expectations ➜ type density function will also have parameters Expectation = sample average ◮ same number of parameters, but different interpretation Poisson sampling Plugging in ZM ◮ cannot use parameter values of the population model! LNRE models ➥ Goal is to find a function g ( π ) that corresponds to a very Pooling types Type density fine histogram of the ZM (or fZM) type population LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up 0.010 0.012 0.014 0.016 0.018 0.020 π π

Zipf-Mandelbrot as a LNRE model LNRE models ◮ Find a function g ( π ) that matches a very fine histogram Baroni & Evert of the Zipf-Mandelbrot law (as a population model) Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Zipf-Mandelbrot as a LNRE model LNRE models ◮ Find a function g ( π ) that matches a very fine histogram Baroni & Evert of the Zipf-Mandelbrot law (as a population model) Computing ◮ This could be done directl by trial and error for every expectations possible combination of ZM parameters a and b : ugly Expectation = sample average ◮ we don’t even know which family of functions to use Poisson sampling Plugging in ZM ◮ there must be a better way! LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Zipf-Mandelbrot as a LNRE model LNRE models ◮ Find a function g ( π ) that matches a very fine histogram Baroni & Evert of the Zipf-Mandelbrot law (as a population model) Computing ◮ This could be done directl by trial and error for every expectations possible combination of ZM parameters a and b : ugly Expectation = sample average ◮ we don’t even know which family of functions to use Poisson sampling Plugging in ZM ◮ there must be a better way! LNRE models Pooling types ◮ Luckily, there is an analytical solution Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Summary of the next few steps . . . for the less mathematically inclined among us LNRE models ◮ Plug together g ( π ) and the ZM law for π k Baroni & Evert Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Summary of the next few steps . . . for the less mathematically inclined among us LNRE models ◮ Plug together g ( π ) and the ZM law for π k Baroni & Evert ◮ Math happens Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Summary of the next few steps . . . for the less mathematically inclined among us LNRE models ◮ Plug together g ( π ) and the ZM law for π k Baroni & Evert ◮ Math happens Computing ◮ Out comes ZM formulated in terms of g ( π ) expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Summary of the next few steps . . . for the less mathematically inclined among us LNRE models ◮ Plug together g ( π ) and the ZM law for π k Baroni & Evert ◮ Math happens Computing ◮ Out comes ZM formulated in terms of g ( π ) expectations Expectation = sample average ◮ And now . . . another detour (sorry!) Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Meet G , the type distribution LNRE models ◮ There is a way to derive ZM’s g analytically Baroni & Evert . . . but it requires another detour Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Meet G , the type distribution LNRE models ◮ There is a way to derive ZM’s g analytically Baroni & Evert . . . but it requires another detour Computing ◮ We can easily calculate the number of types with π ≥ ρ , expectations Expectation = which we call the type distribution G ( ρ ) sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Meet G , the type distribution LNRE models ◮ There is a way to derive ZM’s g analytically Baroni & Evert . . . but it requires another detour Computing ◮ We can easily calculate the number of types with π ≥ ρ , expectations Expectation = which we call the type distribution G ( ρ ) sample average Poisson sampling ◮ According to the ZM law, for ρ = π k there are exactly Plugging in ZM LNRE models k types with π ≥ ρ (viz. the types w 1 , . . . , w k ), i.e.: Pooling types Type density LNRE models G ( π k ) = k Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Meet G , the type distribution LNRE models ◮ There is a way to derive ZM’s g analytically Baroni & Evert . . . but it requires another detour Computing ◮ We can easily calculate the number of types with π ≥ ρ , expectations Expectation = which we call the type distribution G ( ρ ) sample average Poisson sampling ◮ According to the ZM law, for ρ = π k there are exactly Plugging in ZM LNRE models k types with π ≥ ρ (viz. the types w 1 , . . . , w k ), i.e.: Pooling types Type density LNRE models G ( π k ) = k Zipf-Mandelbrot as LNRE model The problem ◮ From this equation we will be able to work out G Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Meet G , the type distribution LNRE models ◮ There is a way to derive ZM’s g analytically Baroni & Evert . . . but it requires another detour Computing ◮ We can easily calculate the number of types with π ≥ ρ , expectations Expectation = which we call the type distribution G ( ρ ) sample average Poisson sampling ◮ According to the ZM law, for ρ = π k there are exactly Plugging in ZM LNRE models k types with π ≥ ρ (viz. the types w 1 , . . . , w k ), i.e.: Pooling types Type density LNRE models G ( π k ) = k Zipf-Mandelbrot as LNRE model The problem ◮ From this equation we will be able to work out G Type distribution Zipf-Mandelbrot The ZM & fZM ◮ With the help of G we can then derive the LNRE LNRE models Wrapping up formulation of ZM in terms of a type density function g ◮ NB: upper case G stands for the type distribution, lower case g for the type density function (standard notation)

Sneak preview: from G to g � 1 LNRE models ◮ G ( ρ ) = g ( π ) d π Baroni & Evert ρ Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Sneak preview: from G to g � 1 LNRE models ◮ G ( ρ ) = g ( π ) d π Baroni & Evert ρ ◮ � B Computing A g ( π ) d π = number of types with A ≤ π k ≤ B expectations ◮ G ( ρ ) = number of types with ρ ≤ π k Expectation = sample average ◮ there are no types with π k > 1 Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Sneak preview: from G to g � 1 LNRE models ◮ G ( ρ ) = g ( π ) d π Baroni & Evert ρ ◮ � B Computing A g ( π ) d π = number of types with A ≤ π k ≤ B expectations ◮ G ( ρ ) = number of types with ρ ≤ π k Expectation = sample average ◮ there are no types with π k > 1 Poisson sampling Plugging in ZM ➥ G ′ = − g , or equivalently g = − G ′ LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Sneak preview: from G to g � 1 LNRE models ◮ G ( ρ ) = g ( π ) d π Baroni & Evert ρ ◮ � B Computing A g ( π ) d π = number of types with A ≤ π k ≤ B expectations ◮ G ( ρ ) = number of types with ρ ≤ π k Expectation = sample average ◮ there are no types with π k > 1 Poisson sampling Plugging in ZM ➥ G ′ = − g , or equivalently g = − G ′ LNRE models Pooling types ◮ This is the second fundamental theorem of calculus Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Sneak preview: from G to g � 1 LNRE models ◮ G ( ρ ) = g ( π ) d π Baroni & Evert ρ ◮ � B Computing A g ( π ) d π = number of types with A ≤ π k ≤ B expectations ◮ G ( ρ ) = number of types with ρ ≤ π k Expectation = sample average ◮ there are no types with π k > 1 Poisson sampling Plugging in ZM ➥ G ′ = − g , or equivalently g = − G ′ LNRE models Pooling types ◮ This is the second fundamental theorem of calculus Type density LNRE models ◮ Intuitively: Zipf-Mandelbrot as LNRE model ◮ If you increase ρ , say from ρ to ρ + x , G decreases The problem Type distribution (fewer types ➜ minus sign) Zipf-Mandelbrot ◮ The amount by which it decreases (number of types The ZM & fZM LNRE models between ρ and ρ + x ) is proportional to g ( ρ ) Wrapping up

Calculating G from the Zipf-Mandelbrot law LNRE models ◮ According to the ZM law, for ρ = π k there are exactly Baroni & Evert k types with π ≥ ρ (viz. the types w 1 , . . . , w k ), i.e.: Computing expectations G ( π k ) = k Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Calculating G from the Zipf-Mandelbrot law LNRE models ◮ According to the ZM law, for ρ = π k there are exactly Baroni & Evert k types with π ≥ ρ (viz. the types w 1 , . . . , w k ), i.e.: Computing expectations G ( π k ) = k Expectation = sample average Poisson sampling Plugging in ZM LNRE models ◮ Insert ZM formula for the type probabilities π k : Pooling types Type density LNRE models � C � Zipf-Mandelbrot = k G as LNRE model ( k + b ) a The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Calculating G from the Zipf-Mandelbrot law LNRE models ◮ According to the ZM law, for ρ = π k there are exactly Baroni & Evert k types with π ≥ ρ (viz. the types w 1 , . . . , w k ), i.e.: Computing expectations G ( π k ) = k Expectation = sample average Poisson sampling Plugging in ZM LNRE models ◮ Insert ZM formula for the type probabilities π k : Pooling types Type density LNRE models � C � Zipf-Mandelbrot = k G as LNRE model ( k + b ) a The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models ☞ Find a function G that satisfies this equation Wrapping up ◮ err . . .

Calculating G from the Zipf-Mandelbrot law LNRE models � � C Baroni & Evert G = k ( k + b ) a Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Calculating G from the Zipf-Mandelbrot law LNRE models � � C Baroni & Evert G = k ( k + b ) a Computing expectations Expectation = sample average ◮ ZM: k �→ π k = C Poisson sampling ( k + b ) a ⇐ ⇒ G: π k �→ k Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Calculating G from the Zipf-Mandelbrot law LNRE models � � C Baroni & Evert G = k ( k + b ) a Computing expectations Expectation = sample average ◮ ZM: k �→ π k = C Poisson sampling ( k + b ) a ⇐ ⇒ G: π k �→ k Plugging in ZM ◮ To get back from π k to k , all we have to do is to solve LNRE models Pooling types the Zipf-Mandelbrot equation for k , obtaining: Type density LNRE models Zipf-Mandelbrot a · ( π k ) − 1 1 as LNRE model k = C a − b The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Calculating G from the Zipf-Mandelbrot law LNRE models � � C Baroni & Evert G = k ( k + b ) a Computing expectations Expectation = sample average ◮ ZM: k �→ π k = C Poisson sampling ( k + b ) a ⇐ ⇒ G: π k �→ k Plugging in ZM ◮ To get back from π k to k , all we have to do is to solve LNRE models Pooling types the Zipf-Mandelbrot equation for k , obtaining: Type density LNRE models Zipf-Mandelbrot 1 a · ( π k ) − 1 as LNRE model k = C a − b The problem Type distribution Zipf-Mandelbrot The ZM & fZM ◮ We can now define G by LNRE models Wrapping up a · ρ − 1 1 G ( ρ ) := C a − b and have found a function that satisfies G ( π k ) = k

From G to g LNRE models a · π − 1 1 Baroni & Evert g ( π ) = − G ′ ( π ) a − b with G ( π ) = C Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

From G to g LNRE models a · π − 1 1 Baroni & Evert g ( π ) = − G ′ ( π ) a − b with G ( π ) = C Computing expectations Expectation = sample average ☞ (trivial) math happens Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

From G to g LNRE models 1 a · π − 1 Baroni & Evert g ( π ) = − G ′ ( π ) a − b with G ( π ) = C Computing expectations Expectation = sample average ☞ (trivial) math happens Poisson sampling Plugging in ZM a / a ) · π − 1 1 a − 1 LNRE models g ( π ) = ( C Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

From G to g LNRE models a · π − 1 1 Baroni & Evert g ( π ) = − G ′ ( π ) a − b with G ( π ) = C Computing expectations Expectation = sample average ☞ (trivial) math happens Poisson sampling Plugging in ZM a / a ) · π − 1 1 a − 1 LNRE models g ( π ) = ( C Pooling types Type density LNRE models ◮ Simplify by renaming constants: Zipf-Mandelbrot as LNRE model g ( π ) = C ∗ · π − α − 1 The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

From G to g LNRE models a · π − 1 1 Baroni & Evert g ( π ) = − G ′ ( π ) a − b with G ( π ) = C Computing expectations Expectation = sample average ☞ (trivial) math happens Poisson sampling Plugging in ZM a / a ) · π − 1 1 a − 1 LNRE models g ( π ) = ( C Pooling types Type density LNRE models ◮ Simplify by renaming constants: Zipf-Mandelbrot as LNRE model g ( π ) = C ∗ · π − α − 1 The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models ◮ α = 1 a replaces ZM’s a as “slope” parameter (0 < α < 1) Wrapping up

From G to g LNRE models a · π − 1 1 Baroni & Evert g ( π ) = − G ′ ( π ) a − b with G ( π ) = C Computing expectations Expectation = sample average ☞ (trivial) math happens Poisson sampling Plugging in ZM a / a ) · π − 1 1 a − 1 LNRE models g ( π ) = ( C Pooling types Type density LNRE models ◮ Simplify by renaming constants: Zipf-Mandelbrot as LNRE model g ( π ) = C ∗ · π − α − 1 The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models ◮ α = 1 a replaces ZM’s a as “slope” parameter (0 < α < 1) Wrapping up ◮ C ∗ is normalizing constant determined from constraint � 1 π · g ( π ) d π = 1 0

The cutoff parameter B LNRE models ◮ We are not quite done yet: we lost one parameter ( b ) Baroni & Evert g ( π ) = C ∗ · π − α − 1 Computing expectations Expectation = sample average Poisson sampling Plugging in ZM LNRE models Pooling types Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The cutoff parameter B LNRE models ◮ We are not quite done yet: we lost one parameter ( b ) Baroni & Evert g ( π ) = C ∗ · π − α − 1 Computing expectations Expectation = sample average ◮ According to the Zipf-Mandelbrot law, there are no types Poisson sampling Plugging in ZM with π > π 1 (where typically π 1 ≪ 1), but g ( π = 1) > 0 LNRE models Pooling types no matter what value α takes Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The cutoff parameter B LNRE models ◮ We are not quite done yet: we lost one parameter ( b ) Baroni & Evert g ( π ) = C ∗ · π − α − 1 Computing expectations Expectation = sample average ◮ According to the Zipf-Mandelbrot law, there are no types Poisson sampling Plugging in ZM with π > π 1 (where typically π 1 ≪ 1), but g ( π = 1) > 0 LNRE models Pooling types no matter what value α takes Type density LNRE models ◮ We need an “upper threshold” parameter Zipf-Mandelbrot as LNRE model ◮ Obvious choice: π 1 , but for mathematical reasons the The problem Type distribution threshold parameter B close rather than equal to π 1 Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

The cutoff parameter B LNRE models ◮ We are not quite done yet: we lost one parameter ( b ) Baroni & Evert g ( π ) = C ∗ · π − α − 1 Computing expectations Expectation = sample average ◮ According to the Zipf-Mandelbrot law, there are no types Poisson sampling Plugging in ZM with π > π 1 (where typically π 1 ≪ 1), but g ( π = 1) > 0 LNRE models Pooling types no matter what value α takes Type density LNRE models ◮ We need an “upper threshold” parameter Zipf-Mandelbrot as LNRE model ◮ Obvious choice: π 1 , but for mathematical reasons the The problem Type distribution threshold parameter B close rather than equal to π 1 Zipf-Mandelbrot The ZM & fZM ◮ Surprise, surprise: B = a − 1 LNRE models Wrapping up b ☞ b is back!

The LNRE ZM model LNRE models � Baroni & Evert C · π − α − 1 0 ≤ π ≤ B g ( π ) = Computing 0 π > B expectations Expectation = sample average ◮ shape parameter 0 < α < 1 (“slope”) Poisson sampling Plugging in ZM ◮ (upper) cutoff parameter 0 < B ≤ 1 LNRE models Pooling types ◮ C = 1 − α Type density LNRE models B 1 − α Zipf-Mandelbrot ◮ relation to Zipf-Mandelbrot law: as LNRE model The problem Type distribution Zipf-Mandelbrot a = 1 The ZM & fZM S = ∞ LNRE models α Wrapping up b = 1 − α B · α

Expectations under the LNRE ZM model LNRE models Baroni & Evert � 1 ( N π ) m e − N π g ( π ) d π Computing � � V m ( N ) = E expectations m ! 0 Expectation = � B sample average = C Poisson sampling ( N π ) m e − N π π − α − 1 d π m ! · Plugging in ZM 0 LNRE models Pooling types = . . . = C m ! · N α · γ ( m − α, NB ) Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution Zipf-Mandelbrot The ZM & fZM LNRE models Wrapping up

Expectations under the LNRE ZM model LNRE models Baroni & Evert � 1 ( N π ) m e − N π g ( π ) d π Computing � � V m ( N ) = E expectations m ! 0 Expectation = � B sample average = C Poisson sampling ( N π ) m e − N π π − α − 1 d π m ! · Plugging in ZM 0 LNRE models Pooling types = . . . = C m ! · N α · γ ( m − α, NB ) Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution ◮ The (lower) incomplete Gamma function γ is a so-called Zipf-Mandelbrot The ZM & fZM LNRE models special function ➜ well-understood by mathematicians Wrapping up

Expectations under the LNRE ZM model LNRE models Baroni & Evert � 1 ( N π ) m e − N π g ( π ) d π Computing � � V m ( N ) = E expectations m ! 0 Expectation = � B sample average = C Poisson sampling ( N π ) m e − N π π − α − 1 d π m ! · Plugging in ZM 0 LNRE models Pooling types = . . . = C m ! · N α · γ ( m − α, NB ) Type density LNRE models Zipf-Mandelbrot as LNRE model The problem Type distribution ◮ The (lower) incomplete Gamma function γ is a so-called Zipf-Mandelbrot The ZM & fZM LNRE models special function ➜ well-understood by mathematicians Wrapping up ◮ γ and m ! = Γ( m + 1) can be computed efficiently ◮ This and several similar properties make the LNRE formulations of ZM and fZM convient and robust