Bayesian Method for Repeated Threshold Estimation Alexander Petrov - PowerPoint PPT Presentation

Bayesian Method for Repeated Threshold Estimation Alexander Petrov Department of Psychology Ohio State University

Motivation: Perceptual Learning � Non-stationary thresholds � Dynamics of learning is important � Must use naïve observers � Low motivation � high lapsing rates � Slow learning � many sessions � Large volume of low-quality binary data 2 http://alexpetrov.com/pub/vss06/ 5/6/06

Objective: Data Reduction 3 http://alexpetrov.com/pub/vss06/ 5/6/06

Isn’t This a Solved Problem? � Up/down (Levitt, 1970) � PEST (Taylor & Creelman, 1967) � BEST PEST (Pentland, 1980) � QUEST (Watson & Pelli, 1979) � ML-Test (Harvey, 1986) � Ideal (Pelli, 1987) � YAAP (Treutwein, 1989) � and many others… 4 http://alexpetrov.com/pub/vss06/ 5/6/06

We Solve a Different Problem � Standard methods: � Adaptive stimulus placement � Stopping criterion � Threshold estimation � Our method: � Threshold estimation � Integrate information across blocks 5 http://alexpetrov.com/pub/vss06/ 5/6/06

Weibull Psychometric Function α β = − − − α β ( ; , ) 1 exp( exp((log log ) )) W x x α β γ λ = γ + − γ − λ α β ( ; , , , ) (1 ) ( ; , ) P x W x 1- λ � Threshold log α � Slope β � Guessing rate γ γ � Lapsing rate λ log α log x 6 http://alexpetrov.com/pub/vss06/ 5/6/06

Two Kinds of Parameters � Threshold log α Parameters of interest θ � Slope β � Guessing rate γ Nuisance parameters φ � Lapsing rate λ The nuisance parameters are harder to estimate but change more slowly than the threshold parameter. 7 http://alexpetrov.com/pub/vss06/ 5/6/06

Get the Best of Both Worlds Use long data sequences to constrain the nuisance parameters; use short sequences to estimate the thresholds. 8 http://alexpetrov.com/pub/vss06/ 5/6/06

Joint Posterior of θ k , φ θ φ = … … ( , | ; , ) p y y y y y − + 1 1 1 k k k k n ∏∫ θ φ θ φ θ θ φ θ ( | , ) ( ) ( ) ( ) ( | , ) p p p p p d y y k k k i i i i ≠ i k Information about φ extracted Likelihood of Priors from the other data sets current data Modified prior for the current block 9 http://alexpetrov.com/pub/vss06/ 5/6/06

Two-Pass Algorithm � Pass 1: for each block i , calculate = ∫ φ θ θ φ θ ( | ) ( ) ( | , ) p p p d y y i i � Pass 2: for each block k , calculate ∏∫ θ φ = θ φ θ φ φ ( , | ) ( | , ) ( ) ( ) ( | ) p p p p p y y y k k k k k i ≠ i k 10 http://alexpetrov.com/pub/vss06/ 5/6/06

Posterior Thresholds = ∫ − θ φ θ φ θ φ 1 ( ) (.75; , ) ( , | ) p T P p d d y k k k k k posterior normal Posterior density -3 -2.5 -2 -1.5 -1 -0.5 0 75% threshold 11 http://alexpetrov.com/pub/vss06/ 5/6/06

Some Details � Vaguely informative priors: α ∝ μ σ (log ) N( , ) p α α β ∝ μ σ ( ) N( , ) p β β λ ∝ ( ) Beta( , ) p a b λ λ � Implemented on a grid: log α x β x λ � Assume γ = .5 for 2AFC data � MATLAB software available at http://alexpetrov.com/softw/ 12 http://alexpetrov.com/pub/vss06/ 5/6/06

Simulation 1: Stationary α = − = log 1.204 const β = 1.5 λ = .10 = − 1.217 T 75 13 http://alexpetrov.com/pub/vss06/ 5/6/06

Stimulus Placement � 2 interleaved 0 staircases -0.5 log intensity � 100 trials/block -1 � 10 catch -1.5 � 40 x 3down/1up � 50 x 2down/1up -2 0 20 40 60 80 100 trial number � 100 runs of 12 blocks each 14 http://alexpetrov.com/pub/vss06/ 5/6/06

Threshold Estimators ML Estimator Mean Med Std median mean ML -1.24 -1.23 .27 true Median -1.26 -1.23 .28 Frequency Mean -1.30 -1.27 .31 Std. dev. 0.41 0.36 .15 1200 Monte Carlo estimates -3 -2.5 -2 -1.5 -1 -0.5 0 True 75% threshold = -1.217 Estimated threshold 15 http://alexpetrov.com/pub/vss06/ 5/6/06

β x λ Distribution from Pass 1 1 5 true β and λ ML β and λ 0.9 4 Slope BETA 0.8 P(correct) 3 0.7 2 0.6 1 0.5 0.05 0.1 0.15 0.2 -4 -3 -2 -1 0 Lapsing rate LAMBDA log intensity 16 http://alexpetrov.com/pub/vss06/ 5/6/06

Catch Trials Are Worthwhile ML Estimator Mean Med Std median mean ML -1.24 -1.22 .31 true Median -1.29 -1.26 .30 Frequency Mean -1.36 -1.33 .34 Std. dev. 0.58 0.57 .16 1200 Monte Carlo estimates No catch trials presented -3 -2.5 -2 -1.5 -1 -0.5 0 True 75% threshold = -1.217 Estimated threshold 17 http://alexpetrov.com/pub/vss06/ 5/6/06

Simulation 2: With Learning α = − − − / 800 t log 0.693 ( 2) e β = 1.5 λ = .10 0 -0.5 T75 -1 -1.5 -2 0 1000 2000 3000 4000 5000 6000 18 http://alexpetrov.com/pub/vss06/ 5/6/06

Group Learning Curve, N= 100 0 True learning curve Reconstruction ± CI 95 -0.2 -0.4 -0.6 -0.8 ML threshold -1 -1.2 -1.4 -1.6 -1.8 -2 0 10 20 30 40 50 60 Block number 19 http://alexpetrov.com/pub/vss06/ 5/6/06

More Realistic Sample, N= 10 0 True learning curve Reconstruction ± CI 95 -0.2 -0.4 -0.6 -0.8 ML threshold -1 -1.2 -1.4 -1.6 -1.8 -2 0 10 20 30 40 50 60 Block number 20 http://alexpetrov.com/pub/vss06/ 5/6/06

Individual Runs 0 0 0 -1 -1 -1 -2 -2 -2 0 0 0 -1 -1 -1 -2 -2 -2 21 http://alexpetrov.com/pub/vss06/ 5/6/06

The Method Performs Well ML Estimator Mean Med Std median mean ML -0.03 -0.02 .28 true Median -0.05 -0.03 .29 Frequency Mean -0.08 -0.05 .32 Std. dev. 0.42 0.39 .15 6000 Monte Carlo estimates Similar to the stationary case -1.5 -1 -0.5 0 0.5 1 No systematic bias over time Estimated threshold - true threshold 22 http://alexpetrov.com/pub/vss06/ 5/6/06

Example: Actual Data, N= 8 Jeter, Dosher, Petrov, & Lu (2005) 1 0.5 0 -0.5 75% threshold In high noise -1 -1.5 -2 In no noise -2.5 -3 0 2 4 6 8 10 12 14 16 Block number 23 http://alexpetrov.com/pub/vss06/ 5/6/06

Future Work � Sensitivity to priors? � Compare with standard ML methods � Individual differences � Estimate slope in addition to threshold � Non-stationary β and λ ? � Recommended stimulus placement? � Hierarchical models 24 http://alexpetrov.com/pub/vss06/ 5/6/06

5/6/06 25 The End http://alexpetrov.com/pub/vss06/