Modeling User Behavior and Interactions M d li U B h i d I t ti Lecture 2: Interpreting Behavior Data Eugene Agichtein Emory University Emory University Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Lecture 2 Plan • Explicit Feedback in IR – Query expansion Query expansion – User control • From Clicks to Relevance Click • 3. Rich Behavior Models – + Browsing – + Session/Context information – + Eye tracking, mouse movements, … E ki 2 Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Recap: Information Seeking Process “Information-seeking … includes recognizing … the g g information problem, establishing a plan of search conducting the search, conducting the search, evaluating the results, and … iterating through the process.”- th h th ” Marchionini, 1989 – Query formulation Q y – Action (query) Relevance Feedback (RF) – Review results – Refine query R fi Adapted from: M. Hearst, SUI, 2009 3 Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Why relevance feedback? • You may not know what you’re looking for, but you’ll know when you see it you ll know when you see it • Query formulation may be difficult; simplify the • Query formulation may be difficult; simplify the problem through iteration • Facilitate vocabulary and concept discovery • Boost recall: “find me more documents like this…” Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Types of Relevance Feedback • Explicit feedback: users explicitly mark relevant and irrelevant documents irrelevant documents • Implicit feedback: system attempts to infer user intentions based on observable behavior • Blind feedback: feedback in absence of any Blind feedback: feedback in absence of any evidence, explicit or otherwise � will not discuss Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Relevance Feedback Example Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
How Relevance Feedback Can be Used • Assume that there is an optimal query – The goal of relevance feedback is to bring the user query The goal of relevance feedback is to bring the user query closer to the optimal query • How does relevance feedback actually work? How does relevance feedback actually work? – Use relevance information to update query – Use query to retrieve new set of documents • What exactly do we “feed back”? – Boost weights of terms from relevant documents – Add terms from relevant documents to the query – Note that this is hidden from the user 7 Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Relevance Feedback in Pictures Initial query Initial query x x x x x o x x x x Δ x x x x o x o o Δ Δ x x x o x o o x x x x x x non-relevant documents R Revised query i d o relevant documents Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Classical Rocchio Algorithm • Used in practice: r r r r 1 1 1 1 ∑ ∑ = α + β − γ q q d d m 0 j j r r D D ∈ ∈ d D d D r nr j r j nr q m = modified query vector; q 0 = original query vector; α β γ : weights (hand chosen or set empirically); α , β , γ : weights (hand-chosen or set empirically); D r = set of known relevant doc vectors; D nr = set of known irrelevant doc vectors • New query – Moves toward relevant documents – Away from irrelevant documents Away from irrelevant documents Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Rocchio in Pictures = α ⋅ query vector original query vector + β β ⋅ p positive feedback vector Typically, γ < β T i ll β − γ ⋅ negative feedback vector α = 1 . 0 0 4 0 8 0 0 0 4 0 8 0 0 Original query β = (+) 0 . 5 2 4 8 0 0 2 1 2 4 0 0 1 Positive Feedback ( ) (-) Negative feedback Negative feedback γ γ = 0 0 . 25 25 8 8 0 0 4 4 4 4 0 16 0 16 2 2 0 0 1 1 1 1 0 0 4 4 New query q y -1 6 3 7 7 0 -3 Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia
Relevance Feedback Example: Initial Query and Top 8 Results Query: New space satellite applications • want high recall � � 1. 0.539, 08/13/91, NASA Hasn't Scrapped Imaging Spectrometer � 2. 0.533, 07/09/91, NASA Scratches Environment Gear From Satellite Plan 3 0 528 04/04/90 Science Panel Backs NASA Satellite Plan But Urges 3. 0.528, 04/04/90, Science Panel Backs NASA Satellite Plan, But Urges Launches of Smaller Probes 4. 0.526, 09/09/91, A NASA Satellite Project Accomplishes Incredible Feat: Staying Within Budget 5. 0.525, 07/24/90, Scientist Who Exposed Global Warming Proposes Satellites for Climate Research 6 0 524 08/22/90 Report Provides Support for the Critics Of Using Big 6. 0.524, 08/22/90, Report Provides Support for the Critics Of Using Big Satellites to Study Climate 7. 0.516, 04/13/87, Arianespace Receives Satellite Launch Pact From Telesat Canada � � 8. 0.509, 12/02/87, Telecommunications Tale of Two Companies Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 11
Relevance Feedback Example: Expanded Query • 2.074 new 2 074 15 106 15.106 space • 30.816 satellite 5.660 application • 5 991 nasa 5.991 nasa 5 196 eos 5.196 eos • 4.196 launch 3.972 aster • 3.516 instrument 3.446 arianespace p • 3.004 bundespost 2.806 ss • 2.790 rocket 2.053 scientist • 2.003 broadcast 1.172 earth • 0.836 oil 0.646 measure Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 12
Top 8 Results After Relevance Feedback � 1. 0.513, 07/09/91, NASA Scratches Environment Gear From Satellite Plan � 2 0 500 08/13/91 NASA H � 2. 0.500, 08/13/91, NASA Hasn't Scrapped Imaging Spectrometer 't S d I i S t t � 3. 0.493, 08/07/89, When the Pentagon Launches a Secret Satellite, Space Sleuths Do Some Spy Work of Their Own py � 4. 0.493, 07/31/89, NASA Uses 'Warm‘ Superconductors For Fast Circuit � 5. 0.492, 12/02/87, Telecommunications Tale of Two Companies � 5 0 492 12/02/87 T l i ti T l f T C i • 6. 0.491, 07/09/91, Soviets May Adapt Parts of SS-20 Missile For Commercial Use • 7. 0.490, 07/12/88, Gaping Gap: Pentagon Lags in Race To Match the Soviets In Rocket Launchers • 8 0 490 06/14/90 R 8. 0.490, 06/14/90, Rescue of Satellite By Space Agency To Cost $90 Million f S lli B S A T C $90 Milli Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 13
Positive vs Negative Feedback • Positive feedback is more valuable than negative feedback (so, set γ < β ; e.g. γ = 0.25, β i f db k ( β 0 25 β = 0.75). • Many systems only allow positive feedback ( γ =0) ( γ 0). Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 14
Relevance Feedback: Assumptions • A1: User has sufficient knowledge for a reasonable initial query – User does not have sufficient initial knowledge – Not enough relevant documents for initial query – Examples: • Misspellings (Brittany Speers) • Cross-language information retrieval • Vocabulary mismatch (e.g., cosmonaut/astronaut) • A2: Relevance prototypes are “well-behaved” Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 15
A2: Relevance prototypes “well- behaved” • Relevance feedback assumes that relevance prototypes are “well-behaved” – All relevant documents are clustered together – Different clusters of relevant documents, but they have significant vocabulary overlap • Violations of A2: – Several (diverse) relevance examples. • Pop stars that worked at McDonalds Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 16
Relevance Feedback: Problems • Long queries are inefficient for typical IR engine. g q yp g – Long response times for user. – High cost for retrieval system. – Partial solution: • Only reweight certain prominent terms Perhaps top 20 by term frequency P h t 20 b t f • Users are often reluctant to provide explicit feedback feedback • It’s often harder to understand why a particular document was retrieved after relevance feedback document was retrieved after relevance feedback Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 17
Probabilistic relevance feedback • Rather than reweighting in a vector space… • If user marked some relevant and irrelevant documents, th then we can build a classifier, such as a Naive Bayes b ild l ifi h N i B model: – P(t k |R) = | D rk | / | D r | – P(t k |NR) = (N k - | D rk |) / (N - | D r |) • t k = term in document; D rk = known relevant doc containing t k ; N k = total number of docs containing t k • And then use these new term weights for re-ranking the remaining results g • Can also use Language Modeling Techniques (See EDS Lectures) Lectures) Eugene Agichtein, Emory University RuSSIR 2009, Petrozavodsk, Karelia 18
Recommend
More recommend
