Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Aggregating Referee Scores: an Algebraic Approach Rolf Haenni R easoning under UN certainty Group Institute of Computer Science and Applied Mathematics University of Berne, Switzerland COMSOC’08 2nd International Workshop on Computational Social Choice Liverpool, UK 3–5 September 2008 Rolf Haenni, University of Berne, Switzerland Slide 1 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 2 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 2 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 2 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 2 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 2 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 3 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion RUN Research Group Jacek Jonczy Reto Kohlas Rolf Haenni Michael Wachter Supported by: • Swiss National Science Foundation (Project PP002-102652) • Hasler Foundation (U/Projects No. 2034 & 2042) • Leverhulme Trust (Progicnet) Rolf Haenni, University of Berne, Switzerland Slide 4 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 5 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Peer Reviewing • Peer reviewing (or refereeing) is the process of evaluating submitted documents by anonymous experts (referees) • Widely applied by scientific journals, conferences, and funding agencies • Submitted documents are typically reviewed by 3–4 referees • Referee reports typically contain: ◮ Scores for various criteria (e.g. originality, clarity, etc.) ◮ Overall score for paper quality (e.g. 1–10) ◮ Level of expertise (e.g. 1–10) ◮ Detailed comments • Papers with highest aggregated scores are accepted ⇒ How? Rolf Haenni, University of Berne, Switzerland Slide 6 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Demo A prototype implementation is available at: http://www.iam.unibe.ch/ ∼ run/referee Rolf Haenni, University of Berne, Switzerland Slide 7 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Formal Setting Input: D = { 1 , . . . , n } → submitted documents R = { 1 , . . . , m } → referees referees ( i ) ⊆ R → referees assigned to document i s i,j = ( q i,j , e i,j ) → referee j ’s score for document i q i,j ∈ [0 , 1] → quality judgement e i,j ∈ [0 , 1] → expertise level Rolf Haenni, University of Berne, Switzerland Slide 8 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Formal Setting (cont.) Output: � s i = s i,j → combined score s i = ( q i , e i ) for document i j ∈ referees ( i ) q i ∈ [0 , 1] → combined quality judgement e i ∈ [0 , 1] → combined expertise level S = { s 1 , . . . , s n } → set of combined scores ( D , � ) → total preorder over D r : D → N → ranking function over D Note that classifying the documents (e.g. accepted/rejected) is a special case of a total preorder � Rolf Haenni, University of Berne, Switzerland Slide 9 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Example Referees Total Preorder 1 2 3 4 5 ⊗ 4 � { 1 , 3 } � 2 1 – – r (1) = 2 s 1 , 1 s 1 , 3 s 1 , 4 s 1 Documents 2 – – r (2) = 1 s 2 , 1 s 2 , 2 s 2 , 5 s 2 3 – s 3 , 2 s 3 , 3 s 3 , 4 – s 3 r (3) = 2 4 s 4 , 1 – s 4 , 3 – s 4 , 5 s 4 r (4) = 4 Rolf Haenni, University of Berne, Switzerland Slide 10 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Problem Formulation • Problem 1: Find an appropriate combination operator ⊗ • Problem 2: Find an appropriate total preorder � • Solution: Apply the opinion calculus (i) Transform scores s i,j into opinions ϕ i,j (ii) Apply the combination operator ⊗ defined for independent opinions ⇒ ϕ i (iii) Use various probabilistic transformations f ∈ { g, h, p } to turn each ϕ i into a Bayesian opinion f ( ϕ i ) (iv) Use the natural total order � 0 of Bayesian opinions to define � Rolf Haenni, University of Berne, Switzerland Slide 11 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Outline Introduction 1 Problem Formulation 2 The Opinion Calculus 3 Evaluating Referee Scores 4 Conclusion 5 Rolf Haenni, University of Berne, Switzerland Slide 12 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Opinions • The opinion calculus is an algebraic version of the Dempster’s theory of lower and upper probabilities (Dempster, 1967) for two-valued hypotheses H ∈ { yes, no } • Terminology and references: ◮ (Hajek and Valdes, 1991) → Dempster pairs, dempsteroids ◮ (Jøsang, 1997) → opinions, subjective logic ◮ (Daniel, 2002) → d-pairs, Dempster’s semigroup • An opinion relative to H is a triple ϕ = ( b, d, i ) ∈ [0 , 1] 3 ◮ b + d + i = 1 ◮ b = degree of belief of H ◮ d = degree of disbelief H ◮ i = degree of ignorance relative to H • Dempster’s theory provides a probabilistic interpretation for b , d , and i Rolf Haenni, University of Berne, Switzerland Slide 13 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Opinion Triangle e = (0 , 0 , 1) Ignorance Disbelief Belief u = (1 2 , 1 n = (0 , 1 , 0) p = (1 , 0 , 0) 2 , 0) Rolf Haenni, University of Berne, Switzerland Slide 14 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Opinion Classes neutral positive: b > d simple negative simple positive negative: b < d indi ff erent indifferent: b = d simple: b = 0 or d = 0 extremal: b = 1 or d = 1 neutral: i = 1 negative positive Bayesian: i = 0 extremal Bayesian extremal negative positive Rolf Haenni, University of Berne, Switzerland Slide 15 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion Combining Opinions • Let ϕ 1 = ( b 1 , d 1 , i 1 ) and ϕ 2 = ( b 2 , d 2 , i 2 ) be independent: � b 1 b 2 + b 1 i 2 + i 1 b 2 , d 1 d 2 + d 1 i 2 + i 1 d 2 � i 1 i 2 ϕ 1 ⊗ ϕ 2 = , 1 − b 1 d 2 − d 1 b 2 1 − b 1 d 2 − d 1 b 2 1 − b 1 d 2 − d 1 b 2 • Let ϕ i = ( b i , d i , i i ) , 1 ≤ i ≤ n , be independent: ϕ 1 ⊗ · · · ⊗ ϕ n � �� � �� � � 1 , 1 , 1 � � � = ( b i + i i ) − i i ( d i + i i ) − i i i i K K K i i i i i � � � for K = ( b i + i i ) + ( d i + i i ) − i i > 0 i i i • Click here to start demo Rolf Haenni, University of Berne, Switzerland Slide 16 of 27
Introduction Problem Formulation The Opinion Calculus Evaluating Referee Scores Conclusion The Opinion Monoid • Φ = { ( b, d, i ) : b + d + i = 1 } is not closed under ⊗ ◮ ⊗ is undefined for p = (1 , 0 , 0) and n = (0 , 1 , 0) ◮ Add inconsistent opinion z = (1 , 1 , − 1) ◮ Define p ⊗ n = n ⊗ p = z ◮ Define ϕ ⊗ z = z ⊗ ϕ = z , for all ϕ ∈ Φ • Φ z = Φ ∪ { z } is closed under ⊗ ◮ ⊗ is commutative ◮ ⊗ is associative • Therefore, (Φ z , ⊗ ) is a commutative semigroup ◮ e = (0 , 0 , 1) is the identity element: e ⊗ ϕ = ϕ ⊗ e = ϕ ◮ z = (1 , 1 , − 1) is the zero element: z ⊗ ϕ = ϕ ⊗ z = z • Therefore, (Φ z , ⊗ , e ) is a commutative monoid with zero element z Rolf Haenni, University of Berne, Switzerland Slide 17 of 27
Recommend
More recommend
