Consistent bibliometric rankings of Scientists and Departments Denis Bouyssou Thierry Marchant CNRS Paris, France Ghent University Ghent, Belgium EBPM Workshop, Paris December
If you do not know Thierry. . .
Introduction Introduction Evaluation of research: the old way papers (informally but seriously) evaluated by peers scientists (informally but seriously) evaluated by peers departments (informally but seriously) evaluated by peers 2
Introduction Introduction Evaluation of research: the old way papers (informally but seriously) evaluated by peers scientists (informally but seriously) evaluated by peers departments (informally but seriously) evaluated by peers Problems cost time (some) documented biases 2
Introduction Introduction Evaluation of research: the old way papers (informally but seriously) evaluated by peers scientists (informally but seriously) evaluated by peers departments (informally but seriously) evaluated by peers Problems cost time (some) documented biases Evaluation of research: the new way bibliometric indices supposedly objective supposedly at low cost supposedly giving the right incentives 2
Introduction Bibliometric indices Many possible indices counting of papers counting of citations sum of Impact Factors Markovian indices h-index and its variants 3
Introduction Potential problems Indices field normalization (Mathematics vs Molecular Biology) coauthors multiple affiliations books & other publications publications not in English . . . 4
Introduction Potential problems Indices field normalization (Mathematics vs Molecular Biology) coauthors multiple affiliations books & other publications publications not in English . . . Database adequate coverage of publications? adequate time window for collecting citations? quality of data? cois-´ Eric ´ names with diacritical signs and/or T EX ligatures (Fran¸ Effla¨ ır) complicated names (Bou y Sou) 4
Introduction Properties of Bibliometric indices Bibliometric Indices what properties? how to compare them? how to combine them? 5
Introduction Properties of Bibliometric indices Bibliometric Indices what properties? how to compare them? how to combine them? Motivation choosing adequate bibliometric indices should be a subject of scientific investigation this choice is tremendously important this choice should not be in the hands of evaluation bureaucrats 5
Introduction Potential problems 1/2 Evaluation of authors h-index the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) author f : 4 papers with 4 citations each author g : 3 papers with 6 citations each i h ( f ) = 4 > i h ( g ) = 3 6
Introduction Potential problems 1/2 Evaluation of authors h-index the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) author f : 4 papers with 4 citations each author g : 3 papers with 6 citations each i h ( f ) = 4 > i h ( g ) = 3 both authors publish a new paper with 6 citations i h ( f ∗ ) = 4 = i h ( g ∗ ) = 4 6
Introduction Potential problems 1/2 Evaluation of authors h-index the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) author f : 4 papers with 4 citations each author g : 3 papers with 6 citations each i h ( f ) = 4 > i h ( g ) = 3 both authors publish a new paper with 6 citations i h ( f ∗ ) = 4 = i h ( g ∗ ) = 4 both authors publish a new paper with 6 citations i h ( f ∗∗ ) = 4 < i h ( g ∗∗ ) = 5 6
Introduction Potential problems 2/2 Evaluation of authors and departments h-index the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) 7
Introduction Potential problems 2/2 Evaluation of authors and departments h-index the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) Department a author a 1 : 4 papers each one cited 4 times author a 2 : 4 papers each one cited 4 times h-index of both authors is 4 h-index of the department is 4 7
Introduction Potential problems 2/2 Evaluation of authors and departments h-index the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) Department a Department b author a 1 : 4 papers each one author b 1 : 3 papers each one cited 4 times cited 6 times author a 2 : 4 papers each one author b 2 : 3 papers each one cited 4 times cited 6 times h-index of both authors is 4 h-index of both authors is 3 h-index of the department is 4 h-index of the department is 6 7
Introduction Potential problems 2/2 Evaluation of authors and departments h-index the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) Department a Department b author a 1 : 4 papers each one author b 1 : 3 papers each one cited 4 times cited 6 times author a 2 : 4 papers each one author b 2 : 3 papers each one cited 4 times cited 6 times h-index of both authors is 4 h-index of both authors is 3 h-index of the department is 4 h-index of the department is 6 the “best” department contains the “worst” authors! 7
Outline Outline 1 Setting 2 Axioms 3 Scoring rules 4 Limitation and future research 8
Setting Outline 1 Setting 9
Setting Model of Authors Authors an author is a function f from N to N f ( x ) is the number of papers by this author having received x citations 10
Setting Model of Authors Authors an author is a function f from N to N f ( x ) is the number of papers by this author having received x citations Set of all Authors X is the set of all functions f from N to N such that � f ( x ) is finite x ∈ N 10
Setting Model of Authors Authors an author is a function f from N to N f ( x ) is the number of papers by this author having received x citations Set of all Authors X is the set of all functions f from N to N such that � f ( x ) is finite x ∈ N Objective build a binary relation � on X f � g is “given their publication/citation record, scientists f is at least as good as scientist g ” 10
Setting Model of Authors Authors an author is a function f from N to N f ( x ) is the number of papers by this author having received x citations Set of all Authors X is the set of all functions f from N to N such that � f ( x ) is finite x ∈ N Objective build a binary relation � on X f � g is “given their publication/citation record, scientists f is at least as good as scientist g ” Limitations coauthors are ignored in this talk 10
Setting Notation and remarks Notation 0 is an author without any paper 1 x is an author with 1 paper having received x citations 11
Setting Notation and remarks Notation 0 is an author without any paper 1 x is an author with 1 paper having received x citations Remarks it makes sense to add two authors f and g : f + g it makes sense to multiply an author f by an integer n : n · f 11
Setting Model of Departments Departments a department of size k is an element of X k : ( f 1 , f 2 , . . . , f k ) 12
Setting Model of Departments Departments a department of size k is an element of X k : ( f 1 , f 2 , . . . , f k ) Set of all Departments � X k D = k ∈ N 12
Setting Model of Departments Departments a department of size k is an element of X k : ( f 1 , f 2 , . . . , f k ) Set of all Departments � X k D = k ∈ N Objective build a binary relation � on D A � B is “given their publication/citation record of the scientists in departments A and B , department A is at least as good as department B ” 12
Setting Model of Departments Departments a department of size k is an element of X k : ( f 1 , f 2 , . . . , f k ) Set of all Departments � X k D = k ∈ N Objective build a binary relation � on D A � B is “given their publication/citation record of the scientists in departments A and B , department A is at least as good as department B ” Limitations multiple affiliations are ignored field normalization is ignored 12
Axioms Outline 2 Axioms 13
Axioms Axioms Consistency Let A = ( a 1 , a 2 , . . . , a k ) and B = ( b 1 , b 2 , . . . , b k ) be two departments of size k . If a i � b i , for all i ∈ { 1 , 2 , . . . , k } then A � B Furthermore if a i ≻ b i , for some i ∈ { 1 , 2 , . . . , k } then A ⊲ B 14
Axioms Axioms Consistency Let A = ( a 1 , a 2 , . . . , a k ) and B = ( b 1 , b 2 , . . . , b k ) be two departments of size k . If a i � b i , for all i ∈ { 1 , 2 , . . . , k } then A � B Furthermore if a i ≻ b i , for some i ∈ { 1 , 2 , . . . , k } then A ⊲ B Independence For all f, g ∈ X and all x ∈ N f � g ⇔ f + 1 x � g + 1 x 14
Recommend
More recommend
