Clarification Questions Dataset Dataset Creation ( , , ) triples post question answer post Original post question Clarification question posted in comments answer Edit made to the post in response to the question OR author’s reply to the question comment Dataset Size: ~77 K triples Domains: Askubuntu, Unix, Superuser Note : We identify a question using the question mark (?) token. We match the edit to the answer using timestamp & word embedding similarity based heuristics. 28
Talk Outline o Clarification Questions Dataset o Problem Formulation: Question Ranking o Expected Value of Perfect Information (EVPI) inspired model o Evaluation o Conclusion 29
Problem Formulation: Question Ranking Post How to configure path or set environment variables? … 30
Problem Formulation: Question Ranking Generate Question Candidates Post What is the make of your wifi card? What version of How to configure path Ubuntu do you have? or set environment variables? … What OS are you using? 31
Problem Formulation: Question Ranking Generate Question Rank the question Candidates candidates Post What version of What is the make Ubuntu do you have? of your wifi card? What OS are What version of How to configure path you using? Ubuntu do you have? or set environment variables? … What is the make What OS are of your wifi card? you using? 32
Problem Formulation: Question Ranking How to configure path or set environment variables for installation? I'm aiming to install ape, a simple code for pseudopotential generation. I'm having this error message while running ./configure <error message> So I have the library but the program installation isn't finding it. Any help? Thanks in advance! What version of Ubuntu do you have? Shortlist of How are you installing ape? useful questions Do you have GSL installed? 33
Talk Outline o Clarification Questions Dataset o Problem Formulation: Question Ranking o Expected Value of Perfect Information (EVPI) inspired model o Evaluation o Conclusion 34
Expected Value of Perfect Information (EVPI) inspired model Key Idea How to configure path or set environment variables for installation? I'm aiming to install ape, a simple code for pseudopotential generation. I'm having this error message while running ./configure <error message> So I have the library but the program installation isn't finding it. Any help? Thanks in advance! Possible questions (a) What version of Ubuntu do you have? à Just right 35
Expected Value of Perfect Information (EVPI) inspired model Key Idea How to configure path or set environment variables for installation? I'm aiming to install ape, a simple code for pseudopotential generation. I'm having this error message while running ./configure <error message> So I have the library but the program installation isn't finding it. Any help? Thanks in advance! Possible questions (a) What version of Ubuntu do you have? à Just right (b) What is the make of your wifi card? à Not useful 36
Expected Value of Perfect Information (EVPI) inspired model Key Idea How to configure path or set environment variables for installation? I'm aiming to install ape, a simple code for pseudopotential generation. I'm having this error message while running ./configure <error message> So I have the library but the program installation isn't finding it. Any help? Thanks in advance! Possible questions (a) What version of Ubuntu do you have? à Just right (b) What is the make of your wifi card? à Not useful (c) Are you running Ubuntu 14.10 kernel 4.4.0-59- generic on an x86 64 architecture ? à Unlikely to add value 37
Expected Value of Perfect Information (EVPI) inspired model o Use EVPI to identify questions that add the most value to the given post Avriel, Mordecai, and A. C. Williams. "The value of information and stochastic programming." Operations Research 18.5 (1970) 38
Expected Value of Perfect Information (EVPI) inspired model o Use EVPI to identify questions that add the most value to the given post o Definition: Value of Perfect Information VPI (x) How much value does x add to a given information content c? Avriel, Mordecai, and A. C. Williams. "The value of information and stochastic programming." Operations Research 18.5 (1970) 39
Expected Value of Perfect Information (EVPI) inspired model o Use EVPI to identify questions that add the most value to the given post o Definition: Value of Perfect Information VPI (x) How much value does x add to a given information content c? o Since we have not acquired x, we define its value in expectation Avriel, Mordecai, and A. C. Williams. "The value of information and stochastic programming." Operations Research 18.5 (1970) 40
Expected Value of Perfect Information (EVPI) inspired model o Use EVPI to identify questions that add the most value to the given post o Definition: Value of Perfect Information VPI (x) How much value does x add to a given information content c? o Since we have not acquired x, we define its value in expectation Likelihood of x given c EVPI (x|c) = P (x|c) Utility(x, c) x X Value of updating c with x Avriel, Mordecai, and A. C. Williams. "The value of information and stochastic programming." Operations Research 18.5 (1970) 41
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem 42
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem EVPI ( | ) q i p p : given post q i : question from set of question candidates Q 43
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem Likelihood of a j being the answer to q i on post p EVPI ( | ) = P ( | , ) a j p q i q i p p : given post q i : question from set of question candidates Q 44
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem Likelihood of a j being the answer to q i on post p EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j Utility of updating the post p with answer a j p : given post q i : question from set of question candidates Q 45
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem Likelihood of a j being the answer to q i on post p EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j Utility of updating the post p with answer a j p : given post q i : question from set of question candidates Q a j : answer from set of answer candidates A 46
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem We rank questions based on their EVPI value Likelihood of a j being the answer to q i on post p EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j Utility of updating the post p with answer a j p : given post q i : question from set of question candidates Q a j : answer from set of answer candidates A 47
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem Likelihood of a j being the answer to q i on post p EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j Utility of updating the post p with answer a j p : given post q i : question from set of question candidates Q 1 a j : answer from set of answer candidates A 48
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem 2 Likelihood of a j being the answer to q i on post p EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j Utility of updating the post p with answer a j p : given post q i : question from set of question candidates Q 1 a j : answer from set of answer candidates A 49
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem 2 Answer Modeling EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j Utility of updating the post p with answer a j p : given post q i : question from set of question candidates Q 1 a j : answer from set of answer candidates A 50
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem 2 Answer Modeling EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j 3 Utility of updating the post p with answer a j p : given post q i : question from set of question candidates Q 1 a j : answer from set of answer candidates A 51
Expected Value of Perfect Information (EVPI) inspired model EVPI formulation for our problem 2 Answer Modeling EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j 3 Utility Calculator p : given post q i : question from set of question candidates Q 1 a j : answer from set of answer candidates A 52
Expected Value of Perfect Information (EVPI) inspired model 1 Question & Answer Candidate Generator 53
Expected Value of Perfect Information (EVPI) inspired model 1 Question & Answer Candidate Generator Dataset of (post, question, answer) Post as Documents Lucene Post p as Search query Engine 54
Expected Value of Perfect Information (EVPI) inspired model 1 Question & Answer Candidate Generator Dataset of Ten posts similar (post, question, answer) to given post p p 1 Post as Documents p 2 Lucene Post p as p j Search query Engine p 10 55
Expected Value of Perfect Information (EVPI) inspired model 1 Question & Answer Candidate Generator Dataset of Ten posts similar Questions paired (post, question, answer) to given post p with those posts q 1 p 1 Post as Documents q 2 p 2 Lucene q j Post p as p j Search query Engine q 10 p 10 56
Expected Value of Perfect Information (EVPI) inspired model 1 Question & Answer Candidate Generator Dataset of Ten posts similar Questions paired Answers paired (post, question, answer) to given post p with those posts with those posts q 1 p 1 a 1 Post as Documents q 2 p 2 a 2 Lucene q j Post p as p j a j Search query Engine q 10 p 10 a 10 57
Expected Value of Perfect Information (EVPI) inspired model 1 Question & Answer Candidate Generator Dataset of Ten posts similar Questions paired Answers paired (post, question, answer) to given post p with those posts with those posts q 1 p 1 a 1 Post as Documents q 2 p 2 a 2 Lucene q j Post p as p j a j Search query Engine q 10 p 10 a 10 58
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j 59
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 P ( | , ) a j p q i 60
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 P ( | , ) a j p q i p q i Post Question LSTM LSTM Word embedding module p q i 61
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 P ( | , ) Emb ans ( , ) a j p q i p q i Feedforward p q i Post Question LSTM LSTM Word embedding module p q i 62
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 P ( | , ) ≈ cosine_sim ( Emb ans ( , ) , ) a j p a j q i p q i Feedforward Average p q i Post Question LSTM LSTM Word embedding module p q i a j 63
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 P ( | , ) ≈ cosine_sim ( Emb ans ( , ) , ) a j p a j q i p q i Feedforward Average Emb ans ( , ) p q i Training p q i Post Question LSTM LSTM Word embedding module p q i a j 64
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 P ( | , ) ≈ cosine_sim ( Emb ans ( , ) , ) a j p a j q i p q i Feedforward Average Emb ans ( , ) p q i Training Close to true a i p q i paired with p a i : Ubuntu 14.04 LTS Post Question q i : Which version of LSTM LSTM Ubuntu do you have? Word embedding module p q i a j 65
Expected Value of Perfect Information (EVPI) inspired model Answer Modeling 2 P ( | , ) ≈ cosine_sim ( Emb ans ( , ) , ) a j p a j q i p q i Feedforward Average Emb ans ( , ) p q i Training Close to true a i p q i paired with p Close to a k paired with q k a i : Ubuntu 14.04 LTS similar to true q i Post Question q i : Which version of LSTM LSTM Ubuntu do you have? Word embedding module a k : Ubuntu 11.10 q K : What OS are you using? p q i a j 66
Expected Value of Perfect Information (EVPI) inspired model Utility Calculator 3 EVPI ( | ) = P ( | , ) U ( + ) a j p q i q i p p a j A a j 67
Expected Value of Perfect Information (EVPI) inspired model Utility Calculator 3 U ( + ) p a j 68
Expected Value of Perfect Information (EVPI) inspired model Utility Calculator 3 Value between 0 and 1 U ( + ) p a j Feedforward a j p q i Answer Post Question LSTM LSTM LSTM Word embedding module a j p q i 69
Expected Value of Perfect Information (EVPI) inspired model Utility Calculator 3 Value between 0 and 1 U ( + ) p a j Training Feedforward a j p q i Answer Post Question LSTM LSTM LSTM Word embedding module a j p q i 70
Expected Value of Perfect Information (EVPI) inspired model Utility Calculator 3 Value between 0 and 1 U ( + ) p a j Training Feedforward Label q i : Which version of Ubuntu do you have? y = 1 a i : Ubuntu 14.04 LTS a j p q i Answer Post Question LSTM LSTM LSTM Word embedding module a j p q i 71
Expected Value of Perfect Information (EVPI) inspired model Utility Calculator 3 Value between 0 and 1 U ( + ) p a j Training Feedforward Label q i : Which version of Ubuntu do you have? y = 1 a i : Ubuntu 14.04 LTS a j p q i q j : What OS are you using? y = 0 a j : Ubuntu 11.10 Answer Post Question LSTM LSTM LSTM q k : What is the make of your wifi card? Word embedding module y = 0 a k : TP-Link TL-WDN4800 a j p q i 72
Expected Value of Perfect Information (EVPI) inspired model Utility Calculator 3 Value between 0 and 1 U ( + ) p a j Training ( Minimize binary cross-entropy) Feedforward Label q i : Which version of Ubuntu do you have? y = 1 a i : Ubuntu 14.04 LTS a j p q i q j : What OS are you using? y = 0 a j : Ubuntu 11.10 Answer Post Question LSTM LSTM LSTM q k : What is the make of your wifi card? Word embedding module y = 0 a k : TP-Link TL-WDN4800 a j p q i 73
Expected Value of Perfect Information (EVPI) inspired model 1 2 3 Question & Answer Answer Utility Candidate Generator Modeling Calculator 74
Expected Value of Perfect Information (EVPI) inspired model 1 2 3 Question & Answer Answer Utility Candidate Generator Modeling Calculator Train time behavior: For each (p, q, a) in our train set 1. Generate question candidates (Q) and answer candidates (A) 2. Train Answer Model and Utility Calculator using joint loss function : loss ans (p, q, a, Q) + loss util (y, p, q, a) 75
Expected Value of Perfect Information (EVPI) inspired model 1 2 3 Question & Answer Answer Utility Candidate Generator Modeling Calculator Train time behavior: For each (p, q, a) in our train set 1. Generate question candidates (Q) and answer candidates (A) 2. Train Answer Model and Utility Calculator using joint loss function : loss ans (p, q, a, Q) + loss util (y, p, q, a) Test time behavior: Given a post from our test set 1. Generate question candidates (Q) and answer candidates (A) 2. Calculate P(a j |p, q i ) for each q Q using Answer Model 3. Calculate U(p + a j ) for each a A using Utility Calculator 4. Rank questions by EVPI (q i | p) = P(a j | p, q i ) U(p + a j ) a j A 76
Talk Outline o Clarification Questions Dataset o Problem Formulation: Question Ranking o Expected Value of Perfect Information (EVPI) inspired model o Evaluation o Conclusion 77
Evaluation Too much disk read/write when launching an application I have Xubuntu 13.04 on an old Dell Inspiron. Post When I launch an application it takes a pretty long time to be launched and I see a lot of disk read/write. If the system was short on memory, this would be understandable as the system would use swap. But that's not the case in my situation (i.e. I have this problem even when the RAM is almost empty). Question Candidates 1. How much ram do you have installed ? and what size it the swap disk partition ? 2. If you do not have any problem with getting a little techy then may i suggest a method ? 3. How is it slow exactly ? boot time ? hdd read/write ? cpu time ? graphics rendering ? 4. What is the longest time you have let it run ? 5. This may be a silly question but ... did you make your usb stick bootable ? 6. Do your system were recently updated ? 7. Why not have two ssds in raid 1 for redundancy ? 8. Is that a `parted -- list` on the synology device ? 9. Can you tell us a little about your configuration ? 10. Did you turn hardware virtualization on in bios/efi ? 78
Evaluation Too much disk read/write when launching an application I have Xubuntu 13.04 on an old Dell Inspiron. Post When I launch an application it takes a pretty long time to be launched and I see a lot of disk read/write. If the system was short on memory, this would be understandable as the system would use swap. But that's not the case in my situation (i.e. I have this problem even when the RAM is almost empty). Question Candidates Contains more than one good question! 1. How much ram do you have installed ? and what size it the swap disk partition ? 2. If you do not have any problem with getting a little techy then may i suggest a method ? 3. How is it slow exactly ? boot time ? hdd read/write ? cpu time ? graphics rendering ? 4. What is the longest time you have let it run ? 5. This may be a silly question but ... did you make your usb stick bootable ? 6. Do your system were recently updated ? 7. Why not have two ssds in raid 1 for redundancy ? 8. Is that a `parted -- list` on the synology device ? 9. Can you tell us a little about your configuration ? 10. Did you turn hardware virtualization on in bios/efi ? 79
Evaluation Evaluation set creation process We recruit 10 Unix admin experts using UpWork o Given a post and the set of ten question candidates o § Mark the one best question Mark any other valid questions § 80
Evaluation Evaluation set creation process We recruit 10 Unix admin experts using UpWork o Given a post and the set of ten question candidates o § Mark the one best question Mark any other valid questions § We annotate a total of 500 posts from our test set o Each post is annotated by two experts o 81
Evaluation Evaluation set creation process We recruit 10 Unix admin experts using UpWork o Given a post and the set of ten question candidates o § Mark the one best question Mark any other valid questions § We annotate a total of 500 posts from our test set o Each post is annotated by two experts o Union of Bests : Questions marked as best by either of the annotators o Intersection of Valids : Questions marked as valid by both annotators o Best Valid A1 A2 A1 A2 Union of Bests: { Q2 , Q3 } Q1 Intersection of Valids: { Q1 , Q3 , Q5 } Q2 Q3 Q4 Q5 82
Evaluation Baseline Models 1. Random: Randomly permute the 10 candidate questions 83
Evaluation Baseline Models 1. Random: Randomly permute the 10 candidate questions 2. Bag-of-ngrams: Train linear classifier using bag-of-ngrams of p, q and a 84
Evaluation Baseline Models 1. Random: Randomly permute the 10 candidate questions 2. Bag-of-ngrams: Train linear classifier using bag-of-ngrams of p, q and a 3. Community QA (Nakov et al., 2017) : o SemEval Task: Rank comments by relevance to post on Qatar Living o Winning model: Logistic regression trained with string similarity & word embedding based features (Nandi et al., 2017) o Our baseline: We retrain this model on our dataset 85
Evaluation Baseline Models 1. Random: Randomly permute the 10 candidate questions 2. Bag-of-ngrams: Train linear classifier using bag-of-ngrams of p, q and a 3. Community QA (Nakov et al., 2017) : o SemEval Task: Rank comments by relevance to post on Qatar Living o Winning model: Logistic regression trained with string similarity & word embedding based features (Nandi et al., 2017) o Our baseline: We retrain this model on our dataset 4. Neural (p, q) Value between Feedforward 0 and 1 p q i Post Ques LSTM LSTM Word embedding module p q i 86
Evaluation Baseline Models 1. Random: Randomly permute the 10 candidate questions 2. Bag-of-ngrams: Train linear classifier using bag-of-ngrams of p, q and a 3. Community QA (Nakov et al., 2017) : o SemEval Task: Rank comments by relevance to post on Qatar Living o Winning model: Logistic regression trained with string similarity & word embedding based features (Nandi et al., 2017) o Our baseline: We retrain this model on our dataset 4. Neural (p, q) Value between Feedforward 0 and 1 5. Neural (p, q, a) p i q i a i Post Ques Ans LSTM LSTM LSTM Word embedding module p i q i a i 87
Evaluation Baseline Models 1. Random: Randomly permute the 10 candidate questions 2. Bag-of-ngrams: Train linear classifier using bag-of-ngrams of p, q and a 3. Community QA (Nakov et al., 2017) : o SemEval Task: Rank comments by relevance to post on Qatar Living o Winning model: Logistic regression trained with string similarity & word embedding based features (Nandi et al., 2017) o Our baseline: We retrain this model on our dataset 4. Neural (p, q) Value between Feedforward 0 and 1 5. Neural (p, q, a) p i q i a i Both Neural (p, q, a) and EVPI (q | p, a) have similar no. of parameters Post Ques Ans LSTM LSTM LSTM Word embedding module p i q i a i 88
Evaluation RESULTS Union of Best EVPI Neural (p, q, a) Neural (p, q) Community QA Bag-of-ngrams Random 0 Precision @1 89
Evaluation RESULTS Union of Best EVPI Neural (p, q, a) Neural (p, q) Community QA Bag-of-ngrams 19.4 Random 17.5 0 5 10 15 20 25 30 35 40 Precision @1 90
Evaluation RESULTS Union of Best EVPI Neural (p, q, a) 25.2 Neural (p, q) Non-linear vs linear Community QA Bag-of-ngrams 19.4 Random 17.5 0 5 10 15 20 25 30 35 40 Precision @1 91
Evaluation RESULTS Union of Best EVPI Neural (p, q, a) 25.2 Explicitly modeling “answer” is useful Neural (p, q) 21.9 Community QA Bag-of-ngrams 19.4 Random 17.5 0 5 10 15 20 25 30 35 40 Precision @1 92
Evaluation RESULTS Union of Best EVPI Neural (p, q, a) 25.2 Neural (p, q) 21.9 Both use only (p, q) Community QA 23.1 Bag-of-ngrams 19.4 Random 17.5 0 5 10 15 20 25 30 35 40 Precision @1 93
Evaluation RESULTS Union of Best EVPI 27.7 Neural (p, q, a) 25.2 Neural (p, q) 21.9 Community QA 23.1 Bag-of-ngrams 19.4 Random 17.5 0 5 10 15 20 25 30 35 40 Precision @1 Note : Difference between EVPI and all baselines is statistically significant with p < 0.05 94
Evaluation RESULTS Union of Best EVPI 27.7 Mainly differ in their loss function Neural (p, q, a) 25.2 Neural (p, q) 21.9 Community QA 23.1 Bag-of-ngrams 19.4 Random 17.5 0 5 10 15 20 25 30 35 40 Precision @1 Note : Difference between EVPI and all baselines is statistically significant with p < 0.05 95
Evaluation RESULTS Intersection of Valids Union of Best 36.1 EVPI 27.7 34.4 Neural (p, q, a) 25.2 31.6 Neural (p, q) 21.9 33.6 Community QA 23.1 25.6 Bag-of-ngrams 19.4 26.4 Random 17.5 0 5 10 15 20 25 30 35 40 Precision @1 Note : Difference between EVPI and all baselines is statistically significant with p < 0.05 96
Evaluation RESULTS Intersection of Valids Union of Best 32.2 EVPI 23.4 31.8 Neural (p, q, a) 22.7 Not statistically significant 30 Neural (p, q) 20.9 30.8 Community QA 21.2 27.6 Bag-of-ngrams 19.4 26.4 Random 17.5 0 5 10 15 20 25 30 35 Precision @3 97
Talk Outline o Clarification Questions Dataset o Problem Formulation: Question Ranking o Expected Value of Perfect Information (EVPI) inspired model o Evaluation o Conclusion 98
CONCLUSION ü Key Contributions Create a dataset of ~77K clarification questions (and answers) with context § Introduce novel model that integrates deep learning with classic notion of § expected value of perfect information Create an evaluation set of size 500 with expert human annotations § 99
CONCLUSION ü Key Contributions Create a dataset of ~77K clarification questions (and answers) with context § Introduce novel model that integrates deep learning with classic notion of § expected value of perfect information Create an evaluation set of size 500 with expert human annotations § ü Key findings A context can have multiple good clarification question § Explicitly modeling the answer helps in identifying good questions § EVPI formalism provides leverage over similarly expressive feedforward network § 100
Recommend
More recommend
