Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum Max Planck - PowerPoint PPT Presentation

Sreyasi Nag Chowdhury, Niket Tandon, Gerhard Weikum Max Planck Institute for Informatics, Saarbrücken, Germany

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” Text-only Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” Visual objects: bicycle, bus, car Text-only Text + visual Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” “Biking by the river” Visual objects: bicycle, bus, car Visual objects: train, piano Text-only Text + visual Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” “Biking by the river” Visual objects: bicycle, bus, car Visual objects: train, piano Text-only Text-only Text + visual Text + visual Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” “Biking by the river” “Riding for a cause.” Visual objects: bicycle, bus, car Visual objects: train, piano Visual objects: person, bicycle CSK: (riding bicycle, be, environment friendly) Text-only Text-only Text + visual Text + visual Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” “Biking by the river” “Riding for a cause.” Visual objects: bicycle, bus, car Visual objects: train, piano Visual objects: person, bicycle Text-only Text-only Text/visual Text + visual Text + visual Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” “Biking by the river” “Riding for a cause.” Visual objects: bicycle, bus, car Visual objects: train, piano Visual objects: person, bicycle CSK: (riding bicycle, be, environment friendly) Text-only Text-only Text/visual Text + visual Text + visual Text + visual + CSK Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

User Query Concrete Abstract Q: bicycle in street Q: environment friendly traffic “Wow! Double - decker buses still run!” “Biking by the river” “Riding for a cause.” Visual objects: bicycle, bus, car Visual objects: train, piano Visual objects: person, bicycle CSK: (riding bicycle, be, environment friendly) Text-only Text-only Text/visual Text + visual Text + visual Text + visual + CSK Our contribution Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 1

 CSK: Where do we get it from?  CSK: How do we use it?  CSK: How to combine noisy signals?  CSK: Does it help? Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 2

CSK: WHERE DO WE GET IT FROM?  Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 3

CSK: WHERE DO WE GET IT FROM?  Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood  Our corpus : Wiki articles from domain ‘tourism’ Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 3

CSK: WHERE DO WE GET IT FROM?  Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood  Our corpus : Wiki articles from domain ‘tourism’  Pruned by Jaccard Similarity Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 3

CSK: WHERE DO WE GET IT FROM?  Existing CSK knowledge bases: WordNet, ConceptNet, WebChild, Knowlywood  Our corpus : Wiki articles from domain ‘tourism’  Pruned by Jaccard Similarity “tourism” “be travel for” “recreation, leisure, family, business purposes” Domain-specific “people” “fall in” “love” ReVerb triples --------------------------------------------------------------------------------------------------- “the bloody hell” “be” “you”  ~22,000 CSK triples Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 3

CSK: HOW DO WE USE IT?  Query string: travel with backpack  CSK to expand query • t1: (tourists, use, travel maps) • t2: (tourists, carry, backpack) • t3: (backpack, is a type of, bag) Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 4

CSK: HOW DO WE USE IT?  Query string: travel with backpack  CSK to expand query • t1: (tourists, use, travel maps) • t2: (tourists, carry, backpack) • t3: (backpack, is a type of, bag)  Document x with features • Textual: “A tourist reading a map by the road” • Visual: person, bag, bottle, bus Text-only systems Text + visual + CSK systems Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 4

CSK: HOW DO WE USE IT?  Query string: travel with backpack  CSK to expand query • t1: (tourists, use, travel maps)  CSK bridge vocabulary gap • t2: (tourists, carry, backpack) between query and document • t3: (backpack, is a type of, bag)  CSK establish relations between concepts  Document x with features • Textual: “A tourist reading a map by the road”  CSK diminish noise from • Visual: person, bag, bottle, bus modalities – ensemble effect Text-only systems Text + visual + CSK systems Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 4

CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Document x Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 5

CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 5

CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x Visual features x v : backpack, person Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 5

CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x Visual features x v : backpack, bag, container person, casual agent, organism Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 5

CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Textual features x x Document x Visual features x v : backpack, bag, container person, casual agent, organism Query: “group excursion” CSK features Query expansion: (an excursion, be trip by, a group of people) (organized excursions, book through, a tour company) Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 5

CSK: HOW DO WE USE IT? A tour group is standing on the grass with ruins in the background. Group of people standing in front of a stone structure. Visual features x v : backpack, bag, container person, casual agent, organism Query: “group excursion ” Query expansion: (an excursion, be trip by, a group of people) (organized excursions, book through, a tour company) Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 5

CSK: HOW TO COMBINE NOISY SIGNALS?  Mixture LM:  Commonsense-aware LM:  Smoothed LM: , where  Basic LM: , where Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 6

CSK: HOW TO COMBINE NOISY SIGNALS?  Mixture LM:  Commonsense-aware LM:  Smoothed LM: , where  Basic LM: , where Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 6

CSK: HOW TO COMBINE NOISY SIGNALS?  Mixture LM:  Commonsense-aware LM: CSK triple  Smoothed LM: , where  Basic LM: , where Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 6

CSK: HOW TO COMBINE NOISY SIGNALS?  Mixture LM:  Commonsense-aware LM: Probabilities based on word-wise overlaps  Smoothed LM: , where  Basic LM: , where Sreyasi Nag Chowdhury, AKBC 2016 17/06/2016 6