QUALS EDITION Verb Physics Relative Physical Knowledge of Actions and Objects Max Forbes Yejin Choi
[Gao et al., 2016] [Angeli and Manning, 2014] [Gordon and Schubert, 2012] [Li et al., 2014]
Physical properties of objects What is the physical world like? strength size If I drop this How big are dogs? styrofoam ball into Tennis balls? Cars? the steel table, will either break?
“I am larger than a chair”
“I am larger than a chair”
“I am larger than a pen” “I am larger than a stone” “I am larger than a chair” “I am larger than a ball” “I am larger than a towel” [Grice, 1975] [Sorower et al., 2011] [Misra et al., 2016]
“The horse was as small as a dog!” ⟹ horse = size dog ?
“Hey robot, pass me the <unk>.” “OK.” (attempts to pick up table)
“I picked up the <thing>.” “I took a drink from the <thing>.” “The <thing> shattered when it hit the ground
Two related problems Physical properties implied by predicates Physical properties of objects size “I picked up the <thing>.” “I took a drink from the <thing>.” weight “The <thing> shattered when it hit the ground strength
1. Introduction 2. Related work 3. Approach 4. Model 5. Data 6. Evaluation
Pattern-based IE “ how often do [Gordon et al., 2010] you sleep?” [Gordon and Schubert, 2012] Word embeddings [Rubinstein et al., 2015] “is yellow” “is large” Commonsense knowledge base completion [Angeli and Manning, 2013] [Li et al., 2016] [Angeli and Manning, 2014] “not all birds can fly”
Verbs grounded in robotics + vision [Tellex et al., 2011] [Misra et al., 2014] [She and Chai, 2016] [Gao et al., 2016] “ cutting changes the number of pieces” Overcoming reporting Semantic proto-roles bias [Dowty, 1991] [Sorower et al., 2011] [Kako, 2006] [Misra et al., 2016] [Reisinger et al., 2015]
1. Introduction 2. Related work 3. Approach 4. Model 5. Data 6. Evaluation
Two related problems Physical properties implied by predicates Physical properties of objects size “I picked up the <unk>.” “I took a drink from the <unk>.” weight “The <unk> shattered when it hit the ground strength
Attributes x > size y x > weight y x > speed y x > strength y x < rigidness y
“I threw the _____”
“I threw the _____” ball stone chair
“I threw the _____” ball stone chair game party
“I threw the _____” ball stone chair
x threw y
x threw y x is bigger than y
x threw y x is bigger than y x weighs more than y as a result, y will be moving faster than x
Action frame x threw y ⟹ x > size y ⟹ x > weight y ⟹ x < speed y
Terminology Action frames — simple syntax-based verb constructions that compare two objects
Terminology Action frames — simple syntax-based verb constructions that compare two objects x threw y PERSON threw x into y PERSON threw on x distinct action frames for the same verb
Terminology Action frames — simple syntax-based verb constructions that compare two objects x threw y PERSON threw x into y PERSON threw on x Objects — non-abstract nouns ball evil ✘ ✓ train time ✘ ✓
Two related problems Physical properties implied by predicates Physical properties of objects size “I picked up the <thing>.” “I took a drink from the <thing>.” weight “The <thing> shattered when it hit the ground strength
Two related problems Physical properties implied by predicates Physical properties of objects Example size “I picked up the <unk>.” takes values in { > , < , ' } “I took a drink from the F = “x threw y” <unk>.” attribute: size weight correct value: > “The <unk> shattered when intuition: “x threw y” it hit the ground ⟹ x > size y strength
Two related problems Physical properties implied by predicates Physical properties of objects Example Example size “I picked up the <unk>.” takes values in { > , < , ' } takes values in { > , < , ' } “I took a drink from the F = “x threw y” = (person, ball) J p,q <unk>.” attribute: size weight attribute: size correct value: > correct value: > “The <unk> shattered when intuition: people are intuition: “x threw y” it hit the ground generally larger ⟹ x > size y than balls strength
Solving both puzzles together x threw y Action frame FRAME KNOWLEDGE
Solving both puzzles together person, ball x threw y person, stone person, chair FRAME KNOWLEDGE OBJECT KNOWLEDGE
Solving both puzzles together person, ball x threw y person > size ball ⟹ x > size y person, stone person > size stone person, chair person > size chair FRAME KNOWLEDGE OBJECT KNOWLEDGE
Solving both puzzles together person, ball x threw y person > size ball ⟹ x > size y person, stone person > size stone person, chair person > size chair FRAME KNOWLEDGE OBJECT KNOWLEDGE
OBSERVABLE IN LANGUAGE (!) person, ball x threw y person > size ball ⟹ x > size y person, stone person > size stone person, chair person > size chair FRAME KNOWLEDGE OBJECT KNOWLEDGE
1. Introduction 2. Related work 3. Approach 4. Model 5. Data 6. Evaluation
High level model OBJECT PAIRS ACTION FRAMES
High level model OBJECT PAIRS ACTION FRAMES
High level model OBJECT PAIRS ACTION FRAMES
High level model OBJECT PAIRS ACTION FRAMES
F a Random variables v t Take values in { > , < , ' } OBJECT PAIRS ACTION FRAMES
F a Random variables v t Take values in { > , < , ' } OBJECT PAIRS ACTION FRAMES F size threw 1 ≈ “ x threw y ”
F a Random variables v t { > , < , ' } Take values in OBJECT PAIRS ACTION FRAMES F size threw 1 ≈ “ x threw y ” threw 1 = > ) := p (“ x threw y ” ⇒ x > size y ) p ( F size
F a J a Random variables Random variables v t p,q { > , < , ' } { > , < , ' } Take values in Take values in OBJECT PAIRS ACTION FRAMES
F a J a Random variables Random variables v t p,q { > , < , ' } { > , < , ' } Take values in Take values in OBJECT PAIRS ACTION FRAMES J size ( person , ball) person , ball ≈
F a J a Random variables Random variables v t p,q { > , < , ' } { > , < , ' } Take values in Take values in OBJECT PAIRS ACTION FRAMES J size ( person , ball) person , ball ≈ person ,ball = > ) := p ( person > size ball) p ( J size
F a J a Random variables Random variables v t p,q { > , < , ' } { > , < , ' } Take values in Take values in OBJECT PAIRS ACTION FRAMES
F a J a Random variables Random variables v t p,q { > , < , ' } { > , < , ' } Take values in Take values in OBJECT PAIRS ACTION FRAMES
Object pair random variables J person, stone size J person, rock J person, house
J person, ψ o stone size J person, rock Object similarity J person, binary factors house
Verb similarity binary factors J person, ψ v ψ o stone v squish size J person, v throw rock J person, v walk v walk house Action frames grouped by verb
Similar frame construction binary factor F throw 1 J person, ψ v ψ o stone v squish F throw 3 size J person, ψ f v throw rock F throw 2 J person, v walk house Several action frames per verb
F throw 1 J person, ψ v ψ o stone v squish ψ s F throw 3 size J person, ψ f v throw rock F throw 2 J person, v walk house Action-object compatibility binary factors
… strength … strength v squish strength More attributes J person, stone ψ a size F throw 1 size J person, ψ v ψ o stone size v squish ψ s size size F throw 3 size J person, ψ f size v throw rock size F throw 2 size J person, size v walk house ψ a Similar attribute binary factors weight weight J person, weight v throw house … …
µ f Loopy belief propagation v ψ f µ OBJECT PAIRS ACTION FRAMES
1. Introduction 2. Related work 3. Approach 4. Model 5. Data 6. Evaluation
Why collect data? OBJECT PAIRS ACTION FRAMES
Why collect data? OBJECT PAIRS ACTION FRAMES
Why collect data? OBJECT PAIRS ACTION FRAMES - Small seed set (5%) breaks symmetry - Evaluate generalizability (dev = 45%, test = 50%)
Selecting frames and objects Verbs - took - grew - washed - trimmed “Action” verbs - squished [Levin, 1993] - got - looked - wrote - entered - kept - lived - played - …
Selecting frames and objects Verbs Action frames - took - grew … - washed - x squished y - trimmed - x squished on y - squished - PERSON squished Syntax + surface + - got crowdsourcing x with y - looked - PERSON squished - wrote x on y - entered … - kept - lived - played - …
Recommend
More recommend
