Katholieke Universiteit Leuven Computer Science Department LREC May - PowerPoint PPT Presentation

Spatial Role Labeling: Task Definition and Annotation Scheme Parisa Kordjamshidi Martijn Van Otterlo Marie-Francine Moens Katholieke Universiteit Leuven Computer Science Department LREC May 2010 1

Introduction • Problem Setting • Multimodal environment • Unrestricted language • Machine learning • Task definition (Spatial role labeling) • Lack of and difficulty of making Data • Annotation scheme Spatial Role Labeling: Task Definition and Annotation Scheme 2

Motivation 1. so from here exactly opposite is my desk. 2. and next to that left of that is my computer, perhaps a meter away. 3. (breathing) ähm. 4. next to that at the wall is my kitchen, first there is my fridge all the way to the right. Room Description [Bateman, et.al, 2006] Spatial Role Labeling: Task Definition and Annotation Scheme 3

Related schemes • SpatialML • Generalized upper model (GUM) • Geographical scheme (Q. Shen, et.al. 2009) • Spatial temporal markup(STM) (Pustejovsky and Moszkowicz, 2009) • ... Spatial Role Labeling: Task Definition and Annotation Scheme 4

Spatial role labeling task ON(book,table) Landmark Landmark Trajectory IN(table, living room) Give me that book on the table in the living room. IN(book,living room)? Spatial Indicator Spatial Indicator Trajectory Spatial Role Labeling: Task Definition and Annotation Scheme 5

Labeling the parse tree Spatial Role Labeling: Task Definition and Annotation Scheme 6

Holistic spatial semantics The reference entity in relation to which the location or motion of the trajector is The entity whose location or motion determined. is of relevance. Landmark Trajector A binary component; whether there is perceived motion or not. In terms of its beginning, middle and Motion Path end. A region of space which is Region defined in relation to a Direction landmark. The direction along the axes provided by the different frames of reference. Frame of Reference Three types of frame of reference: intrinsic, relative or absolute. Spatial Role Labeling: Task Definition and Annotation Scheme 7

Relational representation TRAJECTOR ( id, token ) LANDMARK ( id, token, path ) SPATIAL-INDICATOR ( id, token,general-type, specific-type, spatial-value ) MOTION-INDICATOR ( id, token ) SR ( id, trajector, landmark, spatial-indicator, frame-of-reference, motion-indicator ) Spatial Role Labeling: Task Definition and Annotation Scheme 8

Trajector a. She is at school. < TRAJECTOR id= ʼ 1 ʼ > She </ TRAJECTOR > TRAJECTOR( 1, she ). Spatial Role Labeling: Task Definition and Annotation Scheme 9

Landmark Path a. The balloon passed over the house. < LANDMARK id= ʼ 1 ʼ path= ʼ ZERO ʼ >the house </ LANDMARK > LANDMARK( 1, the house, ZERO ). Spatial Role Labeling: Task Definition and Annotation Scheme 10

Direction Region a. He is in front of the bush. < SPATIAL-INDICATOR id = ʼ 1 ʼ general-type = ʼ DIRECTION ʼ specific-type = ʼ RELATIVE ʼ spatial-value = ʼ FRONT ʼ > in front of< /SPATIAL-INDICATOR > SPATIAL-INDICATOR ( 1, in front of,DIRECTION, RELATIVE, FRONT ) b. John is in the room. < SPATIAL-INDICATOR id = ʼ 1 ʼ general-type = ʼ REGION ʼ specific-type = ʼ RCC8 ʼ spatial-value = ʼ TPP ʼ > in < / SPATIAL- INDICATOR > SPATIAL-INDICATOR ( 1, in ,REGION, RCC8, TPP ) Spatial Role Labeling: Task Definition and Annotation Scheme 11

Motion a.The bird flew to its nest. < MOTION-INDICATOR id = ʼ 1 ʼ > flew to </ MOTIONINDICATOR > MOTION-INDICATOR( 1, flew to ) Spatial Role Labeling: Task Definition and Annotation Scheme 12

+ Frame of Spatial relation (SR) Reference She went to school. < TRAJECTOR id= ʼ 1 ʼ > She</ TRAJECTOR > < LANDMARK id= ʼ 1 ʼ path= ʼ END ʼ > school </ LANDMARK > < SPATIAL-INDICATOR id= ʼ 1 ʼ general-type= ʼ REGION ʼ specific-type= ʼ RCC8 ʼ spatial- value= ʼ TPP ʼ > to </ SPATIALINDICATOR > < MOTION-INDICATOR id= ʼ 1 ʼ > went to </ MOTIONINDICATOR > < SR id= ʼ 1 ʼ trajector= ʼ 1 ʼ landmark= ʼ 1 ʼ spatial-indicator= ʼ 1 ʼ frame-of-reference= ʼ INTRINSIC ʼ motion-indicator= ʼ 1 ʼ /> Spatial Role Labeling: Task Definition and Annotation Scheme 13

Mapping to formal spatial relations Spatial Relations Distance Direction Region Qualitative Relative Absolute Quantitative RCC8 close .... DC TPP BELOW EC LEFT 30 km West TPP-1 .... BEHIND NTPP South RIGHT East ABOVE North PO NTPP-1 SW FRONT SW NW EQ SE Spatial Role Labeling: Task Definition and Annotation Scheme 14

Mapping to formal spatial relations Direction Region a. He is in front of the bush. < SPATIAL-INDICATOR id = ʼ 1 ʼ general-type= ʼ DIRECTION ʼ specific-type= ʼ RELATIVE ʼ spatial-value= ʼ FRONT ʼ > in front of < /SPATIAL-INDICATOR > SPATIAL-INDICATOR ( 1, in front of,DIRECTION, RELATIVE, FRONT ) b. John is in the room. < SPATIAL-INDICATOR id = ʼ 1 ʼ general-type= ʼ REGION ʼ specific-type= ʼ RCC8 ʼ spatial-value= ʼ TPP ʼ > in < / SPATIAL-INDICATOR > SPATIAL-INDICATOR ( 1, in ,REGION, RCC8, TPP ) Spatial Role Labeling: Task Definition and Annotation Scheme 15

Mapping to formal spatial relations She went to school. < TRAJECTOR id= ʼ 1 ʼ > She</ TRAJECTOR > < LANDMARK id= ʼ 1 ʼ path= ʼ END ʼ > school </ LANDMARK > <SPATIAL-INDICATOR id= ʼ 1 ʼ general-type= ʼ REGION ʼ specific-type= ʼ RCC8 ʼ spatial-value= ʼ TPP ʼ > to </SPATIALINDICATOR> <MOTION-INDICATOR id= ʼ 1 ʼ > went to </MOTIONINDICATOR > < SR id= ʼ 1 ʼ trajector= ʼ 1 ʼ landmark= ʼ 1 ʼ spatial-indicator= ʼ 1 ʼ frame-of-reference= ʼ INTRINSIC ʼ motion-indicator= ʼ 1 ʼ /> Spatial Role Labeling: Task Definition and Annotation Scheme 16

More complex descriptions Sentence level: I: Complex locative statements The vase is in the living room, on the table under the window. II: Sequential scene descriptions Behind the shops is a church, to the left of the church is the town hall, in front of the town hall is a fountain. III: Path and route descriptions The man came from between the shops, ran along the road and disappeared down the alley by the church. Discourse level: so from here exactly opposite is my desk. and next to that left of that is my computer, perhaps a meter away. (breathing) ähm, . next to that at the wall is my kitchen, first there is my fridge all the way to the right. desk computer desk in kitchen Spatial Role Labeling: Task Definition and Annotation Scheme 17

The vase is still in the living room, on the table under the window. < TRAJECTOR id = ʼ 1 ʼ > The vase < TRAJECTOR \> < LANDMARK id = ʼ 1 ʼ path = ʼ ZERO ʼ > the living room < LANDMARK \> < LANDMARK id = ʼ 2 ʼ path = ʼ ZERO ʼ > the table < LANDMARK \> < LANDMARK id = ʼ 3 ʼ path = ʼ ZERO ʼ > the window < LANDMARK \> < SPATIAL-INDICATOR id = ʼ 1 ʼ general-type = ʼ REGION ʼ specific-type = ʼ RCC8 ʼ spatial-value = ʼ NTPP ʼ > in < SPATIAL-INDICATOR \> < SPATIAL-INDICATOR id = ʼ 2 ʼ general-type = ʼ REGION ʼ specific-type = ʻ RCC8 ʼ spatial-value = ʼ EC ʼ > on < SPATIAL-INDICATOR \> < SPATIAL-INDICATOR id = ʼ 3 ʼ general-type = ʼ DIRECTION ʼ specific-type = ʻ RELATIVE ʼ spatial-value = ʼ BELOW ʼ > under < SPATIAL-INDICATOR \> < SR id= ʼ 1 ʼ trajector = ʼ 1 ʼ landmark = ʼ 1 ʼ spatial-indicator = ʼ 1 ʼ frame-of-reference = ʼ INTRINSIC ʼ motion-indicator = ʼ NIL ʼ \> < SR id= ʼ 2 ʼ trajector = ʼ 1 ʼ landmark = ʼ 2 ʼ spatial-indicator = ʼ 2 ʼ frame-of-reference = ʼ INTRINSIC ʼ motion-indicator = ʼ NIL ʼ \> < SR id=3 trajector =1 landmark =3 spatial-indicator =3 frame-of-reference =INTRINSIC motion- indicator =NIL\> Spatial Role Labeling: Task Definition and Annotation Scheme 18

Conclusion & Future directions • Conclusion • General definition of the task • Language-independent scheme • Covering spatial semantics including dynamic and static spatial information • Ongoing work and Future directions • Getting annotated corpus • Machine learning (Statistical relational learning) • Spatial reasoning, combining multimodal information Spatial Role Labeling: Task Definition and Annotation Scheme 19

Thank you ! Questions? Spatial Role Labeling: Task Definition and Annotation Scheme 20