Spatial Cognition 2016 MULTIMODAL SEMANTIC SIMULATIONS OF LINGUISTICALLY UNDERSPECIFIED MOTION EVENTS Nikhil Krishnaswamy and James Pustejovsky, Brandeis University August 5, 2016, Spatial Cognition 2016, Philadelphia, PA, USA
● Remarkable number of concepts in human mental model Spatial Cognition 2016 ● Mental models are adaptable ● Can make sense of new situations, contexts, and kinds of knowledge ● Can be revised based on new experience ● Mental models are embodied and multimodal ● Embodiment maps concepts between domains Foundations ● Modalities (perceptual and effector) constitute aspects of representation ● “Simulation”: mental instantiation of an utterance, based on embodiment 2 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
● Spatial/temporal algebraic interval logic Spatial Cognition 2016 ● Allen Temporal Relations (Allen, 1983) ● Region Connection Calculus (RCC8) (Randell et al., 1992) ● RCC-3D (Albath, et al., 2010) Past/Related Research ● Generative Lexicon, DITL (Pustejovsky, 1995; Pustejovsky and Moszkowicz, 2011) ● Static scene generation ● WordsEye (Coyne and Sproat, 2001) ● LEONARD (Siskind, 2001) ● Stanford NLP Group (Chang et al., 2015) ● QSR/Game AI approaches to scenario-based simulation (Forbus et al., 2001; Dill, 2011) ● Spatial constraint mapping to animation (Bindiganavale and Badler, 1998) 3 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 emporal Relations T Allen 4 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 Region Connection Calculus 5 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 WordsEye 6 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 Cognitive Linguistic Simulation “Enter p the parking lot” Path depends on bounds of parking lot “Enter” is a path verb (Pustejovsky and Moszkowicz, 2011) 7 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 Cognitive Linguistic Simulation “Hurry m to the car” Path depends on location of car “Hurry” is a manner of motion verb (Pustejovsky and Moszkowicz, 2011) 8 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
● Path verbs designate a distinguished Spatial Cognition 2016 value in the state-to-state location change ● Change in value is tested ● Manner of motion verbs iterate a state- Events as Programs to-state location change ● Change in value is assigned /reassigned ● Verbs can be realized as programs enacted over arguments (Naumann, 1999) 9 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 ● Programs are compositional ● Program’s linguistic representation can be broken down into subevents Events as Programs ● Simple programs ● translocate, rotate, grasp, hold, release, etc. ● Complex programs ● put, stack, etc. 10 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
● VoxML: Visual Object Concept Modeling Language Spatial Cognition 2016 (Pustejovsky and Krishnaswamy, 2016) ● Annotation and modeling language for “voxemes” ● Visual instantiation of a lexeme ● Scaffold for mapping from lexical information to simulated objects and operationalized behaviors ● Encodes afforded behaviors for each object ● Gibsonian - afforded by object structure (e.g. grasp, move, lift) (Gibson, 1977; 1979) VoxML ● Telic - goal-directed, purposeful (e.g. drink from) (Pustejovsky, 1995) 11 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 VoxML 12 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 VoxML 13 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 VoxSim: Software Architecture We begin by inpu+ng a sentence in plain English Put the spoon in the mug 14 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 VoxSim: Software Architecture From a dependency parse, we extract labeled en<<es in the scene, and verbs those en<<es may afford put Voxeme: PROGRAM Put the spoon in spoon Voxeme: OBJECT the mug Voxeme: [in] mug RELATION(OBJECT) 15 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 VoxSim: Software Architecture Resolve the parsed sentence into a predicate-logic formula put(x,y) Voxeme: PROGRAM put x := spoon put(spoon,in(mug)) Voxeme: OBJECT spoon Voxeme: y := in(z) RELATION(OBJECT) z: = mug [in] mug 16 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 VoxSim: Software Architecture Each predicate is opera<onalized ● in(z): takes object, according to its type structure outputs location ● put(x,y): path verb ● while(!at(y), move(x)) put(spoon,in(mug)) 17 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
● Object bounds may not Spatial Cognition 2016 contour to geometry ● e.g. Concave objects ● Semantic information Semantic Processing imposes further constraints ● “in cup”: (PO | TPP | NTPP) with area denoted by cup’s interior ● Interpenetrates bounds, but not geometry 18 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 ● Can test be satisfied with current object configuration? Semantic Processing ● Can test be satisfied by reorienting objects? ● Can test be satisfied at all? 19 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 Rig Attachment ● Temporary parent-child relationship between joint on rig and manipulated object ● Allows agent and object to move together ● “Object model” + “Action model” = “Event model” 20 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 Demo 21 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 ● Platform for incorporating motion/dynamic semantics into visualization ● Visualization → Simulation → Minimal Model ● Runtime visualization generation necessitates assigning values in the simulation to parameters unspecified in minimal model ● e.g. speed, direction, etc. Discussion ● Complete set of primitive programs in a particular domain unknown 22 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
● Monte-Carlo simulation generation with multiple Spatial Cognition 2016 evaluation tasks ● Given visualization with randomly-assigned underspecified variables, choose best description ● Given description, choose best visualization from randomly-generated set Work ● Automatic evaluation of actual simulation result vs. DITL-derived satisfaction conditions Future ● Corpus building for linked videos and simulations with event labels for machine learning of event classification 23 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Spatial Cognition 2016 Acknowledgments Brandeis University Student Workers Jessica Huynh Paul Kang Subahu Rayamajhi Amy Wu Beverly Lum Victoria Tran 24 August 5, 2016 Philadelphia, PA, USA Nikhil Krishnaswamy | nkrishna@brandeis.edu
Recommend
More recommend
