7/13/2012 NLP: Two pictures Wordnet and Word Sense Problem NLP Disambiguation Semantics NLP Trinity Parsing Part of Speech Lecture delivered at the summer school on NLP, IIIT Hyderabad, Tagging 10July, 2012 Vision Speech Morph Analysis Marathi French by HMM Pushpak Bhattacharyya Statistics and Probability Hindi English Computer Science and Engineering Department Language CRF + IIT Bombay MEMM Knowledge Based {pb}@cse.iitb.ac.in Algorithm NLP: Thy Name is Disambiguation • I went with my friend to the bank to withdraw some money, but was disappointed to find it closed • POS disambiguation: bank (NN/VM), withdraw (NN/VM), closed Background (JJ/VM) • Sense Disambiguation: bank (finance/place), withdraw (take out/go_away) • Co-reference disambiguation: it (bank/money)/I/friend) • Pro-drop disambiguation: “… <I/friend> was disappointed…” • Scope disambiguation: for with (my _friend/my_friend_to_the bank) 1
7/13/2012 Where there is a will Where there is a will, There are hundreds of There is a way relatives Stages of processing • Phonetics and phonology • Morphology • Lexical Analysis Where there is a will, • Syntactic Analysis • Semantic Analysis • Pragmatics There are hundreds of relatives • Discourse 2
7/13/2012 Lexical Disambiguation Ambiguity of Multiwords First step: part of Speech Disambiguation • The grandfather kicked the bucket after suffering from cancer. • Dog as a noun (animal) • Dog as a verb ( to pursue) • This job is a piece of cake Sense Disambiguation • Put the sweater on • Dog ( as animal) • He is the dark horse of the match • Dog ( as a very detestable person) Needs word relationships in a context Google Translations of above sentences: • The chair emphasised the need for adult education दादा क � सर से पी�ड़त होने क े बाद बा�ट� लात मार� . े क का एक टुकड़ा है . Very common in day to day communications इस काम क े क Satellite Channel Ad: Watch what you want, when you want ःवेटर पर रखो . (two senses of watch) े अंधेरे घोड़ा है . वह मैच क e.g., Ground breaking ceremony/research Lexical Analysis Ambiguity of Named Entities • Essentially refers to dictionary access and • Bengali: ��� ����� ������ ��� obtaining the properties of the word English: Government is restless at home . (*) e.g. dog Chanchal Sarkar is at home noun (lexical property) take-’s’-in-plural (morph property) • Hindi: दैिनक दबंग दुिनया animate (semantic property) English: everyday bold world 4-legged (-do-) Actually name of a Hindi newspaper in Indore carnivore (-do) • High degree of overlap between NEs and MWEs Challenge: Lexical or word sense disambiguation • Treat differently - transliterate do not translate 3
7/13/2012 Challenges in Syntactic Textual Humour (2/2) Processing: Structural Ambiguity • Scope • It is not hard to meet the expenses now a day, you find 1.The old men and women were taken to safe locations them everywhere (old men and women) vs. ((old men) and women) 2. No smoking areas will allow Hookas inside • Preposition Phrase Attachment • Teacher: What do you think is the capital of Ethiopia? Student: What do you think? • I saw the boy with a telescope (who has the telescope? ) Teacher: I do not think, I know • I saw the mountain with a telescope Student: I do not think I know (world knowledge: mountain cannot be an instrument of seeing ) • I saw the boy with the pony-tail (world knowledge: pony-tail cannot be an instrument of seeing ) Very ubiquitous: newspaper headline “ 20 years later, BMC pays father 20 lakhs for causing son’s death” Textual Humour (1/2) Example of WSD Operation , surgery, surgical operation, surgical procedure, surgical 1. Teacher (angrily): did you miss the class yesterday? • process -- (a medical procedure involving an incision with instruments; Student: not much performed to repair damage or arrest disease in a living body; "they will schedule the operation as soon as an operating room is available"; "he 2. A man coming back to his parked car sees the died while undergoing surgery") TOPIC->(noun) surgery#1 sticker "Parking fine". He goes and thanks the Operation , military operation -- (activity by a military or naval force (as • policeman for appreciating his parking skill. a maneuver or campaign); "it was a joint operation of the navy and air force") TOPIC->(noun) military#1, armed forces#1, armed services#1, 3. Son : mother, I broke the neighbour's lamp shade. military machine#1, war machine#1 Mother : then we have to give them a new one. Operation -- ((computer science) data processing in which the result is • completely specified by a rule (especially the processing that results Son : no need, aunty said the lamp shade is from a single instruction); "it can perform millions of operations per irreplaceable. second") TOPIC->(noun) computer science#1, computing#1 mathematical process, mathematical operation , operation -- • 4. Ram : I got a Jaguar car for my unemployed ((mathematics) calculation by mathematical methods; "the problems at youngest son. the end of the chapter demonstrated the mathematical processes involved in the derivation"; "they were learning the basic operations of Shyam : That's a great exchange! arithmetic") TOPIC->(noun) mathematics#1, math#1, maths#1 5. Shane Warne should bowl maiden overs, instead of bowling maidens over IS WSD NEEDED IN LARGE APPLICATIONS? 4
7/13/2012 Word ambiguity � topic drift in IR How about WSD and MT? Zaheer Khan, the India fast bowler, भारत क े तेज ग�दबाज , जह�र खान , इं�ल�ड क has been ruled out of the remainder े Drifted topic due to inapplicable sense!!! {case, container} �खलाफ ौृंखला क े शेष क े बाहर शासन �कया of the series against England. गया है . (ruled in the administrative sense??) He will return to India and will be Query word: replaced by left-arm seamer RP वह भारत लौटने और बाएँ हाथ क े तेज ग�दबाज “Madrid bomb blast case” Singh. आरपी िसंह �ारा ूितःथा�पत �कया जाएगा . {case, suit, lawsuit} जह�र लॉ�स� म� पहले टेःट क े दौरान हैम�ःशंग Zaheer picked up a hamstring injury चोट उठाया . (lifted??) during the first Test at Lord's. Drifted topic due to expanded term!!! He had been withdrawn from the {suit, apparel} वह भारत क� वेःट इंड�ज म� हाल ह� म� एक सह� squad for India's recent Test series (correct??) टखने क� चोट क े कारण टेःट in the West Indies due to a right ौृंखला क े िलए ट�म से वापस ले िलया गया था . ankle injury. 50 Our observations 45 43.75 43.75 On error Percentages 40 Due to various Factors CLEF 2007 35 31.25 30 Wordnet Error Percentage Transliteration Translation Disambiguation 25 25 Stemmer Dictionary Ranking 20 18.75 15 12.5 10 6.25 5 0 0 Hindi-English Marathi-English 5
7/13/2012 Wordnet: History Psycholinguistic Theory • Human lexical memory for nouns as a hierarchy. • The first wordnet in the world was for English developed • Can canary sing? - Pretty fast response. at Princeton over 15 years. • Can canary fly? - Slower response. • Does canary have skin? – Slowest response. • The Eurowordnet- linked structure of European language wordnets was built in 1998 over 3 years with funding (can move, has skin) from the EC as a a mission mode project. Animal • Wordnets for Hindi and Marathi being built at IIT Bombay are amongst the first IL wordnets. • All these are proposed to be linked into the (can fly) Bird IndoWordnet which eventually will be linked to the English and the Euro wordnets. (can sing) canary Wordnet - a lexical reference system based on psycholinguistic theories of human lexical memory. Essential Resource for WSD: Basic Principle Wordnet Word Forms • Words in natural languages are polysemous. Word Meanings F 1 F 2 F 3 … F n • However, when synonymous words are put (rely) ( bank ) ( depend ) E 1,3 E 1,2 together, a unique meaning often emerges. M 1 E 1,1 • Use is made of Relational Semantics. ( embankme nt ) (bank) M 2 E 2,… E 2,2 (bank) E 3,2 E 3,3 M 3 … … E m,n M m 6
Recommend
More recommend
