Neural Text Summarization Piji Li NLP Center, Tencent AI Lab pijili@tencent.com Paper Reading, Sep.6, 2018 Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 1 / 63
Table of Contents Introduction 1 Methods 2 Conclusion 3 Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 2 / 63
Table of Contents Introduction 1 Methods 2 Conclusion 3 Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 3 / 63
Introduction Text Summarization The goal of automatic text summarization is to automatically produce a succinct summary, preserving the most important information for a single document or a set of documents about the same topic (event). 7/11/2017 mogren.one/graphics/illustrations/mogren_summarization.svg Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 4 / 63 http://mogren.one/graphics/illustrations/mogren_summarization.svg 1/1
Introduction Text Summarization - Categories Input: Single-Document Summarization (SDS) Multi-Document Summarization (MDS) Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 5 / 63
Introduction Single-Document Summarization Document Cambodian leader Hun Sen on Friday rejected opposition parties ' demands for talks outside the country , accusing them of trying to `` internationalize '' the political crisis . Government and opposition parties have asked King Norodom Sihanouk to host a summit meeting after a series of post-election negotiations between the two opposition groups and Hun Sen 's party to form a new government failed . Opposition leaders Prince Norodom Ranariddh and Sam Rainsy , citing Hun Sen 's threats to arrest opposition figures after two alleged attempts Summary on his life , said they could not negotiate freely in Cambodia and called for talks at Sihanouk 's residence in Beijing .Hun Sen , however , Cambodian government rejects rejected that .`` I would like to make it clear that all meetings related to Cambodian opposition's call for talks abroad affairs must be conducted in the Kingdom of Cambodia , '' Hun Sen told reporters after a Cabinet meeting on Friday .`` No-one should internationalize Cambodian affairs . It is detrimental to the sovereignty of Cambodia , '' he said .Hun Sen 's Cambodian People 's Party won 64 of the 122 parliamentary seats in July 's elections , short of the two-thirds majority needed to form a government on its own .Ranariddh and Sam Rainsy have charged that Hun Sen 's victory in the elections was achieved through widespread fraud .They have demanded a thorough investigation into their election complaints as a precondition for their cooperation in getting the national assembly moving and a new government formed ……. Figure 1: Single-document summarization. Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 6 / 63
Introduction Multi-Document Summarization Documents Summary Fingerprints and photos of two men who boarded the doomed Malaysia Airlines passenger jet are Flight MH370, carrying 239 being sent to U.S. authorities so they can be compared against records of known terrorists and people vanished over the criminals. The cause of the plane's disappearance has baffled investigators and they have not said South China Sea in less than that they believed that terrorism was involved, but they are also not ruling anything out. The investigation into the disappearance of the jetliner with 239 passengers and crew has centered so an hour after taking off from far around the fact that two passengers used passports stolen in Thailand from an Austrian and an Kuala Lumpur, with two Italian. The plane which left Kuala Lumpur, Malaysia, was headed for Beijing. Three of the passengers boarded the passengers, one adult and two children, were American. …… Boeing 777 using stolen passports. Possible reasons (CNN) -- A delegation of painters and calligraphers, a group of Buddhists returning from a religious gathering in Kuala Lumpur, a three-generation family, nine senior travelers and five could be an abrupt breakup of toddlers. Most of the 227 passengers on board missing Malaysia Airlines Flight 370 were Chinese, the plane or an act of according to the airline's flight manifest. The 12 missing crew members on the flight that terrorism. The government disappeared early Saturday were Malaysian. The airline's list showed the passengers hailed from 14 countries, but later it was learned that two people named on the manifest -- an Austrian and an was determining the "true Italian -- whose passports had been stolen were not aboard the plane. The plane was carrying five identities" of the passengers children under 5 years old, the airline said. …… who used the stolen passports. Investigators were trying to … determine the path of the plane by analysing civilian Vietnamese aircraft spotted what they suspected was one of the doors belonging to the ill-fated Malaysia Airlines Flight MH370 on Sunday, as troubling questions emerged about how two and military radar data while passengers managed to board the Boeing 777 using stolen passports. The discovery comes as ships and aircraft from seven officials consider the possibility that the plane disintegrated mid-flight, a senior source told Reuters. countries scouring the seas The state-run Thanh Nien newspaper cited Lt. Gen. Vo Van Tuan, deputy chief of staff of Vietnam's army, as saying searchers in a low-flying plane had spotted an object suspected of being a door around Malaysia and south of from the missing jet. It was found in waters about 56 miles south of Tho Chu island, in the same Vietnam. area where oil slicks were spotted Saturday. …… Figure 2: Multi-document summarization for the topic “Malaysia Airlines Disappearance”. Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 7 / 63
Introduction Text Summarization - Categories Input: Single-Document Summarization (SDS) Multi-Document Summarization (MDS) Output: Extractive Compressive Abstractive Machine learning methods: Supervised Unsupervised Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 8 / 63
Introduction Text Summarization - History Since 1950s: Concept Weight (Luhn, 1958), Centroid (Radev et al., 2004), LexRank (Erkan and Radev, 2004), TextRank (Mihalcea and Tarau, 2004), Sparse Coding (He et al., 2012; Li et al., 2015) Feature+Regression (Min et al., 2012; Wang et al., 2013) Most of the summarization methods are extractive. Abstractive summarization is full of challenges. Some indirect methods employ sentence fusing (Barzilay and McKeown, 2005) or phrase merging (Bing et al., 2015). The indirect strategies will do harm to the linguistic quality of the constructed sentences. Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 9 / 63
Introduction Text Summarization - History Before the neural summarization era...silent 2012 2015 (Rush et al., 2015) Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 10 / 63
Table of Contents Introduction 1 Methods 2 Conclusion 3 Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 11 / 63
Methods Essential Idea Salience Detection (Words, Sentences) Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 12 / 63
Methods Inspiration from DBN, DNN, CNN Liu, Yan, Sheng-hua Zhong, and Wenjie Li. “Query-Oriented Multi-Document Summarization via Unsupervised Deep Learn- ing.” In AAAI. 2012. Denil, Misha, Alban Demiraj, Nal Kalchbrenner, Phil Blunsom, and Nando de Freitas. “Modelling, visualising and summarising doc- uments with a single convolu- tional neural network.” arXiv preprint arXiv:1406.3830 (2014). Figure 3: Visualization of Parameters. Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 13 / 63
Methods Better Semantic Representations Since 1950s: Concept Weight (Luhn, 1958), Centroid (Radev et al., 2004), LexRank (Erkan and Radev, 2004), TextRank (Mihalcea and Tarau, 2004), Sparse Coding (He et al., 2012; Li et al., 2015) Bag-of-Words (BoWs) Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 14 / 63
Methods Better Semantic Representations Word2vec (Mikolov et al., 2013), Paragraph Vector (Le and Mikolov, 2014), RNN-Sent (Tang et al., 2015), CNN-Sent (Kim, 2014) Improve the performance of PageRank and Data Reconstruction based models. Works: K˚ ageb¨ ack, Mikael, Olof Mogren, Nina Tahmasebi, and Devdatt Dub- hashi. “ Extractive summarization using continuous vector space models .” In CVSC 2014. Yin, Wenpeng, and Yulong Pei. ” Optimizing Sentence Modeling and Selection for Document Summarization .” In IJCAI 2015. Li, Piji, Wai Lam, Lidong Bing, Weiwei Guo, and Hang Li. ” Cascaded attention based unsupervised information distillation for compres- sive summarization .” In EMNLP 2017. Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 15 / 63
Methods Inspiration from NMT Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. ” Neural machine translation by jointly learning to align and translate .” arXiv preprint arXiv:1409.0473 (2014). (citation:4300+) Figure 4: Attention-based seq2seq framework. Figure from OpenNMT (Klein et al., 2017) . Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 16 / 63
Methods 2015 Piji Li (Tencent AI Lab) Neural Text Summarization Sep.6, 2018 17 / 63
Recommend
More recommend
