Normalizing tweets with edit scripts and recurrent neural - PowerPoint PPT Presentation

Nov 01, 2023 •167 likes •373 views

Normalizing tweets with edit scripts and recurrent neural embeddings Grzegorz Chrupaa | Tilburg University Normalizing tweets Convert tweets to canonical form easy to understand for downstream applications Examples I will c wat i can do

Normalizing tweets with edit scripts and recurrent neural embeddings Grzegorz Chrupała | Tilburg University
Normalizing tweets
Convert tweets to canonical form easy to understand for downstream applications
Examples I will c wat i can do I will see what I can do imma jus start puttn it out there I'm going to just start putting it out there
Approaches ● Noisy-channel-style ● Finite-state transducers ● Dictionary-based ○ Hand-crafted ○ Automatically constructed
Labeled vs unlabeled data ● Noisy-channel: P(target|source) ∝ P(source|target) × P(target) labeled unlabeled ● Dictionary lookup: ○ Induce dictionary from unlabeled data ○ Labeled data for parameter tuning
Discriminative model target = argmax target P(diff( source, target ) | source ) ● diff(·,·) transforms source to target ● P(·) is a Conditional Random Field
Signal from raw tweets included via learned text representations .
Architecture
Simple Recurrent Networks Elman, J. L. (1990). Finding structure in time. Cognitive science , 14 (2), 179-211.
Recurrent neural embeddings ● SRN trained to predict next character ● Representation: ● Embed string (at each position) in low- dimensional space
Visualizing embeddings String Nearest neighbors in embedding space should h should d will s will m should a @justth @neenu @raven_ @lanae @despic maybe u maybe y cause i wen i when i
diff - Edit script Input c _ w a t diff DEL INS(see) NIL INS(h) NIL Output see_ w ha t Each position in string labeled with edit op
Features ● Baseline n-gram features c _ w a t c_ _w wa at c_w _wa wat c_wa _wat c_wat ● SRN features ○ 400 MB raw Twitter feed ○ 400 hidden units ○ Activations discretized
Dataset ● Han, B., & Baldwin, T. (2011). Lexical normalisation of short text messages: Makn sens a# twitter. In ACL . ● 549 tweets, with normalized versions ● Only lexical normalizations
Results ● No-op make no changes ● Doc train on and label whole tweets ● OOV train on and label OOV-words
Compared to Han & Bo 2012 Method WER (%) No-op 11.2 S-dict 9.7 GHM-dict 7.6 HB-dict 6.6 Dict-combo 4.9 OOV NGRAM+SRN 4.7
Where SRN features helped 9 cont continued 5 gon gonna 4 bro brother 4 congrats congratulations 3 yall you 3 pic picture 2 wuz what’s 2 mins minutes 2 juss just 2 fb facebook
Conclusion ● Supervised discriminative model performs at state-of-the-art with little training data ● Neural text embeddings effectively incorporate signal from raw tweets

Recommend

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I : Recurrent Neural Networks CHAPTER I Recurrent Neural Networks Introduction In this chapter first the

404 views • 27 slides

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks Stephan Oepen University of Oslo March 24, 2019 Our Roadmap Last Week Language structure: sequences, trees, graphs Recurrent Neural

217 views • 19 slides

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer University of Washington [most slides from Yejin Choi] RECURRENT NEURAL L NE NETWOR WORKS Recurrent Neural Networks (RNNs) Each input

787 views • 29 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

793 views • 21 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

583 views • 47 slides

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views • 18 slides

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent Neural Networks Compact schematic drawing of standard multi-layer perceptron (MLP) output hidden input Modeling sequential data So far we

796 views • 50 slides

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova March 9, 2020 Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/

538 views • 18 slides

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random Variables Text Preprocessing Language Modeling Recurrent Neural Networks (RNN) LSTM Bidirectional RNN Deep RNN d2l.ai

798 views • 55 slides

RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

Outline Recurrent Neural Networks (RNNs) Neural Machine Transla/on NMT basics (Sutskever et al., 2014) ABenCon mechanism (Bahdanau et al., 2015) Marcello Federico 2016 Recurrent Neural Networks (RNNs) Based on slides kindly provided

437 views • 24 slides

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang,

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, Huamin Qu. THE HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY What is a Recurrent Neural Network? H K U S T

397 views • 37 slides

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

Recurrent Neural Network Agenda Recurrent Neural Network

Recurrent Neural Network Agenda Recurrent Neural Network NER sequence tagging RNN

670 views • 46 slides

The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente

The Power of Linear Recurrent The Power of Linear Recurrent Neural Networks Neural Networks Was knnen lineare rekurrente neuronale Netze? Frieder Stolzenburg Overview Frieder Stolzenburg Introduction Recurrent Neural Networks

1.17k views • 66 slides

INF5820: Language Technological Applications Applications of Recurrent Neural Networks Stephan

INF5820: Language Technological Applications Applications of Recurrent Neural Networks Stephan Oepen University of Oslo November 6, 2018 Most Recently: Variants of Recurrent Neural Networks To a first approximation, the de facto consensus in

339 views • 23 slides

Machine Learning for Computational Linguistics Recurrent neural networks (RNNs) ar

Machine Learning for Computational Linguistics Recurrent neural networks (RNNs) ar ltekin University of Tbingen Seminar fr Sprachwissenschaft July 5, 2016 Neural networks: a quick summary Recurrent neural networks July 5, 2016

570 views • 35 slides

Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Sequential Data with Neural Networks Recurrent Neural

303 views • 4 slides

NLP Programming Tutorial 8 - Recurrent Neural Nets Graham Neubig Nara Institute of Science and

NLP Programming Tutorial 8 Recurrent Neural Nets NLP Programming Tutorial 8 - Recurrent Neural Nets Graham Neubig Nara Institute of Science and Technology (NAIST) 1 NLP Programming Tutorial 8 Recurrent Neural Nets Feed Forward Neural

604 views • 34 slides