for Efficient Adaptation in Multi-Task Learning Asa Cooper Stickland - PowerPoint PPT Presentation

Aug 27, 2022 •403 likes •501 views

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning Asa Cooper Stickland and Iain Murray University of Edinburgh Background: BERT Our model builds on BERT (Devlin et al., 2018), a powerful (and big) sentence

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning Asa Cooper Stickland and Iain Murray University of Edinburgh
Background: BERT Our model builds on BERT (Devlin et al., 2018), a powerful (and big) sentence representation model.
Background: BERT Our model builds on BERT (Devlin et al., 2018), a powerful (and big) sentence representation model. Based off the ‘transformer’ architecture, with the key component self-attention. BERT is trained on large amounts of text from the web (think: all of English wikipedia). This model can be fine-tuned on tasks with a text input. Best paper award at NAACL, 238 citations since 11/10/2018, SOTA on many tasks.
Our Approach BERT is a huge model (approx. 100 or 300 million parameters), we don’t want to store many different versions of it. Motivations: Mobile devices, web scale apps. Can we do many tasks with one powerful model?
Our Approach We consider multi-task learning on the GLUE benchmark (Wang et al, 2018), and we want the model to share most parameters but have some task-specific ones to increase flexibility. We concentrate on <1.13× ‘base’ parameters. Where should we add parameters? What form should they take?
Adapters: Basics We can add a simple linear projection down from the normal model dimension d m to d s : V E projects down to d s , we apply function g(), then V D projects back up to d m .
Adapters: PALs V E projects down to d s , we apply function g(), then V D projects back up to d m . Our PALs method shares V D and V E across all layers, so we have the ‘budget’ to make function g() be self-attention.
Experiments
Thanks! Contact me @AsaCoopStick on Twitter, or email a.cooper.stickland@ed.ac.uk. Our paper is on Arxiv, and it's called ‘BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning’. Our poster is on Wednesday at 6:30 pm, Pacific Ballroom #258.

Recommend

Multi-Task Active Learning Yi Zhang Outline Active Learning Multi-Task Active Learning

Multi-Task Active Learning Yi Zhang Outline Active Learning Multi-Task Active Learning Linguistic Annotations (ACL 08) Image Classification (CVPR 08) Current Work and Discussions Constraint-Driven Active Learning

945 views • 47 slides

Multi-Task Learning and Matrix Regularization Andreas Argyriou TTI Chicago Outline

Multi-Task Learning and Matrix Regularization Andreas Argyriou TTI Chicago Outline Multi-task learning and related problems Multi-task feature learning (trace norm, Schatten L p norms, non-convex regularizers) Representer theorems;

1.31k views • 31 slides

Using Efficient Access Control To Protect Multi-Task Execution LI Yan Background Background

Using Efficient Access Control To Protect Multi-Task Execution LI Yan Background Background Task an activity that needs to be accomplished by individual users or a team of users within a defined period of time Multi-task

743 views • 10 slides

Electricity Demand Forecasting by Multi-Task Learning Jean-Baptiste Fiot Francesco Dinuzzo IBM

1 Electricity Demand Forecasting by Multi-Task Learning Jean-Baptiste Fiot Francesco Dinuzzo IBM Research - Ireland Abstract We explore the application of kernel-based multi- of efficient and scalable training algorithms and the inter-

571 views • 8 slides

Improving historical spelling normalization with bi-directional LSTMs and multi-task learning

Problem definition Neural network approach Multi-task learning Improving historical spelling normalization with bi-directional LSTMs and multi-task learning Marcel Bollmann 1 Anders Sgaard 2 1 Ruhr-Universitt Bochum, Germany 2 University of

581 views • 29 slides

Identifying beneficial task relations for multi-task learning in deep neural networks Author:

Identifying beneficial task relations for multi-task learning in deep neural networks Author: Joachim Bingel, Anders Sogaard Presenter: Litian Ma Background Multi-task learning (MTL) in deep neural networks for

567 views • 15 slides

(Even More) Language Modeling: Multi-Task Learning, and Building Blocks of Transformers CMSC

(Even More) Language Modeling: Multi-Task Learning, and Building Blocks of Transformers CMSC 473/673 Frank Ferraro Outline Multi-Task Learning The Attention Mechanism Transformer Language Models as General Language Encoders Remember

1.33k views • 96 slides

Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Structured Losses Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning Junhyuk Oh, Satinder Singh, Honglak Lee, Pushmeet Kohli Oh et al. 2017 https://arxiv.org/abs/1706.05064 Presented by Beln Saldas belen@mit.edu

820 views • 51 slides

Multi-Task & Meta-Learning Basics CS 330 Logistics Homework 1 posted today, due Wednesday,

Multi-Task & Meta-Learning Basics CS 330 Logistics Homework 1 posted today, due Wednesday, October 9 Fill out paper preferences by tomorrow. TensorFlow review session tomorrow, 4:30 pm in Gates B03 Plan for Today Multi-Task Learning - Models

917 views • 42 slides

Multi-task Learning for Precise Object Search from Massive Images/Videos Fan Yang National

Multi-task Learning for Precise Object Search from Massive Images/Videos Fan Yang National Engineering Laboratory for Video Technology School of EE & CS, Peking University Outline Introduction Motivation Challenge Multi-task

852 views • 74 slides

Adaptive Adversarial Multi-task Representation Learning Yuren Mao 1 Weiwei Liu 2 Xuemin Lin 1 1.

ICML 2020 WHU C H I N A Adaptive Adversarial Multi-task Representation Learning Yuren Mao 1 Weiwei Liu 2 Xuemin Lin 1 1. University of New South Wales, Australia. 2. Wuhan University, China. Overview: Adaptive AMTRL (Adversarial Multi-task

503 views • 12 slides

Transfer and Multi-Task Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class

Transfer and Multi-Task Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. The project milestone is next week! 2. HW4 due tonight! 3. HW5 releases shortly (Wed or Fri) Three different options: maximum entropy

773 views • 50 slides

Multi-Task Learning & Transfer Learning Basics CS 330 1 Logistics Homework 1 posted Monday

Multi-Task Learning & Transfer Learning Basics CS 330 1 Logistics Homework 1 posted Monday 9/21 , due Wednesday 9/30 at midnight. TensorFlow review session tomorrow at 6:00 pm PT. Project guidelines posted early next week. 2 Plan for Today

996 views • 38 slides

Deep multi-task learning with evolving weights Machine learning - computer vision published in

Deep multi-task learning with evolving weights Machine learning - computer vision published in European Symposium on Artificial Neural Networks (ESANN 2016) Soufiane Belharbi Romain Hrault Clment Chatelain Sbastien Adam

631 views • 44 slides

Multi-task, Multi-lingual Learning Graham Neubig Site https://phontron.com/class/nn4nlp2018/

CS11-747 Neural Networks for NLP Multi-task, Multi-lingual Learning Graham Neubig Site https://phontron.com/class/nn4nlp2018/ Remember, Neural Nets are Feature Extractors! Create a vector representation of sentences or words for use in

515 views • 32 slides

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning Jiasen Lu April 21, 2020 1 1 Vision and Language Visual Question Answering Image Captioning Visual Commonsense Reasoning Refer Expression 2

1.13k views • 53 slides

The Peculiar Optimization and Regularization Challenges in Multi-Task Learning and Meta-Learning

The Peculiar Optimization and Regularization Challenges in Multi-Task Learning and Meta-Learning Chelsea Finn Stanford training data test datapoint Braque Cezanne By Braque or Cezanne? How did you accomplish this? Through previous

851 views • 43 slides

MOCHA: Federated Multi-Task Learning NIPS 17 Virginia Smith Stanford / CMU Chao-Kai

MOCHA: Federated Multi-Task Learning NIPS 17 Virginia Smith Stanford / CMU Chao-Kai Chiang USC Maziar Sanjabi USC Ameet Talwalkar CMU MACHINE LEARNING WORKFLOW data & problem machine learning model n optimization

1.71k views • 35 slides

Efficient and Scalable Multi-Source Streaming Broadcast on GPU Clusters for Deep Learning

Efficient and Scalable Multi-Source Streaming Broadcast on GPU Clusters for Deep Learning Ching-Hsiang Chu 1 , Xiaoyi Lu 1 , Ammar A. Awan 1 , Hari Subramoni 1 , Jahanzeb Hashmi 1 , Bracy Elton 2 and Dhabaleswar K. (DK) Panda 1 1 Department of

746 views • 28 slides

Task adjustment options: You may select one of the poems provided by your teacher, however, please

ASSESSMENT TASK NOTIFICATION _______________________ SUBJECT: English Student Name YEAR GROUP: 7 Submitted To: TASK TITLE: Multi Modal Presentation __________________________ Name of Unit: The Wonder of Words Type of Task: Multi Modal

343 views • 5 slides

Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep

Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep Multi-Task Learning (AAAI-2020) Reading Group Dec. 11, 2019 Suman Saha (postdoc), Computer Vision Lab @ ETH Zurich M o t t i i v v a a t t i

370 views • 23 slides

Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an

Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an Auto-Regressive Hidden Markov Model Xinze Guan, Raviv Raich, Weng-Keen Wong School of EECS, Oregon State University Email:

629 views • 27 slides

Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-Task Model Fusion Nghia

Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-Task Model Fusion Nghia Hoang (MIT-IBM Watson AI Lab), Thanh Lam (National University of Singapore), Bryan Kian Hsiang Low (National University of Singapore), Patrick Jaillet

521 views • 24 slides

Electricity Demand Forecasting using Multi-Task Learning Jean-Baptiste Fiot, Francesco Dinuzzo

Electricity Demand Forecasting using Multi-Task Learning Jean-Baptiste Fiot, Francesco Dinuzzo Dublin Machine Learning Meetup - July 2017 1 / 32 Outline 1 Introduction 2 Problem Formulation 3 Kernels 4 Experiments 5 Conclusion 2 / 32

420 views • 38 slides