using deep learning to rank and tag
play

Using Deep Learning to rank and tag 6,30 5,90 millions of hotel - PowerPoint PPT Presentation

Jan 09, 2023 •141 likes •1.02k views

0,00 9,80 16,10 16,10 7,40 6,94 Using Deep Learning to rank and tag 6,30 5,90 millions of hotel images 15/11/2018 - PyParis 2018 0,00 Christopher Lennan (Senior Data Scientist) @chris_lennan Tanuj Jain (Data Scientist) @tjainn

  1. 0,00 9,80 16,10 16,10 7,40 6,94 Using Deep Learning to rank and tag 6,30 5,90 millions of hotel images 15/11/2018 - PyParis 2018 0,00 Christopher Lennan (Senior Data Scientist) @chris_lennan Tanuj Jain (Data Scientist) @tjainn #idealoTech 8,10 1 0,32 0,32

  2. 0,00 9,80 16,10 16,10 Agenda 7,40 6,94 idealo.de 1. 6,30 5,90 Business Motivation 2. Models and Training 3. Image Tagging 4. Image Aesthetics 5. 0,00 Summary 6. 8,10 2 0,32 0,32

  3. 0,00 9,80 16,10 16,10 Some Key Facts 7,40 6,94 6,30 5,90 18 More than 18 years 16 million users/month 1 experience Germany's 4th largest 50.000 shops eCommerce website 0,00 Active in 6 different countries Over 330 million offers for 2 (DE, AT, ES, IT, FR, UK) million products 700 “idealos” from 40 Tüv certified comparison portal 2 nations 8,10 0,32 0,32

  4. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 Motivation 0,00 8,10 4 0,32 0,32

  5. 0,00 9,80 16,10 16,10 idealo hotel price comparison 7,40 hotel.idealo.de 6,94 6,30 5,90 2.306.658 accommodations ● 308.519.299 images ● 0,00 ~ 133 images per ● accommodation 8,10 5 0,32 0,32

  6. 0,00 9,80 16,10 16,10 Importance of Photography for Hotels 7,40 6,94 6,30 5,90 “.. after price, photography is the most important factor for travelers and prospects scanning OTA sites..” “.. Photography plays a role of 60% in the decision to book with a particular hotel ..” 0,00 “.. study published today by TripAdvisor , it would seem like photos have the greatest impact driving engagement from travelers r esearching on hotel and B&B pages ..” 8,10 6 0,32 0,32

  7. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 0,00 8,10 7 0,32 0,32

  8. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 0,00 8,10 8 0,32 0,32

  9. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 0,00 8,10 9 0,32 0,32

  10. 1 2 3 4 5 6 7 8 9 10 11 12 13

  11. 0,00 9,80 16,10 16,10 Image Aesthetics 7,40 6,94 Current image placement 6,30 5,90 Position: 19 Position: 1 0,00 8,10 11 0,32 0,32

  12. 0,00 9,80 16,10 16,10 Image Aesthetics 7,40 6,94 Current image placement 6,30 5,90 Position: 17 Position: 3 0,00 8,10 12 0,32 0,32

  13. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 Beautiful images should 0,00 appear earlier in the gallery 8,10 13 0,32 0,32

  14. 1 2 3 4 5 6 7 8 9 10 11 12 13

  15. 1 2 3 4 5 6 7 8 9 10 11 12 13

  16. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 Ensure different areas get 0,00 depicted 8,10 16 0,32 0,32

  17. Bathroom Bedroom 1 2 4 3 Restaurant Facade Fitness Studio Kitchen 6 5 7 8

  18. 0,00 9,80 16,10 16,10 Understanding Image Content 7,40 Two part problem 6,94 6,30 5,90 Tag the image with the hotel property area 1. Predict aesthetic quality 2. 0,00 8,10 18 0,32 0,32

  19. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 Models & Training 0,00 8,10 19 0,32 0,32

  20. 0,00 9,80 16,10 16,10 Transfer Learning 7,40 6,94 6,30 5,90 Use pre-trained CNN that was trained on millions of images ● (e.g. MobileNet or VGG16) Replace top layers so that the output fits with classification task ● 0,00 Train existing and new layer weights ● 8,10 20 0,32 0,32

  21. 0,00 9,80 16,10 16,10 Transfer Learning 7,40 6,94 CNN architecture (VGG16) 6,30 5,90 0,00 8,10 21 0,32 0,32

  22. 0,00 9,80 16,10 16,10 Training regime 7,40 6,94 6,30 5,90 Only train the newly added dense layers with high learning rate 1. Then train all layers with low learning rate 2. Goal: Do not juggle around the pre-trained convolutional weights too much 0,00 8,10 22 0,32 0,32

  23. 0,00 9,80 16,10 16,10 Training regime 7,40 6,94 6,30 5,90 0,00 8,10 23 0,32 0,32

  24. 0,00 9,80 16,10 16,10 Loss functions 7,40 6,94 Cross-entropy loss (CEL) 6,30 5,90 CEL generally used for “one-class” ground truth classifications (e.g. image tagging) ● CEL ignores inter-class relationships between score buckets ● 0,00 8,10 24 source: https://ssq.github.io/2017/02/06/Udacity%20MLND%20Notebook/ 0,32 0,32

  25. 0,00 9,80 16,10 16,10 Loss functions 7,40 6,94 Earth Mover’s Distance (EMD) 6,30 5,90 For ordered classes, classification settings can outperform regressions ● Training on datasets with intrinsic ordering can benefit from EMD loss objective ● 0,00 8,10 25 0,32 0,32

  26. 0,00 9,80 16,10 16,10 GPU training workflow 7,40 6,94 Local AWS 6,30 5,90 build push ECR Custom Setup AMI datasets Docker Dockerfile nvidia-docker image pull image Docker launch Machine 0,00 copy existing model EC2 S3 Train GPU store train outputs launch training container instance with nvidia-docker train script Docker launch copy existing model Machine EC2 Evaluate launch evaluation container GPU with nvidia-docker instance pull image SSH evaluation script 8,10 26 Jupyter notebook 0,32 0,32

  27. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 Image Tagging 0,00 8,10 27 0,32 0,32

  28. 0,00 9,80 16,10 16,10 Tagging Problem 7,40 6,94 6,30 5,90 Given an image, tag it as belonging to a single class ● Multiclass classification model with classes: ● Bedroom ○ Bathroom ○ 0,00 Foyer ○ Restaurant ○ Swimming Pool ○ Kitchen ○ View of Exterior (Facade) ○ Reception ○ 8,10 28 0,32 0,32

  29. 0,00 9,80 16,10 16,10 Multiple Datasets 7,40 6,94 6,30 5,90 Will go over them one-by-one and see: Dataset properties ● Results ● Issues ● 0,00 8,10 29 0,32 0,32

  30. 0,00 9,80 16,10 16,10 Wellness Dataset 7,40 6,94 6,30 5,90 Idealo in-house pre-labelled images ● Mostly pictures of 2 or 3 stars properties ● 0,00 8,10 30 0,32 0,32

  31. 0,00 9,80 16,10 16,10 Wellness Dataset 7,40 6,94 6,30 5,90 Balanced: Equal sample count in ● all categories for all sets 0,00 8,10 31 0,32 0,32

  32. 0,00 9,80 16,10 16,10 Wellness Dataset: Metrics 7,40 6,94 6,30 5,90 Top-1- accuracy: 86% 0,00 8,10 32 0,32 0,32

  33. 0,00 9,80 16,10 16,10 Wellness Dataset: Wrong Predictions 7,40 True Class of these images: BATHROOM, Predicted as: RECEPTION 6,94 6,30 5,90 0,00 Rectangular structure = Reception with high probability → BIAS! 8,10 33 0,32 0,32

  34. 0,00 9,80 16,10 16,10 Wellness Dataset: Wrong Predictions 7,40 6,94 True Class of these images: BATHROOM 6,30 5,90 0,00 Wrong true label of images → NOISE in the dataset! 8,10 34 0,32 0,32

  35. 0,00 9,80 16,10 16,10 Correcting Bias 7,40 6,94 6,30 5,90 Augmentation operations, same for every class: ● Random cropping ○ Rotation ○ Horizontal flipping ○ Data enrichment : ● 0,00 External data from google images ○ 8,10 35 0,32 0,32

  36. 0,00 9,80 16,10 16,10 Augmented Wellness + Google Dataset: Metrics 7,40 6,94 6,30 5,90 Top-1- accuracy: 88% 0,00 8,10 36 0,32 0,32

  37. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 Gotta Clean! 0,00 8,10 37 0,32 0,32

  38. 0,00 9,80 16,10 16,10 Cleaning Dataset 7,40 6,94 6,30 5,90 Hand-cleaned each category: ● Deleted pictures that do not belong in its category ○ Removed duplicates (presence of duplicates can give us wrong ○ metrics) 0,00 Added more images from external sources for classes with a small ○ number of images left after cleaning 8,10 38 0,32 0,32

  39. 0,00 9,80 16,10 16,10 Cleaned Data: Metrics 7,40 6,94 6,30 5,90 Top-1- accuracy: 91% 0,00 8,10 39 0,32 0,32

  40. 0,00 9,80 16,10 16,10 Cleaned Dataset: Results 7,40 6,94 6,30 5,90 Bathroom vs. Reception confusion has almost vanished! ● View_of_exterior vs Pool confusion has reduced ● Foyer performance: ● Most misclassifications of Foyer gets assigned to ○ 0,00 Reception This is human problem as well! ○ 8,10 40 0,32 0,32

  41. 0,00 9,80 16,10 16,10 Foyer or Reception? 7,40 6,94 6,30 5,90 0,00 8,10 41 0,32 0,32

  42. 0,00 9,80 16,10 16,10 Learnings so far 7,40 6,94 6,30 5,90 The model can only be as good as the data (cleaning) ● Foyer is a hard category to predict ● 0,00 8,10 42 0,32 0,32

  43. 0,00 9,80 16,10 16,10 7,40 6,94 6,30 5,90 Understanding Model Decisions 0,00 8,10 43 0,32 0,32

  44. 0,00 9,80 16,10 16,10 Understanding Decisions: Class Activation Maps 7,40 6,94 6,30 Use the penultimate Global Average Pooling Layer (GAP) to get class activation map ● 5,90 Highlights discriminative region that lead to a classification ● 0,00 8,10 44 0,32 0,32

  45. 0,00 9,80 16,10 16,10 Insights With CAM 7,40 Swimming Pool misclassified as Bathroom 6,94 6,30 5,90 CAM 0,00 8,10 45 0,32 0,32

  46. 0,00 9,80 16,10 16,10 Insights With CAM 7,40 Swimming Pool misclassified as Bathroom 6,94 6,30 5,90 CAM 0,00 8,10 46 0,32 0,32

  47. 0,00 9,80 16,10 16,10 Insights With CAM 7,40 Swimming Pool misclassified as Bathroom 6,94 6,30 5,90 CAM 0,00 8,10 47 0,32 0,32

  48. 0,00 9,80 16,10 16,10 Insights With CAM 7,40 Swimming Pool misclassified as Bathroom 6,94 6,30 5,90 0,00 Using rails to misidentify Pool as Bathroom. 8,10 48 0,32 0,32

  49. 0,00 9,80 16,10 16,10 Insights With CAM 7,40 Bathroom correct classification 6,94 6,30 5,90 CAM 0,00 8,10 49 0,32 0,32

Recommend

Deep Reinforcement Le Learning for Me Menti tion on-Ra Rank nking ng Cor Coreference Mod

Deep Reinforcement Le Learning for Me Menti tion on-Ra Rank nking ng Cor Coreference Mod

Deep Reinforcement Le Learning for Me Menti tion on-Ra Rank nking ng Cor Coreference Mod Models Kevin Clark and Christopher D. Manning Stanford University Presented by Zubin Pahuja Coreference Resolution Identify all mentions that

744 views • 48 slides
10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise, Listwise 10.3. Gathering User Input 10.4. LeToR Evaluation 10.5. Beyond Search Advanced Topics in Information Retrieval / Learning to Rank 2 10.1.

1.01k views • 34 slides
2 3 4 5 8 9 MINNEAPOLIS MILWAUKEE MSA RANK #16 MSA RANK #39 CHICAGO MSA RANK #3

2 3 4 5 8 9 MINNEAPOLIS MILWAUKEE MSA RANK #16 MSA RANK #39 CHICAGO MSA RANK #3

2 3 4 5 8 9 MINNEAPOLIS MILWAUKEE MSA RANK #16 MSA RANK #39 CHICAGO MSA RANK #3 INDIANAPOLIS MSA RANK #34 DETROIT MSA RANK #14 COLUMBUS MSA RANK #33 BALTIMORE/ DC CORRIDOR MSA RANK #21 CINCINNATI MSA RANK #28 ATLANTA MSA RANK

726 views • 44 slides
Learning to Rank Learning to Rank with Partially-Labeled Data with Partially-Labeled Data Kevin

Learning to Rank Learning to Rank with Partially-Labeled Data with Partially-Labeled Data Kevin

Learning to Rank Learning to Rank with Partially-Labeled Data with Partially-Labeled Data Kevin Duh University of Washington (Joint work with Katrin Kirchhoff) 1 Motivation Motivation Machine learning can be an effective solution for

472 views • 29 slides
Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides
ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY MEDIA & ENTERTAINMENT SECURITY & DEFENSE AUTONOMOUS MACHINES Image

308 views • 26 slides
Introduction to Information Retrieval http://informationretrieval.org IIR 15-2: Learning to Rank

Introduction to Information Retrieval http://informationretrieval.org IIR 15-2: Learning to Rank

Machine-learned relevance Learning to rank Introduction to Information Retrieval http://informationretrieval.org IIR 15-2: Learning to Rank Hinrich Sch utze Institute for Natural Language Processing, Universit at Stuttgart 2011-08-29

1.3k views • 115 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY SECURITY & DEFENSE MEDIA &

1.45k views • 39 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI Mission Helping people solve challenging problems using AI and deep learning. Developers, data scientists and engineers Self-driving cars,

1.44k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin Clark Deep Learning for NLP 5 years ago No Seq2Seq No Attention No large-scale QA/reading comprehension datasets No TensorFlow or

1.41k views • 92 slides
An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning Microsoft Research Asia (MSRA) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. Deep Residual Learning for Image Recognition. arXiv

1.32k views • 54 slides
Learning to Rank with Learning to Rank with Partially-Labeled Data Partially-Labeled Data Kevin

Learning to Rank with Learning to Rank with Partially-Labeled Data Partially-Labeled Data Kevin

Learning to Rank with Learning to Rank with Partially-Labeled Data Partially-Labeled Data Kevin Duh University of Washington 1 The Ranking Problem The Ranking Problem Definition: Given a set of objects, sort them by preference. objectA

716 views • 59 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning with multi-layer (3~30) neural networks, on a huge training set. State-of-the-art on many AI tasks Computer Vision : Image Classification,

1.53k views • 23 slides
Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Tofu: Parallelizing Deep Learning Systems with Automatic Tiling Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL system is based on dataflow GPU#0 w1 w2 data g1 g2 Forward propagation

405 views • 38 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Online Learning to Rank with Features Authors: Shuai Li, Tor Lattimore, Csaba Szepesvri The

Online Learning to Rank with Features Authors: Shuai Li, Tor Lattimore, Csaba Szepesvri The

Online Learning to Rank with Features Authors: Shuai Li, Tor Lattimore, Csaba Szepesvri The Chinese University of Hong Kong DeepMind University of Alberta Learning to Rank Amazon, YouTube, Facebook, Netflix, Taobao 1 Online Learning to

530 views • 17 slides
Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu 2020 For the full list of references visit: http://bit.ly/deeplearn-sota-2020 Outline Deep Learning Growth, Celebrations, and Limitations

1.1k views • 87 slides
12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Wanli Ouyang, Zsolt Kira, Lawrence Neal, Raymond Yeh, Junting Lou and Teck-Yian Lim Unsupervised Learning in General Unsupervised learning is

1.48k views • 115 slides
Cross-Domain Learning-to-rank with SVM Erheng Zhong 1 1 Department of Computer Science and

Cross-Domain Learning-to-rank with SVM Erheng Zhong 1 1 Department of Computer Science and

Preliminary Cross-domain Learning-to-Rank Summary Cross-Domain Learning-to-rank with SVM Erheng Zhong 1 1 Department of Computer Science and Technology, HKUST COMP621U Presentation, 04/07/2011 Erheng Zhong Cross-Domain Learning-to-rank

620 views • 40 slides
Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-27 Definition Regularization is any modification we make to a learning algorithm that is intended to reduce

179 views • 13 slides
Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples RNNs Some slides borrowed from Fei-Fei Li & Justin Johnson &

1.1k views • 55 slides

More recommend