fixing the image problem of the web using machine learning
play

Fixing the image problem of the web using machine learning Chris - PowerPoint PPT Presentation

Sep 01, 2023 •338 likes •1.06k views

Fixing the image problem of the web using machine learning Chris Heilmann @codepo8, GotoCon Copenhagen, October 2016 Of innovation and impatience CHRIS HEILMANN @CODEPO8 Chris Heilmann @codepo8, Future Decoded, London, Nov 2015 We all know

  1. Fixing the image problem of the web using machine learning Chris Heilmann @codepo8, GotoCon Copenhagen, October 2016

  2. Of innovation and impatience CHRIS HEILMANN @CODEPO8 Chris Heilmann @codepo8, Future Decoded, London, Nov 2015

  3. We all know this character, right? https://en.wikipedia.org/wiki/Mario#Concept_and_creation

  4. But do you know why it looks like it does? https://en.wikipedia.org/wiki/Mario#Concept_and_creation

  5. Red and Blue offered the best contrast to the skin, boots and the game background. https://en.wikipedia.org/wiki/Mario#Concept_and_creation

  6. The cap meant there was no need to worry about hair style, eyebrows and forehead. (There were also not enough pixels for waving hair when falling down a hole)

  7. The large nose and moustache made it possible to avoid a mouth and facial expressions.

  8.  Design by limitations.

  9. ⚙ Everything has its reasons and meaning.

  10. Evolution is happening around us… 🖦 → 💼 → 📲 …and user numbers are shifting.

  11. 📲 T echnology advanced

  12. 📹 Pixels are a side-product of our interactions on 📺 the web.

  13. So much, that they become a nuisance… Type Size (kB) Images 1426 Scripts 357 Video 174 Fonts 123 Stylesheets 76 HTML 67 Other 4 Total 2232 kB http://www.httparchive.org/interesting.php#bytesperpage

  14. 🖍 Inspirational obesity 🐙

  15. • Wrong file formats • Delivering scaled hi-res images to everybody 1.4 MB images… • No automatic conversion and optimisation steps • Hero image instead of text content

  16. 🚁 We need to change that to make the web fast again…

  17. The web is much bigger than our little developer world and growth happens outside of it.

  18. Surgical solutions: Proxy Browsers and Cloud Services https://www.flickr.com/photos/89306448@N00/334479803 {Guerrilla Futures | Jason Tester}

  19.  Here are a few things you can do.

  20. • Huge images for everybody • Unoptimised images The problems: • No alternative content • No training or incentive to add content in CMS

  21. • Better browsers with responsive image support • Automated, loss-less image optimisation tools • File level access to images to extract metadata Our arsenal: • Scripting solutions to offer alternative content • Cloud services with machine learning APIs for intelligent resizing • Machine learning for tagging

  22. Browsers with responsive image support…

  23. https://www.christianheilmann.com/ Media queries are greedy! 2012/12/19/conditional-loading-of- resources-with-mediaqueries/

  24. 🖽 <picture> element and srcset

  25. Support is great!

  26. More info… https://jakearchibald.com/2015/ anatomy-of-responsive-images/

  27. Live demo… https://dev.windows.com/en-us/microsoft- edge/testdrive/demos/picture/

  28. Automated tools for lossless image optimisation

  29. ImageOptim https://imageoptim.com/ PNGOUT, Zopfli, Pngcrush, AdvPNG, extended OptiPNG, JpegOptim, MozJPEG ( jpegtran & jpegrescan), Gifsicle.

  30. File-level access to embedded information in images

  31. EXIF - hidden gems http://code.flickr.net/2012/06/01/parsing- exif-client-side-using-javascript-2/

  32. EXIF - hidden gems http://code.flickr.net/2012/06/01/parsing- exif-client-side-using-javascript-2/

  33. http://code.flickr.net/2012/06/01/parsing- EXIF - THE HIDDEN GEMS exif-client-side-using-javascript-2/

  34. Removing EXIF data… http://removephotodata.com/

  35. Remove locations… http://removephotodata.com/image-to-map.html

  36. How about providing fallback content? http://www.datapointed.net/2010/01/crayola-crayon-color-chart/

  37. Blur-up technique https://css-tricks.com/the-blur-up-technique- for-loading-background-images/

  38. Counting pixels… https://codepo8.github.io/canvas-images-and-pixels/ http://colorify.rocks/index.html

  39. Counting pixels… https://codepo8.github.io/canvas-images-and-pixels/

  40. Colorify.js http://colorify.rocks

  41. Color thief… http://lokeshdhakar.com/projects/color-thief/ https://github.com/jwagner/smartcrop.js/

  42. SCRIPTING SOLUTIONS FOR Intelligent image resizing ALTERNATIVE CONTENT

  43. Smartcrop.js https://github.com/jwagner/smartcrop.js/

  44. Intelligent resizing http://cloudinary.com/blog/ automatically_art_directed_responsive_images

  45. Intelligent resizing http://cloudinary.com/blog/ automatically_art_directed_responsive_images

  46. High contrast tricks https://blog.imgix.com/2015/10/21/automatic- point-of-interest-cropping-with-imgix.html

  47. 🔯 What about information that isn’t in the image?

  48. Machine learning and artificial intelligence to the rescue

  49. Automated alternative text (Facebook) Image may contain: dog, outdoor and nature

  50. How? Image may contain: dog, outdoor and nature https://code.facebook.com/posts/561187904071636

  51. Making photos findable… https://photos.google.com

  52. Learning from good data… http://image-net.org/

  53. Learning from *lots* of data… http://image-net.org/

  54. Learning from *lots* of data… https://github.com/ openimages/dataset

  55. Learning from *lots* of data… https://github.com/ openimages/dataset

  56. Refining the data with language https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html

  57. Compare and enhance…

  58. Detect syntax and mix and match

  59. Add visual information

  60. Adding automated content in context… https://research.googleblog.com/2016/05/aw-so-cute-allo-helps-you-respond-to.html

  61. APIs and demos to use yourself… http://captionbot.ai

  62.  Detecting humans…

  66. Detect Faces… Detection Result: JSON: [ { "faceRectangle": { "width": 109, "height": 109, "left": 62, "top": 62 }, "attributes": { "age": 31, "gender": "male", "headPose": { "roll": "2.9", "yaw": "-1.3", "pitch": "0.0" } "faceLandmarks": { "pupilLeft": { "x": "93.6", "y": "88.2" }, "pupilRight": { "x": "138.4", "y": "91.7" }, ...

  67. Verify Faces Verification Result: JSON: [ { "isIdentical":false, "confidence":0.01 } ]

  68. Cluster automatically…

  69. 🦅 Empowering people…

  70. https://www.youtube.com/watch?v=R2mC-NUAmMk

  71. CHRIS HEILMANN THANKS! @CODEPO8 CHRISTIANHEILMANN.COM

Recommend

Problem Definition Machine learning problems, such as image classification or voice recognition,

Problem Definition Machine learning problems, such as image classification or voice recognition,

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization Zhize Li , Jian Li IIIS, Tsinghua University https://zhizeli.github.io/ Dec 6th, NeurIPS 2018 Problem Definition Machine learning problems, such as image

615 views • 12 slides
Machine Learning on Blue Waters Using TensorFlow with the Image Feature Detection Problem Or:

Machine Learning on Blue Waters Using TensorFlow with the Image Feature Detection Problem Or:

2/8/18 Machine Learning on Blue Waters Using TensorFlow with the Image Feature Detection Problem Or: How I Learned to Stop Worrying And Love AI Presented By: Dr. Aaron D. Saxton Todays Topics Blue Waters overview TensorFlow Basics

533 views • 22 slides
Overview CS 446 What is machine learning? Machine learning : study of computational

Overview CS 446 What is machine learning? Machine learning : study of computational

Overview CS 446 What is machine learning? Machine learning : study of computational mechanisms that learn from data to make predictions and decisions. 1 / 21 Application 1: image classification Birdwatcher takes photos of birds,

1.08k views • 83 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
Image Blob Detection: A Machine Learning Approach Andrew Colello Undergraduate Thesis 2016

Image Blob Detection: A Machine Learning Approach Andrew Colello Undergraduate Thesis 2016

Image Blob Detection: A Machine Learning Approach Andrew Colello Undergraduate Thesis 2016 Union College Department of Computer Science Faculty Advisor: Valerie Barr Background: Golf Ball Problem Photo of landing location Finding bright

429 views • 17 slides
TensorFlow: A System for Learning-Scale Machine Learning Google Brain The Problem Machine

TensorFlow: A System for Learning-Scale Machine Learning Google Brain The Problem Machine

TensorFlow: A System for Learning-Scale Machine Learning Google Brain The Problem Machine learning is everywhere This is in large part due to: 1. Invention of more sophisticated machine learning models 2. Availability of large

330 views • 22 slides
Computer Vision II Bjoern Andres Machine Learning for Computer Vision TU Dresden 2020-05-22

Computer Vision II Bjoern Andres Machine Learning for Computer Vision TU Dresden 2020-05-22

Computer Vision II Bjoern Andres Machine Learning for Computer Vision TU Dresden 2020-05-22 Image decomposition So far, we have studied pixel classification , a problem whose feasible solutions define decisions at the pixels of an image

817 views • 27 slides
Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study of computational mechanisms that learn from data in order to make predictions and decisions. Example 1: image classification A birdwatcher

932 views • 14 slides
COG: COG: Fixing the Intertemporal Intertemporal Pricing Problem Pricing Problem Fixing the

COG: COG: Fixing the Intertemporal Intertemporal Pricing Problem Pricing Problem Fixing the

COG: COG: Fixing the Intertemporal Intertemporal Pricing Problem Pricing Problem Fixing the &amp; Other Comments &amp; Other Comments CAISO MSC Meeting, 8 Feb. 2008 CAISO MSC Meeting, 8 Feb. 2008 Benjamin F. Hobbs bhobbs@jhu.edu Dept. of

290 views • 12 slides
Fixing WTFs - Detecting Image Matches caused by Watermarks, Timestamps, and Frames in Internet

Fixing WTFs - Detecting Image Matches caused by Watermarks, Timestamps, and Frames in Internet

Fixing WTFs - Detecting Image Matches caused by Watermarks, Timestamps, and Frames in Internet Photos Tobias Weyand, Chih-Yun Tsai, Bastian Leibe Fixing WTFs Poster No. 26 Overview Many Computer Vision Applications use Internet photos,

818 views • 38 slides
Decision Trees: Representation Machine Learning 1 Some slides from Tom Mitchell, Dan Roth and

Decision Trees: Representation Machine Learning 1 Some slides from Tom Mitchell, Dan Roth and

Decision Trees: Representation Machine Learning 1 Some slides from Tom Mitchell, Dan Roth and others Key issues in machine learning Modeling How to formulate your problem as a machine learning problem? How to represent data? Which

872 views • 38 slides
Machine Learning Basics Marcello Pelillo University of Venice, Italy Image and Video Understanding

Machine Learning Basics Marcello Pelillo University of Venice, Italy Image and Video Understanding

Machine Learning Basics Marcello Pelillo University of Venice, Italy Image and Video Understanding a.y. 2018/19 What Is Machine Learning? A branch of Artificial Intelligence (AI) . Develops algorithms that can improve their performance using

956 views • 80 slides
Machine Learning Problem Framework Presenters: Bikramjeet Singh(20752928) Priyansh Narang

Machine Learning Problem Framework Presenters: Bikramjeet Singh(20752928) Priyansh Narang

Machine Learning Problem Framework Presenters: Bikramjeet Singh(20752928) Priyansh Narang (20716980) Agenda Background research Brief introduction to Machine Learning ML Problem: Formulation ML Pipeline Questions

924 views • 39 slides
1 Dont Make Me Get Non -Linear! A Grounding Example: Linear Regression Predict

1 Dont Make Me Get Non -Linear! A Grounding Example: Linear Regression Predict

What is Machine Learning? A (Very Short) List of Applications Many different forms of Machine Learning Machine learning widely used in many contexts We focus on the problem of prediction Stock price prediction o Using economic

262 views • 3 slides
? Option 1: Understand the problem, design a solution Option 2: Set it up as a machine learning

? Option 1: Understand the problem, design a solution Option 2: Set it up as a machine learning

? Option 1: Understand the problem, design a solution Option 2: Set it up as a machine learning problem data supervised learning Deep Reinforcement Learning, Decision Making, and Control CS 285 Instructor: Sergey Levine UC Berkeley data

623 views • 50 slides
Image Classification with Fashion-MNIST and CIFAR-10 Khoi Hoang California State University,

Image Classification with Fashion-MNIST and CIFAR-10 Khoi Hoang California State University,

Image Classification with Fashion-MNIST and CIFAR-10 Khoi Hoang California State University, Sacramento Problem Statement Image Classification is one of the most fundamental problem in the field of machine learning. Has many area of

639 views • 10 slides
Image Restoration from a Machine Learning Perspective September 2012 NIST Dianne P. OLeary

Image Restoration from a Machine Learning Perspective September 2012 NIST Dianne P. OLeary

Image Restoration from a Machine Learning Perspective September 2012 NIST Dianne P. OLeary 2012 c 1 Image Restoration Using Machine Learning Dianne P. OLeary Applied and Computational Mathematics Division, NIST Computer Science

651 views • 61 slides
E9 205 Machine Learning for Signal Processing 23-8-17 Outline Basics for Image Processing

E9 205 Machine Learning for Signal Processing 23-8-17 Outline Basics for Image Processing

E9 205 Machine Learning for Signal Processing 23-8-17 Outline Basics for Image Processing Filtering Smoothing Edge Detection Scale Invariant Feature Transform (SIFT) Reference: UCF, Computer Vision Course Link:

599 views • 42 slides
An inverse problem perspective on machine learning Lorenzo Rosasco University of Genova

An inverse problem perspective on machine learning Lorenzo Rosasco University of Genova

An inverse problem perspective on machine learning Lorenzo Rosasco University of Genova Massachusetts Institute of Technology Istituto Italiano di Tecnologia lcsl.mit.edu Feb 9th, 2018 Inverse Problems and Machine Learning Workshop, CM+X

765 views • 48 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Introduction to Machine Learning Hyperparameter Tuning - Problem Definition

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition

Introduction to Machine Learning Hyperparameter Tuning - Problem Definition compstat-lmu.github.io/lecture_i2ml TUNING Recall: Hyperparameters are parameters that are inputs to the training problem, in which a learner I minimizes the

504 views • 7 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

556 views • 17 slides
Machine learning meets super-resolution H. N. Mhaskar Claremont Graduate University, Claremont.

Machine learning meets super-resolution H. N. Mhaskar Claremont Graduate University, Claremont.

Machine learning meets super-resolution H. N. Mhaskar Claremont Graduate University, Claremont. Inverse Problems and Machine Learning February 10, 2018 Goals The problem of super-resolution is dual of the problem of machine learning, viewed

789 views • 41 slides
Image and attribute based identification of Protea species using machine learning techniques

Image and attribute based identification of Protea species using machine learning techniques

Image and attribute based identification of Protea species using machine learning techniques Peter Thompson Supervisor: Dr Willie Brink Stellenbosch University, Applied Mathematics Division Introduction South Africa is very rich in plant

83 views • 7 slides

More recommend