big data architectures facebook
play

Big Data Architectures@ Facebook QCon London 2012 Ashish Thusoo - PowerPoint PPT Presentation

Aug 02, 2023 •32 likes •572 views

Big Data Architectures@ Facebook QCon London 2012 Ashish Thusoo Thursday, March 8, 12 Outline Big Data @ Facebook - Scope & Scale Evolution of Big Data Architectures @ FB Past, Present and Future Questions Thursday, March

  1. Big Data Architectures@ Facebook QCon London 2012 Ashish Thusoo Thursday, March 8, 12

  2. Outline • Big Data @ Facebook - Scope & Scale • Evolution of Big Data Architectures @ FB • Past, Present and Future • Questions Thursday, March 8, 12

  3. Big Data @ FB: Scale • 25 PB of compressed data • equivalent to 300 years of HD-TV video Thursday, March 8, 12

  4. Big Data @ FB: Scale • 150 PB of uncompressed data • equivalent to 3 x the entire written works of mankind from the beginning of recorded history in all languages Thursday, March 8, 12

  5. Big Data @ FB: Scale • 400 TB/day (uncompressed) of new data • That is a lot of disks Thursday, March 8, 12

  6. Big Data @ FB: Scope • Simple reporting • Model generation • Adhoc analysis + data science • Index generation • Many many others... Thursday, March 8, 12

  7. A/B Testing Email #1 Thursday, March 8, 12

  8. A/B Testing Email #2 Thursday, March 8, 12

  9. A/B Testing Email #2 is 3x Better Thursday, March 8, 12

  10. Friend Map By Paul Butler - https://www.facebook.com/notes/facebook-engineering/visualizing-friendships/ 469716398919 Thursday, March 8, 12

  11. Big Data @ FB: Scope • one new job every second • ~ 15% of the company uses the clusters Thursday, March 8, 12

  12. Evolution: 2007-2011 DW Size in TB 30000 25000 22500 15000 8000 7500 800 250 15 0 2007 2008 2009 2010 2011 Thursday, March 8, 12

  13. 2007: Traditional EDW Thursday, March 8, 12

  14. 2007: Traditional EDW Web Clusters MySQL Clusters Thursday, March 8, 12

  15. 2007: Traditional EDW Web Clusters RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  16. 2007: Traditional EDW Scribe Mid-Tier Web Clusters RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  17. 2007: Traditional EDW Scribe Mid-Tier Web Clusters NAS Filers RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  18. 2007: Traditional EDW Scribe Mid-Tier Summarization Cluster Web Clusters NAS Filers RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  19. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters RDBMS Data Warehouse Thursday, March 8, 12

  20. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours RDBMS Data Warehouse Thursday, March 8, 12

  21. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours - Lots of tuning/indexes etc. RDBMS Data Warehouse Thursday, March 8, 12

  22. 2007: Pain Points Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours - Lots of tuning/indexes etc. - Lots of hardware planning RDBMS Data Warehouse Thursday, March 8, 12

  23. 2007: Pain Points - compute close to storage (early map/reduce) Scribe Mid-Tier Web Clusters Summarization Cluster NAS Filers MySQL Clusters - daily ETL > 24 hours - Lots of tuning/indexes etc. - Lots of hardware planning RDBMS Data Warehouse Thursday, March 8, 12

  24. 2007: Limitations • Most use cases were in business metrics - data science, model building etc. not possible • Only summary data was stored online - details archived away Thursday, March 8, 12

  25. 2008: Move to Hadoop Scribe Mid-Tier Summarization Cluster Web Clusters NAS Filers RDBMS Data Warehouse MySQL Clusters Thursday, March 8, 12

  26. 2008: Move to Hadoop Batch Scribe Mid-Tier copier/ Web Clusters loaders Hadoop/Hive Data Warehouse NAS Filers MySQL Clusters RDBMS Data Mart Thursday, March 8, 12

  27. 2008: Immediate Pros • Data science at scale became possible • For the first time all of the instrumented data could be held online • Use cases expanded Thursday, March 8, 12

  28. 2009: Democratizing Data Scribe Mid-Tier Web Clusters Hadoop/Hive Data Warehouse NAS Filers MySQL Clusters RDBMS Data Mart Thursday, March 8, 12

  29. 2009: Democratizing Data Databee & Nectar: Chronos: Data instrumentation & Pipeline schema aware Framework data collection HiPal: Adhoc Scrapes: Hadoop/Hive Data Queries + Data Configuration Warehouse Discovery Driven Thursday, March 8, 12

  30. 2009: Democratizing Data(Nectar) • Typical Nectar Pipeline • Simple schema evolution built in • json encoded short term data • decomposing json for long term storage Thursday, March 8, 12

  31. 2009: Democratizing Data (Tools) • HiPal - data discovery and query authoring • Charting and dashboard generation tools Thursday, March 8, 12

  32. 2009: Democratizing Data (Tools) • Databee: Workflow language • Chronos: Scheduling tool Thursday, March 8, 12

  33. 2009: Cons of Democratization • Isolation to protect against Bad Jobs • Fair sharing of the cluster - what is a high priority job and how to enforce it Thursday, March 8, 12

  34. 2010: Controlling Chaos • Isolation • Reducing operational overhead • Better resource utilization • Measurement, ownership, accountability Thursday, March 8, 12

  35. 2010: Isolation Scribe Mid-Tier Web Clusters Hadoop/Hive Data Warehouse NAS Filers MySQL Clusters Thursday, March 8, 12

  36. 2010: Isolation Scribe Mid-Tier Web Clusters Platinum Warehouse Hive Replication NAS Filers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  37. 2010: Ops Efficiency Scribe HDFS Web Clusters Platinum Warehouse Hive Replication MySQL Clusters Silver Warehouse Thursday, March 8, 12

  38. 2010: Ops Efficiency Scribe HDFS Web Clusters Platinum Warehouse Hive Replication near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  39. 2010: Ops Efficiency Scribe HDFS Web Clusters ptail: Platinum Warehouse parallel tail Hive Replication on hdfs near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  40. 2010: Resource Utilization (Disk) • HDFS-RAID: from 3 replicas to 2.2 replicas • RCFile: Row columnar format for compressing Hive tables Thursday, March 8, 12

  41. 2010: Resource Utilization (CPU) • Continuous copier/ loaders • Incremental scrapes • Hive optimizations to save CPU Thursday, March 8, 12

  42. 2010: Monitoring(SLAs) • Per job statistics rolled up to owner/group/team • Expected time of arrival vs Actual time of arrival of data • Simple data quality metrics Thursday, March 8, 12

  43. 2011: New Requirements • More real time requirements for aggregations • Optimizing resource utilization Thursday, March 8, 12

  44. 2011: Beyond Hadoop • Puma for real time analytics • Peregrine for simple and fast queries Thursday, March 8, 12

  45. 2010: Puma Scribe HDFS Web Clusters ptail: Platinum Warehouse parallel tail Hive Replication on hdfs near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  46. 2010: Puma Scribe HDFS Web Clusters ptail: Platinum Warehouse parallel tail Hive Replication on hdfs near real time data consumers MySQL Clusters Silver Warehouse Thursday, March 8, 12

  47. 2010: Puma Thursday, March 8, 12

  48. 2010: Puma Scribe HDFS ptail: parallel tail on hdfs Thursday, March 8, 12

  49. 2010: Puma Scribe HDFS ptail: parallel tail on hdfs Puma Clusters Thursday, March 8, 12

  50. 2010: Puma Scribe HDFS ptail: parallel tail on hdfs Puma Clusters Hbase Cluster Thursday, March 8, 12

  51. Other Challenges Of HyperGrowth • Moving data centers • Moving sustainably fast Thursday, March 8, 12

  52. HyperGrowth - Moving Data Centers DW Size in TB 30000 25000 22500 15000 8000 7500 800 250 15 0 2007 2008 2009 2010 2011 Thursday, March 8, 12

  53. HyperGrowth - Moving Data Centers • Moved 20 PB of data • Leverage replication with fast switch • 2-3 months to accomplish the entire move Blog Post on FB by Paul Yang: http://www.facebook.com/notes/paul-yang/moving-an-elephant-large- scale-hadoop-data-migration-at-facebook/10150246275318920 Thursday, March 8, 12

  54. Questions Contact Information: ashish.thusoo@gmail.com http://www.linkedin.com/pub/ashish-thusoo/0/5a8/50 https://www.facebook.com/athusoo https://twitter.com/ashishthusoo Thursday, March 8, 12

Recommend

Facebook Exchange Facebook Exchange (FBX) (FBX) Facebook Exchange The Facebook Exchange allows

Facebook Exchange Facebook Exchange (FBX) (FBX) Facebook Exchange The Facebook Exchange allows

Facebook Exchange Facebook Exchange (FBX) (FBX) Facebook Exchange The Facebook Exchange allows programmatic buying of Facebook ad inventory through real-time bidding. Agencies, trading desks and direct marketers can buy Facebook ads using

343 views • 7 slides
Facebook and Cambridge Analytica $1M to buy the data 50 million profiles Analyzing

Facebook and Cambridge Analytica $1M to buy the data 50 million profiles Analyzing

Big data - the large volume of data both structured and unstructured that inundates a business on a day-to-day basis. Big data can be analyzed for insights that lead to better decisions and strategic business moves Facebook and Cambridge

680 views • 38 slides
Facebook Strategies Facebook www.facebook.com Facebook TIPS Idea #1: Share the School Calendar.

Facebook Strategies Facebook www.facebook.com Facebook TIPS Idea #1: Share the School Calendar.

Facebook Strategies Facebook www.facebook.com Facebook TIPS Idea #1: Share the School Calendar. Idea #2: Link to Positive Stories. Idea #3: Post Photos of Events. Idea #4: Congratulate Students and Staff Who Achieve. Facebook TIPS Going

427 views • 39 slides
Facebook 101 FACEBOOK 101 What is Facebook? Facebooks Mission is to give people the

Facebook 101 FACEBOOK 101 What is Facebook? Facebooks Mission is to give people the

Facebook 101 FACEBOOK 101 What is Facebook? Facebooks Mission is to give people the power to share and make the world more open and connected. Mission Statement Facts: More than 500 million active users 50% of our

202 views • 17 slides
Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU controller 10C Disks control address main bus 10D Low Level I/O Techniques data interrupts 10E Higher Level I/O Techniques memory

657 views • 16 slides
Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU

Device I/O I/O architectures: busses 10A I/O Architectures 10B I/O Mechanisms CPU controller 10C Disks control address main bus 10D Low-Level I/O Techniques data interrupts 10E Higher Level I/O Techniques memory

493 views • 17 slides
Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure

Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure

Hive* A Petabyte Scale Data Warehouse Using Hadoop Authors Facebook Data Infrastructure Team CS 743, Fall 2014 Conference Data Engineering (ICDE), 2010 IEEE UNIVERSITY OF WATERLOO Presenter Malek NAOUACH, Nets&Dist Sys November

317 views • 13 slides
A D A C C O U N T SET-UP PROCESS Facebook: 3 STEP SET-UP 1. Facebook Ad Account 2.

A D A C C O U N T SET-UP PROCESS Facebook: 3 STEP SET-UP 1. Facebook Ad Account 2.

A D A C C O U N T SET-UP PROCESS Facebook: 3 STEP SET-UP 1. Facebook Ad Account 2. Facebook Business Page 3. Instagram FACEBOOK & INSTAGRAM SET-UP: Without Business Manager To create, manage and optimize your campaigns, SYVO

563 views • 20 slides
FACEBOOK July 12, 2009 JGS of the Conejo Valley and Ventura County Who or What Is Facebook?

FACEBOOK July 12, 2009 JGS of the Conejo Valley and Ventura County Who or What Is Facebook?

FACEBOOK July 12, 2009 JGS of the Conejo Valley and Ventura County Who or What Is Facebook? Facebook was started at Harvard in 2004 for students, by student Mark Zuckerberg Facebook was so popular that many tried to purchase the

867 views • 23 slides
One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production

One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production

One System To Fit Them All: Shared MySQL Hosting At Facebook Andrew Regner Production Engineer | MySQL Infrastructure Data choices @Facebook Everyone has data to persist Also have: ZippyDB, ODS, Scuba, HBase, Scribe, RocksDB,

590 views • 45 slides
MY FACEBOOK STORY How I got started marketing on Facebook How I built an overseas

MY FACEBOOK STORY How I got started marketing on Facebook How I built an overseas

HOW TO BUILD THE ULTIMATE FACEBOOK LEAD MACHINE MCGRAW-HILL INTERNATIONAL BEST-SELLING AUTHOR JAMES E. DICKS JR. CEO, DNA PULSE, LLC MY FACEBOOK STORY How I got started marketing on Facebook How I built an overseas business with

1.06k views • 67 slides
Scuba: Diving into Data at Facebook Presenter: Lavanya Subramanian 1 Need for Data Analysis

Scuba: Diving into Data at Facebook Presenter: Lavanya Subramanian 1 Need for Data Analysis

Scuba: Diving into Data at Facebook Presenter: Lavanya Subramanian 1 Need for Data Analysis Performance monitoring Detect unexpected performance drops/rises Pattern mining Understand user response to new features Ad revenue

605 views • 17 slides
Realtime Data Processing at Facebook Abhay Venkatesh Actionable reports Why e.g. Chorus:

Realtime Data Processing at Facebook Abhay Venkatesh Actionable reports Why e.g. Chorus:

Realtime Data Processing at Facebook Abhay Venkatesh Actionable reports Why e.g. Chorus: what is trending right now? Realtime monitoring Streaming at e.g. dashboard queries Facebook? Hybrid realtime-batch pipelines e.g.

301 views • 18 slides
Architectures Architectural styles Software architectures Architectures versus middleware

Architectures Architectural styles Software architectures Architectures versus middleware

Architectures Architectures Architectural styles Software architectures Architectures versus middleware Self-management in distributed systems 1 / 26 Architectures Architectural styles Architectural styles Basic idea Organize into

571 views • 33 slides
WHY FACEBOOK? AND HOW TO USE IT TO TAKE OVER THE WORLD 27 July 2015 WHO IS JAMIE WONG? IN THE

WHY FACEBOOK? AND HOW TO USE IT TO TAKE OVER THE WORLD 27 July 2015 WHO IS JAMIE WONG? IN THE

WHY FACEBOOK? AND HOW TO USE IT TO TAKE OVER THE WORLD 27 July 2015 WHO IS JAMIE WONG? IN THE NEXT 15 MINUTES Wi y Facebook? Wi y Facebook isnt really free Types of Paid Advertising Wi at makes good content Useful sites WHY FACEBOOK ?

459 views • 14 slides
Facebook Basics Hannah Digital Literacy Specialist Skokie Public Library What is Facebook?

Facebook Basics Hannah Digital Literacy Specialist Skokie Public Library What is Facebook?

Facebook Basics Hannah Digital Literacy Specialist Skokie Public Library What is Facebook? Facebook is a popular free social networking website launched in 2004 by founder Mark Zuckerberg. Facebook is reported to have 2 billion

653 views • 15 slides
Scuba: Diving into Data at Facebook - Lior et. al Presented By - Sidharth Singla MMATH CS

Scuba: Diving into Data at Facebook - Lior et. al Presented By - Sidharth Singla MMATH CS

Scuba: Diving into Data at Facebook - Lior et. al Presented By - Sidharth Singla MMATH CS OUTLINE Importance History Scuba: Introduction Use Cases Scuba Overview - Data Model; Data Layout; Data ingestion,

450 views • 23 slides
The Art and Science of Data Wrangling Kristen M. Altenburger and Sam Pepose Facebook Core Data

The Art and Science of Data Wrangling Kristen M. Altenburger and Sam Pepose Facebook Core Data

The Art and Science of Data Wrangling Kristen M. Altenburger and Sam Pepose Facebook Core Data Science & Portal AI Georgia Tech CS 4803/7643 Deep Learning February 11, 2020 The performance of machine learning methods is heavily

1.15k views • 67 slides
CS 591: Data Systems Architectures Prof. Manos Athanassoulis mathan@bu.edu

CS 591: Data Systems Architectures Prof. Manos Athanassoulis mathan@bu.edu

CS 591: Data Systems Architectures Prof. Manos Athanassoulis mathan@bu.edu http://manos.athanassoulis.net/classes/CS591 Today big data I want you to speak up! [and you can always interrupt me] data-driven world data systems which are the

860 views • 54 slides
Running a Successful Facebook Ad Campaigns 7th of April 2020 What will be covered today?

Running a Successful Facebook Ad Campaigns 7th of April 2020 What will be covered today?

Running a Successful Facebook Ad Campaigns 7th of April 2020 What will be covered today? Webinar Overview Why advertise on Facebook? Guide to Facebook Ads Where people see Facebook Ads Facebook Ads Manager Creating

696 views • 54 slides
Pfff: PHP Program Analysis at Facebook Yoann Padioleau (Facebook)

Pfff: PHP Program Analysis at Facebook Yoann Padioleau (Facebook)

Pfff: PHP Program Analysis at Facebook Yoann Padioleau (Facebook) http://github.com/facebook/pfff OCaml Users and Developers Workshop 2013 About this talk n Feel free to ask questions during the talk PHP Program Analysis at Facebook n

754 views • 17 slides
Visit Indys 100% UGC Strategy 2014 Media Mix Current Media Mix Facebook CTR Facebook CTR:

Visit Indys 100% UGC Strategy 2014 Media Mix Current Media Mix Facebook CTR Facebook CTR:

Visit Indys 100% UGC Strategy 2014 Media Mix Current Media Mix Facebook CTR Facebook CTR: with UGC Video: 1.15% 2.2% Data: 1/1/18 - 9/4/18 CTR: 4.21% Web User Sessions Heads in Beds Goal: 2,900,000 Goal: 1,214,000 YTD: 2,844,823

356 views • 19 slides
Monitoring Modern Architectures with Data Science QCon 2017 Dave Casper, CTO Abstract Much has

Monitoring Modern Architectures with Data Science QCon 2017 Dave Casper, CTO Abstract Much has

Monitoring Modern Architectures with Data Science QCon 2017 Dave Casper, CTO Abstract Much has changed since simple distributed client/server architectures and so-too have the technologies and industry practices around monitoring.

892 views • 62 slides
Product Management from Facebook to GitHub: Using data to understand users & build products

Product Management from Facebook to GitHub: Using data to understand users & build products

Product Management from Facebook to GitHub: Using data to understand users & build products Identify Define Creating Opportunities Success Experiences Who am I? Cristina Fernndez Counsyl (genetics) GitHub Facebook Identify Define

745 views • 42 slides

More recommend