deploying large payloads at scale
play

Deploying large payloads at scale Ramon van Alteren Wednesday, - PowerPoint PPT Presentation

Aug 07, 2023 •39 likes •489 views

Deploying large payloads at scale Ramon van Alteren Wednesday, November 9, 2011 Hyves 9,7M dutch members (16,7M population) ~7M unique visitors / month (Comscore 09/2011) ~2.3M unique visitors / day 800.000 photo uploads /

  1. Deploying large payloads at scale Ramon van Alteren Wednesday, November 9, 2011

  2. Hyves • 9,7M dutch members (16,7M population) • ~7M unique visitors / month (Comscore 09/2011) • ~2.3M unique visitors / day • 800.000 photo uploads / day • 7M chat messages / day Wednesday, November 9, 2011

  3. Hyves - Operational environment • 3500 node serverpark in 3 datacenters • 6Gbps daily outgoing traffic • System Engineering team: 12 • Development team: 33 Wednesday, November 9, 2011

  4. Weekend project Wednesday, November 9, 2011

  5. Weekend project Result: 4.5 x speed/throughput increase Wednesday, November 9, 2011

  6. Weekend -> Company project Wednesday, November 9, 2011

  7. A Few Minor problems Wednesday, November 9, 2011

  8. A Few Minor problems • compilation took ~40-60 minutes Wednesday, November 9, 2011

  9. A Few Minor problems • compilation took ~40-60 minutes • resulting binary was 750MB Wednesday, November 9, 2011

  10. A Few Minor problems • compilation took ~40-60 minutes • resulting binary was 750MB • Code issues Wednesday, November 9, 2011

  11. A Few Minor problems • compilation took ~40-60 minutes • resulting binary was 750MB • Code issues • gcc 4.5.2 required + added deps Wednesday, November 9, 2011

  12. Part I - Solving the build problem Jenkins to the rescue Wednesday, November 9, 2011

  13. Part I - Solving the build problem Jenkins to the rescue Add distCC to speed up compilation times Wednesday, November 9, 2011

  14. Part I - Solving the build problem Add some serious hardware Wednesday, November 9, 2011

  15. Part I - Solving the build problem Add some serious hardware Compile / build in < 6 mins Wednesday, November 9, 2011

  16. Part II: Deploying Sequential ? Wednesday, November 9, 2011

  17. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds Wednesday, November 9, 2011

  18. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds 450 servers * 8 seconds = 3600 seconds == 1 hour 450 servers * 20 seconds = 9000 seconds == 2.5 hour Wednesday, November 9, 2011

  19. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds 450 servers * 8 seconds = 3600 seconds == 1 hour 450 servers * 20 seconds = 9000 seconds == 2.5 hour Diffs ? Wednesday, November 9, 2011

  20. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds 450 servers * 8 seconds = 3600 seconds == 1 hour 450 servers * 20 seconds = 9000 seconds == 2.5 hour Diffs ? binary diff would be between 10KB - 400MB Even on consecutive runs without Wednesday, November 9, 2011

  21. Part II: Deploying - Bittorrent Wednesday, November 9, 2011

  22. Bittorrent - Previous experiences Naive run using bittorrent to transport 300MB throughout our serverpark • Near-complete network outage due to bandwidth starvation • Several crucial subsystems delayed or unreachable due to network bandwidth shortage Wednesday, November 9, 2011

  23. Bittorrent - Previous experiences Wednesday, November 9, 2011

  24. Bittorrent - The Problem Every server has 1Gb/s link to every other server Wednesday, November 9, 2011

  25. Bittorrent - The Problem Every server has 1Gb/s link to every other server they don’t Wednesday, November 9, 2011

  26. Bittorrent - Actual bandwidth available Core 1-4Gb/s Network Wednesday, November 9, 2011

  27. Bittorrent - Actual bandwidth available Production traffic Core Network Administration traffic Wednesday, November 9, 2011

  28. Murder - Why not ? Murder uses two tricks: • Clients (including the seeder) capped to 1 upload peer • Every client receives every peer from the tracker • No download bandwidth cap (easy to add though) Wednesday, November 9, 2011

  29. Murder - Why not ? Murder uses two tricks: • Clients (including the seeder) capped to 1 upload peer • Every client receives every peer from the tracker • No download bandwidth cap (easy to add though) Peers will still connect all over the place Wednesday, November 9, 2011

  30. Murder - Why not ? Murder uses two tricks: • Clients (including the seeder) capped to 1 upload peer • Every client receives every peer from the tracker • No download bandwidth cap (easy to add though) Peers will still connect all over the place It’s slow, timing run over 25 peers took 7 mins Wednesday, November 9, 2011

  31. DIY: Location aware tracker Wednesday, November 9, 2011

  32. SMDB - Location metadata We have bandwidth information available in our server management database Build two-tier bittorrent swarms: • 1 swarm with 2 peers / rack(uplink) • 1 additional swarm per rack (uplink) • cap every client @ 96mbit/s Wednesday, November 9, 2011

  33. DIY: 2-tier swarms Wednesday, November 9, 2011

  34. DIY - Tracker Tracker in python + Flask: • 1100 lines of code (1900 with tests) • Stores transfer metadata in redis • Connects to our SMDB using REST • Exposes REST interface Wednesday, November 9, 2011

  35. DIY - Client We use a slightly modified rtorrent client Same things twitter modified: • Remove features related to operating on the big bad internet. • Make various timeouts more aggresive • No DHT, UPNP etc. Nice bonus: RPC remote API Wednesday, November 9, 2011

  36. Results - single deploy ~300 hosts 2000 1500 1000 500 0 Without build With build Classic deploy Bittorrent deploy Wednesday, November 9, 2011

  37. Graphs - full webfarm cluster Wednesday, November 9, 2011

  38. Graphs - single node Wednesday, November 9, 2011

  39. Graphs - single rack (24 nodes) Wednesday, November 9, 2011

  40. Bitorrent - Statistics == release: 101482 expected: 287 actual: 107 seeders: 0 progress: 0.00% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 267 seeders: 0 progress: 0.00% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 286 seeders: 0 progress: 0.00% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 1 progress: 0.35% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 2 progress: 42.11% start: 12:04:20 last_completed: 12:06:01 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 5 progress: 44.48% start: 12:04:20 last_completed: 12:06:05 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 34 progress: 45.80% start: 12:04:20 last_completed: 12:06:09 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 95 progress: 48.17% start: 12:04:20 last_completed: 12:06:10 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 95 progress: 48.17% start: 12:04:20 last_completed: 12:06:10 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 97 progress: 48.46% start: 12:04:20 last_completed: 12:06:15 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 101 progress: 49.23% start: 12:04:20 last_completed: 12:06:21 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 108 progress: 50.72% start: 12:04:20 last_completed: 12:06:24 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 118 progress: 52.95% start: 12:04:20 last_completed: 12:06:27 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 131 progress: 55.88% start: 12:04:20 last_completed: 12:06:30 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 172 progress: 67.77% start: 12:04:20 last_completed: 12:06:34 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 213 progress: 80.53% start: 12:04:20 last_completed: 12:06:37 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 239 progress: 87.36% start: 12:04:20 last_completed: 12:06:40 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 246 progress: 91.24% start: 12:04:20 last_completed: 12:06:43 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 265 progress: 97.55% start: 12:04:20 last_completed: 12:06:46 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 274 progress: 98.79% start: 12:04:20 last_completed: 12:06:49 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 280 progress: 99.07% start: 12:04:20 last_completed: 12:06:52 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 283 progress: 99.26% start: 12:04:20 last_completed: 12:06:55 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 284 progress: 99.36% start: 12:04:20 last_completed: 12:06:56 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 284 progress: 99.36% start: 12:04:20 last_completed: 12:06:56 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 285 progress: 99.52% start: 12:04:20 last_completed: 12:07:04 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 286 progress: 99.85% start: 12:04:20 last_completed: 12:07:06 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 287 progress: 100.00% start: 12:04:20 last_completed: 12:07:37 failed: 0 == Wednesday, November 9, 2011

Recommend

Deploying Large Scale AVB/TSN Networks Jeff Koftinoff, Meyer Sound Laboratories, Inc. June 19,

Deploying Large Scale AVB/TSN Networks Jeff Koftinoff, Meyer Sound Laboratories, Inc. June 19,

Deploying Large Scale AVB/TSN Networks Jeff Koftinoff, Meyer Sound Laboratories, Inc. June 19, 2015 Deploying Large Scale AVB Networks ACT 1 Deploying Large Scale AVB Networks What does putting audio/video on a

872 views • 63 slides
Data Science Until now Abstractions for writing and deploying large-scale web applications

Data Science Until now Abstractions for writing and deploying large-scale web applications

Data Science Until now Abstractions for writing and deploying large-scale web applications Managing infrastructure (PaaS, IaaS, Infrastructure-as- Code, FaaS, etc.) Constructing applications (ML APIs, Backend-as-a- Service) Portland

1.02k views • 58 slides
Challenges in Delivering and Deploying Software at Scale in Large Clusters Douglas Thain and Kyle

Challenges in Delivering and Deploying Software at Scale in Large Clusters Douglas Thain and Kyle

Challenges in Delivering and Deploying Software at Scale in Large Clusters Douglas Thain and Kyle Sweeney University of Notre Dame {dthain|ksweene3}@nd.edu Software Deployment on HPC Classic Approach Single process MPI app created by end

1.06k views • 50 slides
Deploying large-scale virtual infrastructures with Kadeploy3 Luc Sarzyniec, S ebastien Badia,

Deploying large-scale virtual infrastructures with Kadeploy3 Luc Sarzyniec, S ebastien Badia,

Deploying large-scale virtual infrastructures with Kadeploy3 Luc Sarzyniec, S ebastien Badia, Emmanuel Jeanvoine, Lucas Nussbaum Grid5000 Plan 1 Introduction Kadeploy3 Kabootstrap Our experiment 2 Scalability challenges 3 Experiment 4

205 views • 18 slides
Deploying*Large*Scale*AVB/TSN* Networks*:* Handout * Jeff+Koftinoff+&lt;jeffk@meyersound.com&gt;+

Deploying*Large*Scale*AVB/TSN* Networks*:* Handout * Jeff+Koftinoff+&lt;jeffk@meyersound.com&gt;+

1 ! ! Deploying*Large*Scale*AVB/TSN* Networks*:* Handout * Jeff+Koftinoff+&lt;jeffk@meyersound.com&gt;+ ( Infocomm(2015((Orlando,(Florida( June(19(2015( ! !

216 views • 10 slides
Deploying Large Scale Webapps Thierry Sans Users respond to speed Amazon found every 100ms of

Deploying Large Scale Webapps Thierry Sans Users respond to speed Amazon found every 100ms of

Deploying Large Scale Webapps Thierry Sans Users respond to speed Amazon found every 100ms of latency cost them 1% in sales Google found an extra . 5 seconds in search page generation time dropped traffic by 20% A broker could

559 views • 20 slides
An open, shared platform for developing, deploying, and accessing planetary-scale applications

An open, shared platform for developing, deploying, and accessing planetary-scale applications

An open, shared platform for developing, deploying, and accessing planetary-scale applications Timothy Roscoe Intel Research at Berkeley Thursday, July 01, 2004 APAN, Cairns, 1 July 2004 PlanetLab is Large collection of machines

533 views • 40 slides
FINANCING LARGE SCALE SOLAR Large Scale Solar Conference - Sydney Gloria Chan Director, Large

FINANCING LARGE SCALE SOLAR Large Scale Solar Conference - Sydney Gloria Chan Director, Large

FINANCING LARGE SCALE SOLAR Large Scale Solar Conference - Sydney Gloria Chan Director, Large Scale Solar Lead April 2017 CONTENTS 1. Introduction to CEFC 2. Investment trends 3. The future of large scale solar 4. Pathway to sustainable

664 views • 21 slides
INCORPORATING LARGE-SCALE CITIZEN INCORPORATING LARGE-SCALE CITIZEN DELIBERATION INTO

INCORPORATING LARGE-SCALE CITIZEN INCORPORATING LARGE-SCALE CITIZEN DELIBERATION INTO

INCORPORATING LARGE-SCALE CITIZEN INCORPORATING LARGE-SCALE CITIZEN DELIBERATION INTO ENVIRONMENTAL CONFLICT RESOLUTION America Speaks presentation to the 2008 ECR Conference Table Introductions Please form groups of 4-5 and give: Your name

848 views • 41 slides
Large-Scale Survey Interviewing Following Large Scale Survey Interviewing Following the 2008

Large-Scale Survey Interviewing Following Large Scale Survey Interviewing Following the 2008

Large-Scale Survey Interviewing Following Large Scale Survey Interviewing Following the 2008 WenChuan Earthquake Zhang Huafeng and Jon Pedersen International Blaise Users Conference (IBUC) Baltimore, USA , 1 Introduction Two household

711 views • 22 slides
Large-scale production of graphene: Introduction Large-scale production of graphene: what is

Large-scale production of graphene: Introduction Large-scale production of graphene: what is

Large-scale production of graphene: Introduction Large-scale production of graphene: what is graphene? Graphene is a truly 2D system made of Carbon atoms arranged in an honeycomb hexagonal lattice Zero-gap semiconductor with a low-energy linear

268 views • 6 slides
Ethics in Techniques for large-scale data Graham J.L. Kemp TECHNIQUES FOR LARGE-SCALE DATA

Ethics in Techniques for large-scale data Graham J.L. Kemp TECHNIQUES FOR LARGE-SCALE DATA

Ethics in Techniques for large-scale data Graham J.L. Kemp TECHNIQUES FOR LARGE-SCALE DATA Masters level course: Chalmers MPALG and MPSOF programmes (DAT345) GU Applied Data Science programme (DIT871) Learning outcomes include:

305 views • 18 slides
Deploying, At An Unusual Scale Andrew Godwin @andrewgodwin

Deploying, At An Unusual Scale Andrew Godwin @andrewgodwin

Deploying, At An Unusual Scale Andrew Godwin @andrewgodwin http://www.flickr.com/photos/whiskeytango/1431343034/ Hi, I'm Andrew. Serial Python developer Django core committer Co-founder of ep.io Hi, I'm Andrew. Serial Python developer

1.03k views • 43 slides
An Email Contact Protocol Experiment in a Large-Scale Survey of U.S. in a Large Scale Survey of

An Email Contact Protocol Experiment in a Large-Scale Survey of U.S. in a Large Scale Survey of

An Email Contact Protocol Experiment in a Large-Scale Survey of U.S. in a Large Scale Survey of U.S. Government Employees DC-AAPOR Summer Conference Preview/Review Bureau of Labor Statistics Conference Center Washington DC Washington, DC

1.17k views • 25 slides
Challenges in Deploying and Managing Large Terminologies: NCI Thesaurus For Protg Workshop June

Challenges in Deploying and Managing Large Terminologies: NCI Thesaurus For Protg Workshop June

Challenges in Deploying and Managing Large Terminologies: NCI Thesaurus For Protg Workshop June 22, 2009 Amsterdam Gilberto Fragoso, Sherri de Coronado Challenge Topics Background EVS, NCI Thesaurus, Vocabulary Development, Distribution

521 views • 33 slides
Efficient Large-Scale Graph Processing on Hybrid CPU and GPU Systems A. Gharaibeh, E.

Efficient Large-Scale Graph Processing on Hybrid CPU and GPU Systems A. Gharaibeh, E.

Efficient Large-Scale Graph Processing on Hybrid CPU and GPU Systems A. Gharaibeh, E. Santos-Neto, L. Costa, M. Ripeanu. IEEE TPC, 2014 Sami (sa894) - R244: Large-scale data processing and optimization Efficient Large-Scale Graph Processing on

381 views • 25 slides
Formal Privacy: Making an Impact at Large Organizations Deploying Differential Privacy for the

Formal Privacy: Making an Impact at Large Organizations Deploying Differential Privacy for the

Formal Privacy: Making an Impact at Large Organizations Deploying Differential Privacy for the 2020 Census of Population and Housing Simson L. Garfinkel Senior Scientist, Confidentiality and Data Access U.S. Census Bureau July 31, 2019 JSM

458 views • 35 slides
Motivation Large-scale distributed systems becoming more common multiple datacenters, cloud

Motivation Large-scale distributed systems becoming more common multiple datacenters, cloud

Census: Location-Aware Membership Management for Large-Scale Distributed Systems James Cowling Dan R. K. Ports Barbara Liskov Raluca Ada Popa Abhijeet Gaikwad* MIT CSAIL *cole Centrale Paris Motivation Large-scale distributed systems

606 views • 48 slides
Large-Scale Electronic Voting Protocols Mike Carpenter Introduction What is meant by large-scale

Large-Scale Electronic Voting Protocols Mike Carpenter Introduction What is meant by large-scale

Large-Scale Electronic Voting Protocols Mike Carpenter Introduction What is meant by large-scale electronic voting protocol: Primarily Internet-based Users voting from their own devices (such as home PC/laptop) Aimed toward actual

721 views • 21 slides
Internet Topology Generation for Large Scale BGP Simulation Jean-Michel Fourneau - Houssame

Internet Topology Generation for Large Scale BGP Simulation Jean-Michel Fourneau - Houssame

Internet Topology Generation for Large Scale BGP Simulation Jean-Michel Fourneau - Houssame Yahiaoui Outlook BGP: Border Gateway Protocol Large Scale Simulation: motivations Large Scale BGP Simulation Model Realistic Topologies

550 views • 17 slides
E Evolution of NTCIR: l Infrastructure of Large-Scale Infrastructure of Large Scale

E Evolution of NTCIR: l Infrastructure of Large-Scale Infrastructure of Large Scale

E Evolution of NTCIR: l Infrastructure of Large-Scale Infrastructure of Large Scale Information Access Technologies Evaluation and Testing Evaluation and Testing Noriko Kando Noriko Kando National Institute of Informatics, Japan

632 views • 61 slides
Large-Scale Machine Learning at Twitter 2 Large-Scale Machine Learning at Twitter Jimmy Lin and

Large-Scale Machine Learning at Twitter 2 Large-Scale Machine Learning at Twitter Jimmy Lin and

Large-Scale Machine Learning at Twitter 2 Large-Scale Machine Learning at Twitter Jimmy Lin and Alek Kolcz Twitter, Inc. 1 Image source:google.com/images Large-Scale Machine Learning at Twitter Outline Outline Is twitter big data?

582 views • 40 slides
Large Scale I nternational I Pv6 Pilot Large Scale I nternational I Pv6 Pilot Network (6NET)

Large Scale I nternational I Pv6 Pilot Large Scale I nternational I Pv6 Pilot Network (6NET)

Large Scale I nternational I Pv6 Pilot Large Scale I nternational I Pv6 Pilot Network (6NET) Network (6NET) Athanassios Liakopoulos (aliako@grnet.gr) Greek Research &amp; Technology Network (GRNET) I I I Global I Pv6 Summit November 2004

638 views • 37 slides
INFRASTRUCTURE 2110414 Large Scale Computing Systems Natawut Nupairoj, Ph.D. Outline 2

INFRASTRUCTURE 2110414 Large Scale Computing Systems Natawut Nupairoj, Ph.D. Outline 2

2110414 - Large Scale Computing Systems 1 LARGE SCALE INFRASTRUCTURE 2110414 Large Scale Computing Systems Natawut Nupairoj, Ph.D. Outline 2 Overview Hardware Virtualization Storage Technology 2110414 - Large Scale Computing

512 views • 29 slides

More recommend