Building the Drupal Borg BADCamp October 23nd 2011 Thursday, - PowerPoint PPT Presentation

Building the Drupal Borg BADCamp October 23nd 2011 Thursday, January 12, 12

In The Beginning Scaling used to be far more complex. Thursday, January 12, 12

The Late 90’s Ease of impleme ntation Java v1 Use-Cases Covered Thursday, January 12, 12

The Late 90’s Ease of impleme ntation J2EE Java v1 Use-Cases Covered Thursday, January 12, 12

The Late 90’s Ease of impleme ntation J2EE Microsoft Stack Java v1 Use-Cases Covered Thursday, January 12, 12

The Late 90’s LAMP Ease of impleme ntation J2EE Microsoft Stack Java v1 Use-Cases Covered Thursday, January 12, 12

A Dark Time • All apps written from scratch • Network databases still immature • “Publishing HTML” still the goal • Remember the web in the 90’s? (sorta sucked) Thursday, January 12, 12

LAMP Shines a Light • Share-nothing runtime • Network-connected database • Free and Open Source • Runs on any hardware Thursday, January 12, 12

And so we grew And so the web grew Thursday, January 12, 12

“Web Scale” • C10k Challenge • Evented I/O wins • Nginx and Lighttpd • Memcache • Distributed reads Thursday, January 12, 12

But the Architecture Thursday, January 12, 12

By 2005... net. LiveJournal Backend: Today Roughly. BIG-IP Global Database perlbal (httpd/proxy) bigip1 mod_perl bigip2 proxy1 master_a master_b proxy2 web1 proxy3 web2 Memcached slave1 slave2 ... slave5 web3 proxy4 mc1 proxy5 web4 mc2 User DB Cluster 1 ... mc3 uc1a uc1b web50 mc4 User DB Cluster 2 ... uc2a uc2b Mogile Storage Nodes mc12 sto2 sto1 User DB Cluster 3 sto8 ... uc3a uc3b Mogile Trackers User DB Cluster 4 tracker1 tracker2 uc4a uc4b MogileFS Database User DB Cluster 5 uc5a uc5b mog_a mog_b http://www.danga.com/words/ Thursday, January 12, 12

Infrastructure Shared Hosting Ease of impleme VPS ntation Dedicated Server HA Web Cluster Custom Cluster Use-Cases Covered Thursday, January 12, 12

Infrastructure Cost is a real Barrier Shared Hosting Ease of impleme VPS ntation Dedicated Server HA Web Cluster Custom Cluster Use-Cases Covered Thursday, January 12, 12

Infrastructure Expertise is a real barrier Shared Hosting Ease of impleme VPS ntation Dedicated Server HA Web Cluster Custom Cluster Use-Cases Covered Thursday, January 12, 12

The Units of Scale Entire hardware servers is the wrong unit to scale. Tons of capacity at a chunk. Thursday, January 12, 12

The Challenge This isn’t just about scaling one website Thursday, January 12, 12

The Challenge This is about scaling the web Thursday, January 12, 12

The Challenge Limited developer talent limits Drupal adoption Thursday, January 12, 12

The Challenge Expertise in infrastructure is even more limited Thursday, January 12, 12

The Challenge Luckily infrastructure is friendly to automation Thursday, January 12, 12

Bring on the Borg! Thursday, January 12, 12

Bring on the Borg! Thursday, January 12, 12

A Brief History of Borg Google / AppEngine CfEngine, Puppet, Chef Heroku Thursday, January 12, 12

This is how we do it Thursday, January 12, 12

This is how we do it Thursday, January 12, 12

Pantheon is the best of both worlds Thursday, January 12, 12

Pantheon will un-block Drupal adoption on infrastructure Thursday, January 12, 12

Inside Pantheon The High Level View Thursday, January 12, 12

Routing Mesh Varnish Firewall “Styx” Request Thursday, January 12, 12

Routing Mesh Routing Mesh Thursday, January 12, 12

Routing Mesh Yggdrasil DNS Routing Mesh Thursday, January 12, 12

Routing Mesh Yggdrasil DNS Routing Mesh Cassandra Thursday, January 12, 12

Routing Mesh Yggdrasil DNS Routing Mesh Cassandra Thursday, January 12, 12

Routing Mesh Yggdrasil DNS Routing Mesh Cassandra Thursday, January 12, 12

Routing Mesh Routing Mesh Yggdrasil Thursday, January 12, 12

Routing Mesh Routing Mesh Yggdrasil DROPs Matrix Thursday, January 12, 12

DROPs Provision, Isolate and Scale at the per-process level. Thursday, January 12, 12

DROPs Provision, Isolate and Scale at the per-process level. Thursday, January 12, 12

DROPs We can scale this too Thursday, January 12, 12

DROPs We can scale this too Thursday, January 12, 12

DROPs We can scale this too Thursday, January 12, 12

Anatomy of a DROP Nginx PHP-FPM APC Cache Service Bindings Thursday, January 12, 12

Valhalla Dev Test Live Twisted DAV Thursday, January 12, 12

Valhalla Dev Test Live Twisted DAV Cassandra Thursday, January 12, 12

Other Service Bindings MySQL MemCache Solr Redis? Cassandra? Thursday, January 12, 12

Drush Access Custom SSH Server drush @pantheon cc all Thursday, January 12, 12

Drush Access Yggdrasil DNS Custom SSH Server drush @pantheon cc all Thursday, January 12, 12

Magic On The Horizon Thursday, January 12, 12

Have questions? Contact Zack Rosen zack@getpantheon.com 1-415-992-7531 Thursday, January 12, 12