FIXING T THE FLYING P PLAN ANE Major SAAS Upgrades by a - PowerPoint PPT Presentation

FIXING T THE FLYING P PLAN ANE Major SAAS Upgrades by a Production DevOps Team of 26

Introduction Calvin Domenico Jesse Campbell Director Sr. Software Engineer, Lead of Development Marie Hetrick Alastair Firth Manager of Hosting Software Engineer Elijah Aydnwylde Brandon Arsenault Sr. Sysadmin, Lead of Operations Project Manager Patrick McAndrew Sr. Sysadmin, Lead of Infrastructure Introduction 2 of 26

The “Before” Environment • ~20 custom-developed services accessed by 10,000+ school districts nationwide • Software not designed for SaaS • Virtualized environment in Managed Hosting datacenter limited visibility and prevented admin access to infrastructure The “Before” Environment 3 of 26

The “Before” Environment Problem Scenario ■ Customers reporting networking issues ■ Troubleshooting isolates load balancer ■ MSP says it can't be Solution ■ Bypass the load balancer Cost ■ Lost customers ■ Man-weeks of troubleshooting and workarounds (attempts to work with MSP almost doubled this) The “Before” Environment 4 of 26

OP OPERA ERATORS ORS can’t OP OPERA ERATE E if they can’t SE SEE 5 of 26

The Project • SOLVE the Managed Services problem without incurring the business and man-hour costs of colocating • DESIGN a datacenter for the purpose of serving this specific software as SaaS • PLAN up to 5x growth within 2 years, as well as upcoming changes to the software (i.e. clustering) • PROOF the new datacenter in a local virtualized environment so that as much of it as possible can be "ported" directly to the new hardware The Project 6 of 26

The Challenge: DON DON’T T LAN AND THE P PLAN ANE 7 of 26

The Challenge • One w week o of t total d downtime for all operations • Six months maximum limit for datacenter design, code development & implementation • Design , Build , Code , Upgrade , and Migrate all at once! The Challenge 8 of 26

The DEVEL VELOP OPME MENT T 9 of 26

The Development Requirements • What to build? ■ Manage multiple layers • Virtual Infrastructure • Machine • Application • Data • Why should we build it? The Development: Requirements 10 of 26

The Development What Did We Build? • Automated Control engine for existing technologies ■ NFS, Git, Puppet, VSphere, bash, perl • Unified control front-end • Extensible framework • No recovery: destroy and rebuild • Easy to pick up and create a new complete stack The Development: What Did We Build? 11 of 26

The Development The Team • Methodology • Mentality • Motivation • Personality • Ownership? • Who writes the spec? The Development: The Team 12 of 26

The Development Outside Stakeholders The Dev Environment Manager/Liason • Tight schedule ■ Fast iterations • Design, Develop, Deploy, Destroy ■ Feature driven design • Communication Devs ■ Oversight / insight • Single point of contact ■ Open access for devs ■ Appeasing stakeholders Ops Infra • Legitimate concerns The Development: The Dev Environment 13 of 26

THE THEN and NO NOW W 14 of 26

Then and Now Time to Create and Deploy a Site 3 – 5 DAYS 24 Vs. HOURS Then and Now: Time to Create and Deploy a Site 15 of 26

Then and Now $ Number of words required to get Time to Bring a Virtual Machine Online a Virtual Machine online $ then 23523 23523 words 30 – 45 1 $ now 5 words ▋ Vs. HOUR DAYS Then and Now: Time to Bring a Virtual Machine Online 16 of 26

Then and Now Time to Configure an Application Server DAYS <5 3 Vs. HOURS, AUTOMATED Then and Now: Time to Configure an Application Server 17 of 26

Then and Now Time to Configure a Database Server WEEK <5 1 Vs. HOURS, AUTOMATED Then and Now: Time to Configure a Database Server 18 of 26

Then and Now Time to Deploy a Patch (Hours) 3 160 40 4,500 HOURS HOURS HOURS HOURS 12 Months Ago 6 Months Ago Today 18 Months Ago Then and Now: Time to Deploy a Patch (Hours) 19 of 26

Then and Now Time to Re-balance Database Layer 1.5 2 People 4/4 Vs. MONTHS OF OVERTIME DECISION-MAK AKING/4 H HOURS R REVIEW Automated Then and Now: Time to Re-balance Database Layer 20 of 26

Then and Now Time to Recover Our Entire Environment 5+ WEEKS <24 Vs. HOURS Then and Now: Time to Recover Our Entire Environment 21 of 26

how did it all COME T TOGETHE THER? ? 22 of 26

How Did it All Come Together? Abstracting Enterprise Components • Abstracting System and Software Components ■ What are our Software Components? • Application Agents • Customer Databases ■ What are our System Components? • Application Servers • Database Servers How Did it All Come Together?: Abstracting Enterprise Components 23 of 26

How Did it All Come Together? Abstracting Harder • What are the relationships between these components? • How can they be abstracted? ■ Cluster • A selection of Customers grouped together and handled by a single Agent ■ Node • An instance of a cluster running on an Application Server • What do these abstractions allow us to infer by relation? How Did it All Come Together?: Abstracting Harder 24 of 26

How Did it All Come Together? Agile Development • Adaptable to ■ Unknown Performance and Needs ■ Changing Requirements • High Visibility provides ■ Decreased Risk ■ Increased Business Value • Collaborative Design promotes ■ Diverse Viewpoints ■ Shared Experience How Did it All Come Together?: Agile Development 25 of 26

end 26 of 26