TESTING PROGRAMMABLE INFRASTRUCTURE (WITH RUBY) @burythehammer - PowerPoint PPT Presentation

@burythehammer MATT LONG TESTING PROGRAMMABLE INFRASTRUCTURE (WITH RUBY)

@burythehammer PROGRAMMABLE INFRASTRUCTURE IS GREAT, BUT WE'RE MISSING SOMETHING. TESTING.

@burythehammer HELLO, I'M MATT I'M A TESTER @burythehammer

@burythehammer I WORK HERE ↑

@burythehammer I AM NOT A SYSADMIN

@burythehammer WHAT IS PROGRAMMABLE INFRASTRUCTURE?

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE PROGRAMMABLE INFRASTRUCTURE IS.. THE APPLICATION OF METHODS AND TOOLING FROM SOFTWARE DEVELOPMENT TO THE MANAGEMENT OF IT INFRASTRUCTURE THE INTERNET

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE WHAT DO WE MEAN BY THIS? ▸ Automated provisioning & configuration ▸ Configuration as code ▸ Version / source controlled

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE TOOLING EXAMPLES

@burythehammer PROGRAMMABLE INFRASTRUCTURE IS AWESOME! Credit: Vault Boy, Bethesda Softworks

@burythehammer BUT IT CAN GET COMPLEX

@burythehammer TESTING IS USED TO MITIGATE COMPLEXITY & RISK

@burythehammer BUT INFRA TESTING IS RARE Credit: Gunshow, KC Green

@burythehammer AN INFRASTRUCTURE HEAVY PRODUCT TESTING A CLOUD BROKER

@burythehammer THE PROBLEM

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE WE WANT TO MOVE TO THE CLOUD... BUT WE'RE WARY OF LOCK IN Large organisation

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CLOUD BROKER USERS

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE BENEFITS ▸ Quick, easy provisioning ▸ one team previously took 3 months ▸ Don't need to work at a low level ▸ Templates for common dev environments ▸ Built in best practice: monitoring, security ▸ Track spending

@burythehammer THIS IS A REALLY COMPLICATED APPLICATION

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE WORKFLOW ▸ Log into Web UI ▸ Fill in information about environment ▸ Broker creates and bootstraps resources ▸ SSH into resources

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE ▸ Log into Web UI ▸ Fill in information about environment WEB TESTING

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE ??? ▸ Broker creates and bootstraps resources ▸ SSH into resources

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TESTING ▸ Broker creates and bootstraps resources ▸ SSH into resources

@burythehammer HOW DO YOU TEST INFRASTRUCTURE?

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE WHAT TO TEST? Deployment scripts unit tests? Are services running? Linting? Can instances 
 access one another? Does the VPN server work?

@burythehammer THIS SEEMS FAMILIAR..

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE ANOTHER TESTING PYRAMID? expensive, slow Does the VPN box work? 
 Can I SSH into a server? Are services running? Can instances access one another? cheap, fast Do our deployment scripts work? 
 Linting? credit: Ubuntu dev quality guide 
 https://developer.ubuntu.com/en/phone/platform/quality/

@burythehammer TOOLING

@burythehammer UNIT TESTING ▸ Bash scripts ▸ Ansible scripts ▸ Terraform scripts

@burythehammer LINTING ▸ Quick sanity check ▸ Available in all tools ▸ Run in CI before committing

@burythehammer UNIT TESTING IS HARD ▸ Cultural issues ▸ Technical issues ▸ Return on investment

@burythehammer INTEGRATION TESTING ▸ Packages installed ▸ Services running ▸ Ports listening

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC ▸ Ruby / RSpec based ▸ Great community ▸ Very readable ▸ Very quick! ▸ Can SSH into instances

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC EXAMPLE describe package('jenkins') do it { should be_installed } end describe service('jenkins') do it { should be_enabled } it { should be_running } end describe port(8080) do it { should be_listening } end

@burythehammer ACCEPTANCE TESTING ▸ SSHing into machines ▸ Using applications ▸ e.g. samba, openvpn

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CUCUMBER ▸ Executable specifications ▸ Both test & business can read it ▸ Available in Ruby, Java, C#... etc ▸ Great reporting ▸ Good CI integration


 @burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CUCUMBER EXAMPLE Scenario: Searching for Wikipedia Given I am on the website "www.google.com" When I search for "Wikipedia" Then the first link should be "www.wikipedia.org"

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE RUBY ▸ Fantastic testing community ▸ More suitable for SSHing into boxes ▸ "Win RM" gem ▸ Ops already familiar with it ▸ Reduces tech stack

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE TOOLS WE DIDN'T USE ▸ Bats ▸ ShUnit2 ▸ Goss ▸ Inspec / TestInfra ▸ AWS Spec ▸ Test Kitchen

@burythehammer OUR SOLUTION

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CLOUD BROKER USERS

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE WEB TEST FRAMEWORK USERS

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE WEB TESTS USERS https://github.com/opencredo/test-automation-quickstart

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TEST FRAMEWORK USERS

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TESTS Serverspec

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE INFRASTRUCTURE TESTING STACK / Serverspec Linting tools

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE SERVERSPEC SMOKE TESTS ▸ Run before everything else ▸ Really quick ▸ Catches obvious errors ▸ Not complex tasks


 
 
 @burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CUCUMBER FOR ACCEPTANCE TESTING Background: 
 Given environment has been created 
 And the following user details: 
 | user_alias | username | public_key | 
 | userA | testuser | test | Scenario: IPA - Login via SSH Key authentication succeeds 
 Given user "userA" is authorised to access environment vms 
 When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" 



 
 
 
 @burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CUCUMBER FOR ACCEPTANCE TESTING Background: 
 Given environment has been created 
 Cloud broker APIs And the following user details: 
 | user_alias | username | public_key | 
 | userA | testuser | test | Scenario: IPA - Login via SSH Key authentication succeeds 
 Given user "userA" is authorised to access environment vms 
 When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" 



 
 
 
 @burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CUCUMBER FOR ACCEPTANCE TESTING Background: 
 Given environment has been created 
 And the following user details: 
 | user_alias | username | public_key | 
 | userA | testuser | test | Scenario: IPA - Login via SSH Key authentication succeeds 
 Given user "userA" is authorised to access environment vms 
 When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" 
 Standard Ruby


 
 
 
 @burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE CUCUMBER FOR ACCEPTANCE TESTING Background: 
 Given environment has been created 
 And the following user details: 
 | user_alias | username | public_key | 
 | userA | testuser | test | Scenario: IPA - Login via SSH Key authentication succeeds 
 Given user "userA" is authorised to access environment vms 
 When user "userA" starts ssh session in host "env" Then I should be able to echo "hello world" 
 RSpec assertions

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE UNDER THE CUCUMBER, PLAIN RUBY Then(/^I should be able to echo "([^"]*)"$/) do |text| cmd = "echo #{text}" output = @session.exec!(cmd) close_ssh(@session) expect(output.to_s.strip).to eql(text) end

@burythehammer LESSONS LEARNED THE GOOD, THE BAD, AND THE UGLY

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE THE GOOD ▸ Specialised tests for each layer ▸ Quick, expressive ServerSpec tests ▸ Power of a full programming language for user tests ▸ ... it's also totally doable! Credit: Overwatch, Blizzard Entertainment

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE THE BAD ▸ Over reliance on acceptance tests ▸ Context switching with two suites ▸ Out of my comfort zone Credit: Futurama, Fox

@burythehammer TESTING PROGRAMMABLE INFRASTRUCTURE THE UGLY ▸ Starting infrastructure is SLOW. ▸ It's expensive...

@burythehammer DESPITE ALL THAT IT WAS WORTH IT

@burythehammer CONCLUSIONS

@burythehammer TESTING IS IMPORTANT BUT OFTEN IGNORED

@burythehammer TESTERS AND OPS SHOULD WORK TOGETHER WE NEED TO GET OUT OF OUR COMFORT ZONES

@burythehammer TOOLS EXIST BUT BE PREPARED TO HACK