How to implement continuous delivery and testing Tamas Csako - PowerPoint PPT Presentation

How to implement continuous delivery and testing Tamas Csako November 2019 1

TAMAS CSAKO, PH.D. Senior Software Testing Manager Head of Cloud & DevTestSecOps Practice EPAM Hungary • 10+ years experience in IT • 6+ years experience in managing and leading teams • Wide technology overview • Community builder, job interviewer, mentor, coach, team player, consultant

Agenda 1 I N T R O D U C T I O N A N D O V E R V I E W 2 O U R B E S T P R A C T I C E S 3 C O N T I N U O U S D E L I V E R Y A N D T E S T I N G 4 S U M M A R Y 3 3

EPAM Cloud and DevTestSecOps Practice KEY FACTS SERVICES 20+ 400+ 8,000 Cloud enablement, migration and engineering Active Projects Cloud, DevOps & Global Delivery Test engineers Centers DevOps Transformation, SDLC Automation MOSTLY USED TOOLS AND TECHNOLOGIES Cyber Security AIOps - AI for IT operations Application & Infrastructure management services (AMS & IMS) PARTNERSHIPS Testing 4

Continuous Testing Blueprint Integration Artefact Test Repository QA Production UAT, INT Deployment Security PROD Functional & check APPROVAL Regression Mock / Test Unit Tests PROD Сode Analysis CI / Build Code Review PERF Server Performance Test Builds / Status Notifications Cloud Version Control Fully Automated CI/CD Pipeline • Built-in quality and Security gates • Dashboard Frequent and fast feedback • Feedback Developer 5

Cloud services - AWS Fully automated infra as code. Build everything. No manual change in important accounts. Do regular POCs of the services as there are numerous changes and new services coming out monthly. Watch the news! SEPARATE DEV/QA/UAT/PROD ENVIRONMENTS • At least in VPC level but best keep them in separate AWS account. • This way you can use the same code to build each of them and test. • Have separate account for manual changes, POCs, sandbox environment. COMPONENTS MIGHT BE BUILT BY DIFFERENT TOOLS • Terraform or CloudFormation for fundamental layer • Packer or Ansible for EC2 AMI build IAM SSO INTEGRATION IS ESSENTIAL FOR BIG COMPANIES • Separate role for users, services, build tools, TechOps-admin groups • Maintain Least Privilege Rule could be hard because of frequent changes. Use AWS managed policies whenever possible. 6

Infrastructure as a code - Terraform HANDLE INFRASTRUCTURE CODE LIKE AN APPLICATION CODE • Versioning your infrastructure • Keep coding standards • Use code review for knowledge sharing • Testing, testing and testing. • Linting, unit and smoke tests (Terratest is a Go library for infrastructure testing) MAKE THE MODULES REUSABLE (WHETHER ACROSS MULTIPLE PROJECTS) AS MUCH AS POSSIBLE •Try to define well designed units and make modules for implementing the required functionality •Well defined input and output parameters •Separation between the logic and the parameters PUSH YOUR INFRASTRUCTURE CODE THROUGH EACH OF YOUR ENVIRONMENTS •Development, Testing, Staging and Production PREPARE ROLLBACK AND DISASTER RECOVERY PLAN •Terraform is a great tool, but it has his own limitation •No automatic rollback capability •It could easily destroy any of your cloud resources, if you are not aware how it works behind the scenes 7

Provisioning and configuration management - Ansible • Ansible is like the Swiss Army Knife of DevOps. • Keep re-usability in mind, separate roles even in code base so it's easier to share for even different projects, teams etc. But keep it simple. • Think declaratively. Try to build roles that they can build from zero but even update an existing env using the same code. (= Idempotence.) Add state to tasks whether possible. • Check if Ansible Galaxy might already have what you try to achieve. Contribute back your improvement ideas. • Naming convention is nice to have for variables, roles, role directory structure and playbooks. Always name tasks in roles. Add comments when your code needs explanation. • Ansible tags are handy for filtering within role tasks so it's good to have them. Group tasks when they have the same tags or when clause. • Use dynamic inventory whenever possible e.g. in cloud environment • Variable precedence is always important to keep in mind so always use variables in the right level but try to keep them in role level. Avoid hard-coded values but use variables and templates. • Secrets should be vaulted. Use at least ansible-vault but central vault is even better. • Clean up debug messages before commit to version control but log and monitor performance of your playbooks. • Try to build roles which can run in multiple OS families. • Gather facts could slow down playbook runtime. References: - https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html 8

Orchestration management - Kubernetes CONTAINERIZE YOUR OPERATIONS • "Run to completion" style automations: • Versioned DB schema upgrades • Database population with versioned test data • Regular cleanup/garbage collection to keep your environments healthy THIRD-PARTY COMPONENTS (OR DEVELOPING YOUR OWN): DO NOT REINVENT THE WHEEL! • Install via Helm charts (the agentless Helm 3 has just been released!) - Infrastructure as Code concept Go for shopping curated charts: github.com/helm/charts/tree/master/stable • Install via Operators and let Kubernetes do maintenance work for you – Infrastructure as Software concept When tasks like updating, scaling, reconfiguring, continuous supervising are not trivial or hard to cover their automation with built-in functionalities of Kubernetes. You can find a lot here: OperatorHub.io • Out of the box solutions doesn't suit you? Try extending functionalities with Sidecar containers instead of altering an upstream • Always think about Security: What kind of container images are pulled into your clusters with a chart? You can validate them (Docker Bench, Claire, Anchore, etc.), then host them from own registry (Harbor, JFrog, Nexus) or even sign them (Notary) 9

Containerization - Docker CI/CD • Prepare for externalized configuration: containerized application • • config override mechanism SECURE YOUR IMAGES • Build upon trusted and minimal base images, strictly avoid "pushed" images • Minimalize your image content to bare-minimal (reduce attack surface) • Limit user permissions inside containers: • run your processes as non-root users • advanced: use uid/gid mapping (docker user namespaces) OPERATIONS/DEBUGGING • Provide healthcheck for your containerized services to be able to detect readiness and liveness • Containerized software should print logs to STDIN/STDERR • Docker logs are ephemeral by default, collect them centrally (ELK/EFK, AWS Cloudwatch, etc) • Attach specially prepared debug container and get a shell just like you have entered to target container with exec but with an extended toolkit CONTAINERIZED TESTING • You can create reproducible and quick test environments even for UI testing (Selenium) 10

Monitoring - Grafana and Prometheus HOW TO DO MONITORING IN A FAST-CHANGING ENVIRONMENT REQUIREMENTS: • Automation Scalability • On-demand • DISCOVERY: Design a flexible tagging schema • • Service type, Environment, Custom settings VISUALIZATION: Compose a dashboard as needed • • Use variables to make the dashboard more dynamic 11

DevTestSecOps Transformation Requires a Holistic Approach • Test Pyramid • Test Design Practices • Living Documentation • Test Data Management Services • Service Virtualization • CI/CD • Shift Left Non-functional Testing • DevTestOps • Shift Left Integration Testing • Environment Provisioning • Production Testing and Monitoring • Automated Pipeline with Quality Gates DevTestSecOps Transformation • Coding Standards Micro services • • Code Review Cloud Native Development • • Static Code Analysis Application Transformation • • Code Quality Gates Refactoring • • Version Control, Branching • Unit Testing • Agile Practices • Engineering Culture 12

How We Are Achieving Our Testing Vision TESTING PYRAMID CONTINUOUS TESTING SHIFT LEFT Clear vision of the target state and Holistic delivery transformation incl. (Automated) quality gates to drive production-proven experience infrastructure and architecture quality in earlier development phases IN-SPRINT TEST AUTOMATION TOOLS & ACCELERATORS METRIC-BASED CONTROL Automation of various test types within EPAM custom tools and accelerators Continuous automated measurment the iteration covering the full test pyramid scope and improvement of QA metrics 13

Continuous Testing Blueprint Integration Artefact Test Repository QA Production UAT, INT Deployment Security PROD Functional & check APPROVAL Regression Mock / Test Unit Tests PROD Сode Analysis CI / Build Code Review PERF Server Performance Test Builds / Status Notifications Cloud Version Control Fully Automated CI/CD Pipeline • CI Built-in quality and Security gates • Dashboard Frequent and fast feedback • Feedback Developer 14

TAMAS_CSAKO@EPAM.COM WWW.LINKEDIN.COM/IN/TAMAS-CSAKO 15