Reliable Host Fencing In CloudStack Rohit Yadav (Software Architect) - PowerPoint PPT Presentation

Reliable Host Fencing In CloudStack Rohit Yadav (Software Architect) Boris Stoyanov (Sr. Software Test Engineer) rohit.yadav@shapeblue.com boris.stoyanov@shapeblue.com @rhtyd / @bsstoyanov The Cloud Specialists

A b o u t M e Rohit Yadav Boris Stoyanov • Software Architect @ • Senior Software Engineer Test C l i c k t o e d i t ShapeBlue @ ShapeBlue • • Contributor and Committer Contributor since 2016 since 2012 • Author and maintainer of CloudMonkey ShapeBlue.com @ShapeBlue The Cloud Specialists

A b o u t S h a p e B l u e C l i c k t o e d i t “ShapeBlue are expert builders of public & private clouds. They are the leading global CloudStack services company.” @ShapeBlue ShapeBlue.com The Cloud Specialists

S h a p e B l u e c u s t o m e rs C l i c k t o e d i t ShapeBlue.com @ShapeBlue The Cloud Specialists

S h a p e B l u e c u s t o m e rs C l i c k t o e d i t ShapeBlue.com @ShapeBlue The Cloud Specialists

S h a p e B l u e c u s t o m e rs C l i c k t o e d i t ShapeBlue.com @ShapeBlue The Cloud Specialists

W h a t i s H A ? High availability is a characteristic of a system, which C l i c k t o e d i t aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period. [source: wikipedia] ShapeBlue.com @ShapeBlue The Cloud Specialists

H A i n C l o u d S t a c k : S t a t u s Qu o • Currently HA is only supported for VMs by C l i c k t o e d i t CloudStack. • VM HA mechanism works for VMs that are marked HA. • Implementation tied to VM as a first class resource, asynchronously scheduled, limited to VM investigation/fencing/restart on new host. ShapeBlue.com @ShapeBlue The Cloud Specialists

H A i n P ro d u c t i o n : S t a t u s Qu o • Investigations are VM centric and not host centric. • C l i c k t o e d i t Limited fencing of host, highly unreliable. • VM HA may end up starting VMs on another host, while the VMs may be running on the faulty. Large environments see corrupt VMs and disks. • Unchecked faulty hosts and faulty neighbors, with no automatic-recovery. • Real world issues seen in a very large KVM environment. ShapeBlue.com @ShapeBlue The Cloud Specialists

A t t e m p t e d S o l u t i o n s : K V M • Check VM for disk activities based on a C l i c k t o e d i t timeout/threshold before re/starting VM. • (Wall) Clocks are not reliable • Maintenance and management issues • No recovery mechanism, fencing still remains unreliable References: https://issues.apache.org/jira/browse/CLOUDSTACK-8762 https://github.com/apache/cloudstack/pull/753 ShapeBlue.com @ShapeBlue The Cloud Specialists

L o n g Te rm S o l u t i o n ? • CloudStack needs a way to perform power C l i c k t o e d i t management tasks for hosts • Solve issues of corrupt disks due to VM HA and unreliable host fencing • Improve experience for admins: granular configuration, feature kill-switch , maintenance, management, reporting, alerts, investigations, reliable fencing and recovery etc. ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t P o w e r M a n a g e m e n t f o r C l o u d S t a c k • Implemented a pluggable out-of-band management framework for CloudStack C l i c k t o e d i t • Granular configuration per host, kill switch at zone/cluster/host level • Default plugin for IPMI 2.0 compliant hosts to support power operations: on, off, reboot, shutdown, status etc. • High quality tests, end-to-end testing based on ipmisim • DIY oobm plugin Reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack ShapeBlue.com @ShapeBlue The Cloud Specialists

B u i l d i n g B l o c k s f o r H o s t H A • Solve reliably fence/recover a host: use the new shiny out-of- band management subsystem C l i c k t o e d i t • What's missing: • Granular HA configuration • Host HA kill-switch: at zone/cluster/host level • Tuning: Threshold based investigation, activity checks, timeouts etc. • Task/Load management, circuit breakers, constraint based state transitions and operations Reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+HA+with+IPMI+Fencing ShapeBlue.com @ShapeBlue The Cloud Specialists

R e t h i n k H A • CloudStack organization units as partitions: Zone, Pod, Cluster, Host, VM. • Separate policy from mechanism: C l i c k t o e d i t Implement framework/managers to enforce policies, have plugins to carry out mechanisms • Define HA for a general resource, pluggable HA provider implementations. • Operational simplicity. • Granular configuration, kill-switch at zone/cluster/host level. Disabled by default. • Threshold based investigations, checking, fencing and recovery. • Leverage existing abstractions. • Integrated resource management. ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : D e s i g n a n d Im p l e m e n t a t i o n • HA Resource Management Service • HA resource lifecycle management C l i c k t o e d i t • HA resource type agnostic • Disabled by default, granular configurations, zone/cluster/host kill- switch, tuning • HA Provider • Resource specific HA plugin • Defines partition and resource type • DIY HA provider for partition: host/hypervisor/etc • One HA provider per resource type, per partition Reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : FS M S t a t e s E xp l a i n e d • HA Resource FSM States • Available C l i c k t o e d i t • Suspect • Checking • Degraded • Recovering, Recovered • Fencing, Fenced • Disabled • Ineligible ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : FS M S t a t e Tra n s i t i o n s C l i c k t o e d i t Reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : L i f e c yc l e m a n a g e m e n t • Granular HA configuration • Kill switch: enable/disable for a partition (zone/cluster/host) C l i c k t o e d i t • HA validation and ownership management • New Background Polling Manager for executor service management • Tasks executor, bounded (ephemeral) queue management • HA Polling tasks: Health Checks, Activity Checks, Recovery Task and Fence Task • FSM transitions based on task execution result • HA resource counter management: track investigation rounds, thresholds, timestamps, recover/fence operations ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : K V M H A P ro vi d e r • STONITH (Shoot The Other Node In The Head) fencing model • Activity check operations, checks for disk access activities on NFS storage C l i c k t o e d i t • Configurable activity check interval and activity checks • Tunable timeouts and thresholds • Request-reply model to check activity checks via adjacent eligible and healthy host(s) • Uses out-of-band management subsystem to carry out recover and fence operations • Recovery is attempted before fencing of the host • Alerting and reporting of operations ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : V M H A – H A P ro vi d e r C o o rd i n a t i o n • Remaps VM-HA host state Host HA VM-HA host state returned to VM-HA framework C l i c k t o e d i t state (KVM) returned based on Host HA states, only for hosts with Host HA enabled . Available Up • For Host HA to work effectively, Suspect/Checking Up (Investigating) existing VM HA framework to work Degraded Alert in tandem with Host HA. • By default Host HA is disabled, no Recovering/Recove Disconnected explicit configuration changes red/Fencing needed for existing users pre/post Fenced Down upgrade. • Currently, done for KVM Ineligible/Disabled -- HAProvider ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : Te s t i n g w i t h S i m u l a t o r H A P ro vi d e r • HA Provider for Simulator provides means and instrumentation to perform end-to- end deterministic testing of the framework. C l i c k t o e d i t • Provides means of validation of the feature and shows pluggability of the framework. • New Simulator APIs provides means of validating FSM sequences and instrumenting internal data structures. • Marvin based integration test, covers FSM transitions, HA operations, validations, configurations, HA ownership. ShapeBlue.com @ShapeBlue The Cloud Specialists

H o s t H A : Te s t i n g i n n e s t e d C l o u d S t a c k e n vi ro n m e n t • Recently, nested CloudStack environments such as Trillian, Bubble etc have tremendously helped with QA efforts. In such environments, hypervisor hosts are C l i c k t o e d i t VMs in another CloudStack environments. • As part of the FR, we've implemented a new out-of-band management plugin for nested CloudStack environment. • This plugin can perform power management operations to start/stop/reboot the host VMs. • The new oobm plugin allows for scalability and load testing of the Host HA feature in nested CloudStack environment. Currently being tested for a large KVM based environment. ShapeBlue.com @ShapeBlue The Cloud Specialists