DIABLO VS HAVANA HOW OPENSTACK HAS MATURED Joe Gordon ABOUT ME - PowerPoint PPT Presentation

DIABLO VS HAVANA HOW OPENSTACK HAS MATURED Joe Gordon

ABOUT ME Developer @ Full time upstream OpenStack developer nova-core ATC since Essex Interested in making OpenStack scale and work contact information jog0 on freenode github.com/jogo launchpad.net/~jogo

OPENSTACK FROM DIABLO TO HAVANA A BRIEF HISTORY OF THE INTEGRATED RELEASE

DIABLO Swift Nova Nova-volume Nova-network Identity Glance Keystone*

ESSEX Swift Nova Nova-volume Nova-network Identity Glance Keystone Horizon

FOLSOM Swift Nova Nova-volume Nova-network Glance Keystone Horizon Quantum (Neutron) Cinder

GRIZZLY Swift Nova Nova-network Glance Keystone Horizon Quantum (Neutron) Cinder

HAVANA Swift (4) Nova (6) Nova-network Glance (2) Keystone (1) Horizon (1) Neutron (6) Cinder (4) Ceilometer (4) Heat (4) Total number of OpenStack services: 32

HAVANA FEATURES SUPPORTED BY DIABLO Swift Nova Nova-network Glance Keystone Horizon Neutron (Quantum) Cinder Ceilometer Heat

HAVANA FEATURES SUPPORTED BY DIABLO Swift Nova Nova-network Glance Keystone Cinder

OPENSTACK HAS GROWN

GROWN OUT NEW SERVICES Heat Ceilometer More in pipeline Icehouse Trove Incubation Ironic Marconi Savanna

GROWN UP Expand existing APIs Split out projects Extensions

DIABLO NOVA EXTENSIONS | Hosts | Host administration | Volumes | Volumes support | Quotas | Quotas management support | SimpleTenantUsage | Simple tenant usage extension | FlavorExtraData | Provide additional data for flavors | SecurityGroups | Security group support | FlavorExtraSpecs | Instance type (flavor) extra specs | Createserverext | Extended support to the Create Server v1.1 API | Multinic | Multiple network support | VSAs | Virtual Storage Arrays support | VolumeTypes | Volume types support | Keypairs | Keypair Support | Rescue | Instance rescue mode | VirtualInterfaces | Virtual interface support | Floating_ips | Floating IPs support

POST DIABLO NOVA EXTENSIONS AdminActions Agents Aggregates AssistedVolumeSnapshots AttachInterfaces AvailabilityZone BareMetalExtStatus BareMetalNodes BlockDeviceMappingV2Boot CellCapacities Cells Certificates Cloudpipe CloudpipeUpdate ConfigDrive ConsoleOutput Consoles Coverage DeferredDelete DiskConfig Evacuate ExtendedAvailabilityZone ExtendedFloatingIps ExtendedIps ExtendedIpsMac ExtendedQuotas ExtendedServerAttributes ExtendedServices ExtendedStatus ExtendedVIFNet ExtendedVolumes FixedIPs FlavorAccess FlavorDisabled FlavorManage FlavorRxtx FlavorSwap FloatingIpDns FloatingIpPools FloatingIps FloatingIpsBulk Fping HideServerAddresses Hypervisors ImageSize InstanceActions Migrations MultipleCreate NetworkAssociationSupport Networks OSInstanceUsageAuditLog OSTenantNetworks QuotaClasses SchedulerHints SecurityGroupDefaultRules ServerDiagnostics ServerPassword ServerStartStop ServerUsage Services Shelve UsedLimits UsedLimitsForAdmin UserData UserQuotas VolumeAttachmentUpdate

SO ONCE WE NORMALIZE FOR THE GROWTH IN OPENSTACK WHAT ELSE HAVE WE DONE IN THE PAST FOUR RELEASES?

CHANGES Drivers Testing and bugs Performance and scalability Forward facing internal changes Process

MORE DRIVERS Nova: baremetal, docker, hyperv, powervm Cinder: coraid, emc, glusterfs, gpfs, nexenta, rbd, sheepdog, soldfire, ...

BETTER TESTING, FEWER BUGS Unit tests Integration tests (Tempest) Lots of fixed bugs

UNIT TESTS PERCENT UNIT TESTS BY LOC Project Diablo Havana Nova 43% 56% Glance 62% 62% Swift 65% 71% Keystone 41% 45%

TEMPEST THE OPENSTACK INTEGRATION TEST SUITE

DIABLO TEMPEST Orignally named 'openstack-integration-tests' Just 107 tests* extensions flavors floating ips image metadata images keypairs servers volumes 2,655 lines of python code First release of Tempest

HAVANA TEMPEST 1275* tests 1150 api tests compute, image, identity, network, object_storage, orchestration, volume 102 test files 92 cli tests 11 scenario tests 18 thirdparty (boto) tests 4 other Parallel 20-30 minutes to run in gate 24,837 lines of python code

HAVANA API TESTS 785 compute 216 identity 24 image 17 network 50 object_storage 54 volume

HAVANA TEMPEST VS OPENSTACK DIABLO Same APIs as Diablo plus new ones Diablo has been 'end of life'ed Many tests cover new features. Ignore those Tempest not designed to be version independent Got 622 tests to run and 154 (27%) to pass.* 110 of which were in compute

HAVANA TEMPEST VS OPENSTACK DIABLO RESULTS Diablo has poor parameter validation Invalid key names Min/max count for multiple creates Changed return codes for negative cases Return empty list and 200 instead of 400 when invalid filter value in nova list servers We shouldn't have done this Poor pagination in Diablo JSON only tests in Diablo, Havana tests XML too Many false negatives in results due to slight changes in now APIs and metadata are used

CRITICAL BUGS Project Diablo Essex Folsom Grizzly Havana Nova 14 22 30 31 33 Glance 5 15 30 13 8 Keystone 31 11 9 8 Cinder 10 3 12 Neutron 27 14 24 Categories Non-gated features Dependencies Race conditions Performance

PERFORMANCE AND SCALABILITY

PERFORMANCE AND SCALABILITY Nova-scheduler Active-active No RPC fanout DB Indices Smarter queries Fewer queries service group Heartbeat to DB every 10 seconds 1k DB calls a second if have 10K nodes. Adjustable interval Alternatives memcache and zookeeper

PERFORMANCE AND SCALABILITY More efficient image work flow Glance, Nova, Cinder Move large files around less PKI Tokens Fewer round trips to keystone

PERFORMANCE AND SCALABILITY CASE STUDY FROM HP PUBLIC CLOUD Diablo 40,836 LOC changed to 239 files in nova 2,265 LOC to manage DB +100s of hours of DB tuning to make it work with more then just a few nodes Havana DB just worked 1,167 LOC changed in 35 files Most of that is in patches in review today Thanks to Thom Leggett for these statistics

FORWARD FACING INTERNAL CHANGES Split out projects Oslo Road to live and continuous upgrades Iterate on APIs

PROCESS

DRIVING PRINCIPLES Never break trunk Master branch is always green Developers are never blocked Transparency Automate everything Egalitarian Be strict. Reduce burden on reviewers

DIABLO Automated Gate 87 Contributors in September 2011 (release of Diablo) No Gate No Git or Gerrit Bzr and Launchpad

HAVANA 346 Contributors in October 2013 (release of Havana) Scale development process Integrated Gate 9 Projects 30+ separate services 1 Merge pipeline New tools for an asynchronous world Tests not pass fail, rather classified with a race percentage Fix that race condition we hit 0.5% of the time Debugging is hard elastic-recheck

HOW WILL OPENSTACK KEEP MATURING WHAT CHALLENGES ARE WE FACING TODAY? More testing Live rolling upgrades Limiting factor to the whole system is reviewers (and not just core) Improving quality while growing the project as a whole Scalability

THANK YOU QUESTIONS? Powered by reveal.js