OpenStack Stable What it actually means to maintain stable branches Matthew Treinish (irc: mtreinish) Matt Riedemann (irc: mriedem) Ihar Hrachyshka (irc: ihrachys) Newton Summit, Austin, TX - 2016/04/25
Agenda Introductions ● Who cares? ● What is it? ● How is it done? ● Q & A ● 2
Introductions Matthew Treinish (HPE) ● Core reviewer on QA projects and stable-maint-core ○ QA PTL from Juno through Mitaka ○ TC Member ○ Matt Riedemann (IBM) ● Core reviewer on Nova projects and stable-maint-core ○ Nova PTL for Newton ○ Ihar Hrachyshka (Red Hat) ● Core reviewer on Neutron projects and stable-maint-core ○ 3
4
Who cares about stable branches? Production clouds (per April 2016 user survey) ● 5
Who cares about stable branches? Distributions ● Red Hat, Mirantis, SUSE, Canonical, etc ○ Release their latest product from stable branches after ○ they have been ‘stable’ for a while already (2-6 months) Support generally runs far longer than upstream ○ Security fixes ○ Production clouds that roll their own packages and patches ● 6
What is “stable branch maintenance”? Enforcing policy ● What is the policy and who enforces it? ○ Backporting fixes ● What’s appropriate and how are these identified? ○ Keeping the CI system running ● What are the issues and how are they fixed? ○ 7
What is the stable branch policy? Support phases ● Phase 1 (first 6 months): all appropriate bug fixes ○ Phase 2 (6-12 months): critical / security fixes ○ Phase 3 (12+ months): security fixes ○ Appropriate fixes ● No features or backward incompatible changes. ○ Clear user impact/benefit. ○ Self-contained and no new dependencies. ○ 8
What is the stable branch policy? Team structure ● Project-specific teams: identifies potential backports, reviews ○ backports, requests releases, monitors CI status. Stable-maint-core: approves project-specific stable core ○ additions and guides the project-specific teams on policy. Review guidelines ● Fixes must be in upstream branches first, using cherry picks ○ and the same Change Id and fall within the support phase. 9
How are stable branches maintained? Maintaining stable branches involves knowing the Continuous ● Integration system as well as actually identifying fixes to backport, reviewing backports and requesting point releases. 10
Periodic Stable Jobs Nightly runs of unit tests, and tempest for each supported branch ● Failures reported to openstack-stable@lists.openstack.org ● All results viewable on openstack-health ● 11
Periodic Stable Jobs 12
Branchless Tempest Starting in Icehouse Tempest doesn’t branch on releases ● Tempest supports running against all stable branches ● Ensures API backwards compatibility between releases ● Each proposed Tempest commit runs against master, and all ● supported stable branches Much more throughput than periodic stable jobs, often the first to ● catch issues 13
Types of failures Upstream Bugs in Bugs in Tests Bugs in Service Infra Breaks dependencies OpenStack Breaks Examples: Examples: Examples: Examples: Examples: -Use of -IP Allocation - Public cloud - Bad nodepool timestamps bugs - Kernel bugs outages images - Global state - races w/ multiple - Incompatible - service outages corruption workers requirements - mirrors broken - races w/ async messaging Get bug reported upstream, pin Backport fixes to Fix on master if versions in upper unit and functional still applicable, constraints and tests when Make infra more Weather the backport fixes global- appropriate, fix resilient storm where possible requirements tempest bugs 14
Stable Failure Rates over Time Tempest Failure Rates Number of Total Individual Tests 1 Month before Juno EOL Run Daily Per Branch 15
Case study: the Neutron process 16
Bug fix flow: ideal All bugs fixed on master are also backported to stable (if affected) master mitaka liberty distro distro distro mitaka liberty kilo-eol 17
Bug fix flow: reality Sometimes stable gets a fix, if someone cared (read: was affected). master mitaka liberty distro distro mitaka distro liberty 18
Backport starvation: reasons Backports done “on demand” ● No process to identify bug fixes to backport ● No process to track backports to merge ● It’s hard work and not very well understood (and not tracked in ● Stackalytics) 19
Starvation solutions: neutron/Liberty+ Backports done weekly ● Get major stable branch consumers on board ● Implement tools and process to manage backport flow ● automate bug fix classification and review proposal process ○ goodies landing in openstack-infra/release-tools ○ 20
Starvation solutions: neutron/Liberty+ $ ./bugs-fixed-since.py neutron <previous-hash> | ./lp-filter-bugs-by-importance.py neutron | ./annotate-lp-bugs.py neutron … https://bugs.launchpad.net/bugs/1513765 "bulk delete of ports cost iptables-firewall too much time" (Critical,Fix Released) [bridge,in- stable-liberty,liberty-backport-potential,linuxbridge,mitaka-rc- potential,ovs,sg-fw] [kevinbenton] ... 21
Starvation solutions: neutron/Liberty+ $ … | grep -v in-stable-mitaka $ … | grep linuxbridge $ … | grep -v in-stable-liberty | grep liberty- backport-potential ... 22
Starvation solutions: neutron/Liberty+ $ ./lp-reset-backport-potential.py neutron python- neutronclient Processing project neutron… Removing liberty-backport-potential tag for 123456 Removing liberty-backport-potential tag for 234567 Processing project python-neutronclient… Removing liberty-backport-potential tag for 345678 Removing liberty-backport-potential tag for 456789 23
Starvation solutions: gotchas More backports - higher chance of regression, but ● author is usually aware of an issue ○ still, it’s worth it ○ 50%+ downstream escalations could be avoided ■ Frequent releases needed for more granular updates ● openstack/releases: ./new_release neutron mitaka ○ 24
Starvation solutions: future Adopt proactive backport practice for other interested projects ● Things missing: ● clear documentation on the process ○ complete backport pipeline automation ○ more involvement from other major contributors ○ 25
Where to get more information ● #openstack-stable channel on Freenode IRC ● openstack-dev mailing list: http://lists.openstack.org/cgi- bin/mailman/listinfo/openstack-dev ● Docs: http://docs.openstack.org/project-team-guide/stable-branches.html ● Weekly IRC meeting: https://wiki.openstack.org/wiki/Meetings/StableTeam ● Periodic Stable Test Results: http://status.openstack.org/openstack-health/#/g/build_queue/periodic-stable 26
Questions? 27
Recommend
More recommend
