Lessons Learned from a Large-Scale Telco OSP+SDN Deployment Guil Barros, Sr. Pr. Product Manager OpenStack, Red Hat Vicken Krissian, Sr. Technical Project Manager, Red Hat Cyril Lopez, Sr. OpenStack Consultant, Red Hat
Agenda Who we are ● ● The Timeline of an OSP engagement ● Challenges ● New Functionality & How to get there DCI & Testing Automation ● ● The future of 3rd party deployments - Composable Services/Roles
Who We Are Cyril Lopez ● Red Hater from eNovance acquisition ● Cloud consultant and devops ● Committing in tripleo project Vicken Krissian ● Red Hater from eNovance acquisition ● Senior Technical Project Manager ● Lead and contribute to the delivery of Cloud projects using Iterative methodology Guil Barros Long-time Red Hater, Consulting, Support, OSP BU ● ● Handles partner engagement for OSP partners
A Timeline of an OSP Engagement RFP Answer RH Services started Fully working POC Delivery RH OSP 8 and SDN platform Pre-Production platform 2016 2015 July Feb March June July Sep Nov Dec User Acceptance Tests Integration Hackfest Go Live Deal signed SDN Integration for OSP 7/ upstream RFP published
Project Challenges Customer ● Going to production (Q4 ‘16.) ○ ○ Expect mature product with full documentation on integrated platform. ○ Familiar working with providers not with communities. ● Red Hat Full integration scheduled on RHT OSP 10 (Q4 ‘16.) ○ ○ Focusing on product rather than services. ○ Upstream first for all code (opensource model.) ● SDN Partner Certification on RHT OSP 10 (Q1 ‘17.) ○ ○ Proprietary solution.
Integration Challenges The first version of Red Hat OpenStack director (based on TripleO) ● delivered in summer 2015 (no SDN integration at the time.) ○ Initial release pains. ● Partner familiar with OSP packstack, needs to learn OSP director. ● Initial certification process not using OSP director + SDN integration. Plan for complete and official SDN integration with RHT OSP 10 but had to ● deal with RHT OSP 8 and 9 with limited resources and process in the meantime ● Certification delays (up to 6 months) Difficulties reproducing the joint platform for testing and certification ●
People Challenges The Customer wants to be autonomous (learning from doing) from the ● beginning with little support in terms of RHT services : ○ Only one dedicated engineer onsite. ○ No workshops to define all together the design and deployment phases. Customer team was new to RHT OpenStack, RHT OSP director and all ● related technologies. ○ Slow deployment and learning pace. ● RHT and SDN partner not having deep knowledge and skills on other partners technology and difficulties to find people with both expertise on the project.
Implementation Challenges Telco sensitive to change. ● Build a solution with Telco security requirements. ● ○ Making OSP Secure, not making a secure OSP. ● Build an easy to deploy solution. ● Integrate SDN controller with a pattern. Define a complete lifecycle. ● ● Certification timeline with new partners. ○ Line up with Customer deadlines. ○ SDN release timeline - OSP is a fast moving target.
Integration Hackfest Objectives ● ○ Having Partner and Red Hat engineering/consultants working together face to face and sharing Knowledge. ○ Verify and validate the technical feasibility and supportability of Integration Use Cases. ● Deliverables ○ Documentation and details procedure to reproduce the implementation on the field. New bugs found with documented workaround. ○
What Red Hat brings to the table Our capability to drive changes. - Upstream development - Project management - Integrated product Many features are new or require development. Lots of investment from all sides to deliver requested functionality. Not a product, an ecosystem.
Feature requests - Upstream development - Existing or entirely new? - Can you get an upstream sponsor to help? - Will it be accepted by the community? - Timelines & Roadmaps - When can we bring the new feature into the GA product? - Does it require special handling from Support? - Do the timelines from all partners line up feature-wise? - What about lifecycle-wise? (i.e. Partner-A LTS release w/ Partner-B LTS release)
Distributed CI - The Long-term Vision - The single largest time saver available. - Automated testing with each GA and pre-release. - Test with RHT-internal OSP puddles. - Takes time to set up and create test harnesses. - But worth every second! - Loop it into your certification chain!
Why Distributed CI? DCI aims to change the way you interact with Red Hat OpenStack. ● Allows Partners and Customers to work on the latest Red Hat OpenStack builds available, as soon as possible, in their own environment. ● Shortens the time and effort for RHOSP Certification since it can run the certification test suite on every change. ● Simplifies the upgrade progress by testing incremental changes. ● Provides automated feedback to Red Hat with helpful logs. ● Focus debugging efforts on the subset of the code that has changed. ● Include RHOSP tests as well as Partner's and customer's specific tests.
Communication - Have a good long-term idea of where you need to be and when. - Which functionality is due when? - Specific version or testing requirements? - Three-way communication is critical. There are many subgroups in each company, all with different touch-points. Having official communication points of contact makes life easier. - Consistent sync-up calls with all partners and customers. It’s surprising how fast misinformation spreads.
Composable Roles? Current OSP Nodes have fixed roles: Controller, Compute, Storage. ● ● Composable roles allows you to define the base OSP roles. ○ Neutron services on Controller 1,3,5 ○ Keystone + Others on Controller 2,4,6 3rd party services inside their own roles ● ○ SDN WebUI + Cassandra on Node 1,3 ○ SDN Manager on Node 2,4 The roles_data.yaml definitions look like this: - name: Controller CountDefault: 1 ServicesDefault: - OS::TripleO::Services::CACerts - OS::TripleO::Services::CephMon - OS::TripleO::Services::CinderApi
Test, Test, Test - Hackfests are your friend - Include all stakeholders IN PERSON! - Engineering also, not just frontline! - File and follow up on bugs while fresh. - Have goals and create a document as the output. - For each major version - Re-implement with prior hackfest document - Note changes, regressions, etc - Create a new document
For next time... - Discovery/Design Workshop at the beginning of the project in order to : - Have all teams involved with the same level of information. - Share detailed Architecture Review. - List requested functionality and implementation targets. - Create feature roadmaps and define clear ownership. - Define communication plans. - Align Upstream, RHT, SDN, and Customer roadmaps early. - Lock in features, architecture, and timelines for all four workstreams. - Put in place a shared integration platform in order to : - Test Early and Often. - Implement CI as soon as possible. - Make more Hackfests !
Questions?
THANK YOU
Recommend
More recommend
