adv advanc anced insta ed instance nce sc schedu heduling by - PowerPoint PPT Presentation

Nov ovembe mber r 2018 18 Berlin Be in How can p How c an plac lacement hel ement help you p you ach achiev ieve e adv advanc anced insta ed instance nce sc schedu heduling by ling by integr int egrate with diffe ate with differe rent nt se servi rvices ces Zhenyu Zheng irc: Kevin_Zheng Huawei Yikun Jiang irc: Yikun Huawei Sheng Hu irc: Tommylikehu Huawei

Ab About us out us • Zhenyu nyu Zheng ng Huawei Technologies Co., Ltd. OpenStack Nova Contributor, Upstream Developer. • Yi Yikun Jiang ang Huawei Technologies Co., Ltd. OpenStack Nova & Cinder Contributor, Upstream Developer. • Sheng ng Hu Hu Huawei Technologies Co., Ltd. OpenStack Cinder Core Reviewer, Upstream Developer.

Con Contents tents • Placement in a Nutshell • How Nova uses Placement and what problems did it solve • How other services interact with Nova through Placement and what feature did we achieve • What can users expect for Stein?

Pl Place acement ment in in a nu a nutshe tshell ll

Pl Place acement ment in in a Nu a Nutsh tshell ell - His History tory • Introduced in Newton(14.0.0) release as a part of Nova. • Goal: Enable more effective accounting of resources in an OpenStack deployment and better scheduling of various entities in the cloud[1]. • A separate RESTful API stack and data model used to track resource provider inventories and usages, along with different classes of resources[1].

Pl Place acement ment in in a Nu a Nutsh tshell ell - No Now • Added a lot of features in the past 4 releases, made it more powerful and easier to use [1]: - Allowing define custom resource classes (Ocata) • - Added Traits APIs in Pike and allow query RPs by Traits in Queens • - Allowing query Allocation Candidates that are members of aggregates (Rocky) • • More and more services are starting to consider to adopt Placement. • Placement is now going through extractions.

Pl Place acement ment in in a Nu a Nutsh tshell ell - Con Contents tents • This session will focus on high-level resource abstractions and workflows. • Er Eric ic Fr Fried ied and Ed Ed Le Leafe afe have given a very good session in Vancouver about implementation details with an very interesting example: • https://www.openstack.org/videos/vancouver-2018/placement- present-and-future-in-nova-and-beyond

Pl Place acement ment in in a Nu a Nutsh tshell ell - Fr From om 100 1000 Fe 0 Feet et • Placement service is straightforward: • A WSGI application send/receive Placement API JSON requests; • A RDBMS for data persistence. • State is managed solely in the DB. Thus RDBMS scaling the placement service could be done by increasing the number of WSGI app instances and scaling the RDBMS using traditional database scaling techniques.

Pl Place acement ment in in a Nu a Nutsh tshell ell - De Deployment ployment • A sample deployment at CERN[2]. • Placement deployed together with Cells V2. • Overall 16 Placement services & 70 cells(200 compute nodes in each cell) made a successful deployment of 14,000 compute nodes in one region. [2] https://www.openstack.org/videos/vancouver-2018/moving-from-cellsv1-to-cellsv2-at-cern

Pl Place acement ment in in a Nu a Nutsh tshell ell - Con Concepts cepts • Re Resource ource Providers oviders: An abstraction data model representing the object that provides certain type/number of resources tracked by placement service, such as com ompute pute no node de & storage orage po pool ol. • Re Resource ource Cl Clas ass: Types of resources, there are standard resource classes (for example DISK_GB, MEMORY_MB, and VCPU) and custom resource classes (prefixed with CUSTOM_*) • In Inventories: entories: Qua uantity ntity of different resource classes that each resource provider can provide, for example, RP_1 has the inventory of 10 100 0 DI DISK_GB SK_GB, 20 2048 48 ME MEMORY_MB MORY_MB and 8 VCP 8 VCPU. • Traits: aits: Describe qu qualitative alitative aspects of the resource provider, for example, the DI DISK_GB SK_GB provided by RP_1 might be soli olid d state ate dr drives ives (S (SSD) SD), , so we can set a `` ``is is_SSD SSD`` `` traits aits fo for RP RP_1 _1.

Pl Place acement ment in in a Nu a Nutsh tshell ell – Con Concepts cepts • Co Cons nsumer umers: s: The user who occupied resources from the resource providers, for example, an instance is a consumer for RP_1(a compute node), who consumed 10 DISK_GB, 1024 MEMORY_MB and 4 VCPU. • Allocations: locations: The data model used to store the resource occupation relationship between resource providers and consumers, a typical allocation record could be consumer_1 occupied 4 unit of resource_1 from resource_provider_1. • Allocation location Can Candi didates dates: : Placement will provide a group of resource providers that are suitable for the requests, they are called the allocation candidates, callers can then use these candidates as an input for their own filter and sort process to select the best candidate.

Pl Place acement ment in in a Nu a Nutsh tshell ell – Con Concepts cepts Compute Node 1 ID = 0 Resource Providers • Ne Neste ted d Re Resource ource Provid oviders: ers: 100 DISK_GB Inventories SSD In Queens release, placement Traits Parent = NULL Root = 0 introduced the ability to allow hierarchical relationship NUMA Cell 0 NUMA Cell 1 between different resource ID = 1 ID = 2 providers to be represented. 8 VCPU 8 VCPU 4096 MEMORY _MB 4096 MEMORY _MB This is very useful for users to represent resources like NUMA Parent = 0 Root = 0 Parent = 0 Root = 0 nodes and SRIOV_NET_VFs. Physical Function 0 Physical Function 1 ID = 3 ID = 4 8 SRIOV_NET_VF 8 SRIOV_NET_VF HW_NIC_OFFLOAD_GENEVE Parent = 1 Root = 0 Parent = 2 Root = 0

Ho How Nova w Nova us uses es Pl Place acement ment & wh & what at pr problems oblems di did d it it so solve lve

No Nova va wor workflow kflow - Pr Problems oblems • In Incor orrect rect res esourc ource e us usag age e rep epor orting: ting: Due to legacy reasons, Nova considers resources are only being reported by a compute node, when reporting, Nova naively calculates resource usage and availability by simply summing amounts across all compute nodes in its database, causing number of problems. • Lar Large ge Sc Scale ale Pr Prob oblem: lem: When scheduling, Nova scheduler retrieves a list of all compute node in the entire deployment and loops through them across all the filters enabled in the deployment, it is extremely wasteful and this inefficiency gets worse the larger the deployment is.

No Nova va wor workflow kflow - Pr Problems oblems • Cr Cros oss-pr projec oject t sche heduling: duling: It was very hard to leverage Nova scheduling and advanced scheduling features by other service(like routed network functionality provided by Neutron) at the same time to achieve an more advanced scheduling process. Using a more generic resource management service makes it much more easier.

No Nova va wor workflow kflow - Repor Report • Co Comp mput ute e No Node de rep epor orts ts to to pl plac acemen ement: t: Logic added in nova-compute(resource_tracker ) to report its’ available • resources to placement as a resource provider and related inventories. Currently only reports VCPU, DISK_GB, MEMORY_MB and VGPU, and • also starts to report some CPU features as traits[3]. Syncing periodically and during initialize, done by request to placement with: • PUT /resource_providers/{rp_uuid}/inventories: { ‘ resource_provider_generation ’: 66, ‘inventories’: { ‘VCPU’: 16, … }}

No Nova va wor workflow kflow – Sc Schedu heduling ling • Nova-Scheduler gather all scheduling related parameters(such as VCPU, MEMORY_MB, DISK_GB, etc. ``Flavor Extra_Specs`` and ``image_properties`` will be translated to ``Traits`` requests. • Nova-Scheduler will call Placement as: • GET • /allocation_candidates? • resources=VCPU:1,MEMORY_MB:1024,DISK_GB:100&required=SSD • OpenStack-API-Version: placement 1.10 (Maximum in Pike)

No Nova va wor workflow kflow – Sc Schedu heduling ling • A typical response will be: { "prov ovider_summa ider_summari ries": es": { "0bd25 d25be bea-5adc adc-4b3 4b39-ac4d c4d-acd6e acd6e98 98d24 d2439 39": ": { "trait its": s": [ "HW_ W_CPU CPU_X _X86_S 86_SSE2" E2"... .. ], ], "resou sources rces": : { "VCPU PU": ": { "used sed": ": 6, "capacit pacity": ": 256 }, }, "MEMOR ORY_MB MB": ": { "used sed": ": 5120, "capacity pacity": ": 59568 }, }, "DISK_ K_GB GB": ": { "used sed": ": 22, "capacity pacity": ": 837 } } } … }, }, "allocation ocation_requ requests": ests": [ { "allocations ocations": ": { "0bd bd25bea 5bea-5a 5adc dc-4b39 b39-ac4d ac4d-acd acd6e98 e98d24 d2439": 39": { "resou sources": ces": { "VCPU": U": 1, "MEMORY ORY_MB MB": ": 512, "DISK_G _GB": B": 1 } } } } ] }