Common Execution Infrastructure (CEI) Subsystem OOI CI System - PowerPoint PPT Presentation

Ocean Observatories Initiative Common Execution Infrastructure (CEI) Subsystem OOI CI System Architecture Team: R3 Kickoff Meeting 1

CEI Developers CEI Developer Patrick Armstrong CEI Senior Developer University of Chicago Pierre Riteau University of Chicago 2 CEI Developer John Bresnahan CEI Developer Argonne National Lab Pierre Riteau (part-time) University of Chicago (part-time) R3 Kickoff Meeting 10/18/12 2

Subsystem Purpose • Allow OOI applications and system to – Provide Highly Available (HA) services – Scale to demand • Enact OOI deployment policies in elastic environment • Provide a deployment foundation for OOI CI R3 Kickoff Meeting 3

Core System Structure: Service Layers R3 Kickoff Meeting 4

CEI Scope • Elastic Computing Services – Implement elastic computing services to provide on-demand scaling and high availability. • Execution Engine Catalog & Repository Services – Working with operations and ITV to develop and refine tools to upload and sync the different deployable type representations adapted to each site. • Process Management Services – Provide the management services for policy-based process execution within specified deployable types intended to support the data distribution services; as such the processes are sequential and require primarily a process to resource match. • Process Catalog & Repository Services – The Process Catalog and Repository Services maintain process definitions as well as lists active processes. • Integration with the National Computing Infrastructure – Provide the capability to deploy OOI processing on the Amazon cloud services as well as academic clouds R3 Kickoff Meeting 5

High Availability and Scaling • High Availability – Towards an always-on service model – Failures in outsourced resources – Providing a pool of replenishable compute resources • Autoscaling – Provide resources for peaks in demand – Ensure good utilization during “valleys” in demand – Flexible resource mix 6 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures 7 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures 7 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures 7 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures 7 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures 7 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures 7 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures EPU 7 R3 Kickoff Meeting 10/1 8/12

Resources for HA and Scaling – Cloud resources are available on-demand, but any particular resource may fail at any time – Applications/processes can absorb new resources – Applications/processes can tolerate failures EPU EPU Management Monitor and regulate set properties based on system-specific and application-specific metrics 7 R3 Kickoff Meeting 10/1 8/12

Managing Resources R3 Kickoff Meeting 8

Elastic Processing Unit (EPU) Management AMQP Other R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management AMQP EPU Management Other Decision Engine R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management instance Other Provisioner Decision Engine R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management instance Other Provisioner DTRS Decision Engine R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management instance Other Provisioner DTRS Decision Engine IaaS R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management instance Other Provisioner DTRS Decision Engine IaaS ou-agent ou-agent ou-agent EE ioncore 1.2 EE matlab 6.1 EE ioncore 1.3 context-agent context-agent context-agent R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management instance Other Provisioner DTRS Decision Engine IaaS CB ou-agent ou-agent ou-agent EE ioncore 1.2 EE matlab 6.1 EE ioncore 1.3 context-agent context-agent context-agent R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management instance Other Provisioner DTRS Decision Engine IaaS CB ou-agent ou-agent ou-agent EE ioncore 1.2 EE matlab 6.1 EE ioncore 1.3 context-agent context-agent context-agent R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management EPU Management instance EPU Management Other Provisioner DTRS Decision Engine IaaS CB ou-agent ou-agent ou-agent EE ioncore 1.2 EE matlab 6.1 EE ioncore 1.3 context-agent context-agent context-agent R3 Kickoff Meeting 9

Elastic Processing Unit (EPU) Management create AMQP EPU Management EPU Management instance EPU Management Other Provisioner DTRS Decision Engine IaaS CB ou-agent ou-agent ou-agent EE ioncore 1.2 EE matlab 6.1 EE ioncore 1.3 context-agent context-agent context-agent R3 Kickoff Meeting 9

Making the EPU HA AMQP Other R3 Kickoff Meeting

Making the EPU HA AMQP Other Bootstrap EPU Dedicated DE R3 Kickoff Meeting

Making the EPU HA AMQP Other create Bootstrap EPU instance Provisioner/DTRS Dedicated DE R3 Kickoff Meeting

Making the EPU HA AMQP Other create Bootstrap EPU instance Provisioner/DTRS Dedicated DE IaaS ou-agent ou-agent ou-agent EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker R3 Kickoff Meeting

Making the EPU HA AMQP Other create Bootstrap EPU instance Provisioner/DTRS Dedicated DE IaaS ou-agent ou-agent ou-agent EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker R3 Kickoff Meeting

Making the EPU HA AMQP Other create Bootstrap EPU instance Provisioner/DTRS Dedicated DE IaaS cloudinit.d ou-agent ou-agent ou-agent EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker EPU Worker R3 Kickoff Meeting

Managing Processes R3 Kickoff Meeting

Creating a Process I AMQP Other Process Dispatcher EE type A instance Decision Engine ee-agent R3 Kickoff Meeting 12

Creating a Process I AMQP Other Process Dispatcher EE type A instance Decision Engine ee-agent request to activate process X R3 Kickoff Meeting 12

Creating a Process I AMQP Other Process Dispatcher EE type A instance Decision Engine ee-agent lookup Process Definition Registry request to activate process X R3 Kickoff Meeting 12

Creating a Process I AMQP Other Process Dispatcher EE type A instance launch Decision Engine ee-agent lookup Process Definition Registry request to activate process X R3 Kickoff Meeting 12

Creating a Process I AMQP Other enter Process Instance Process Dispatcher EE type A instance Registry launch Decision Engine ee-agent lookup Process Definition Registry request to activate process X R3 Kickoff Meeting 12

Creating a Process II AMQP Other Process Instance Process Dispatcher Registry Decision Engine Process Definition Registry R3 Kickoff Meeting 13

Creating a Process II AMQP Other Process Instance Process Dispatcher Registry Decision Engine Process Definition Registry request to activate process X R3 Kickoff Meeting 13

Creating a Process II AMQP Other Process Instance Process Dispatcher Registry Decision Engine lookup Process Definition Registry request to activate process X R3 Kickoff Meeting 13

Creating a Process II AMQP EPU Management Other request instance Process Instance Process Dispatcher Registry Decision Engine lookup Process Definition Registry request to activate process X R3 Kickoff Meeting 13