EGU Vienna 2015 Supervising simulations with the Prodiguer Messaging - PowerPoint PPT Presentation

EGU Vienna 2015 Supervising simulations with the Prodiguer Messaging Platform Mark A. Greenslade, Nicolas Carenton, Sebastien Denvil Institut Pierre Simon Laplace, Paris, France

CONVERGENCE Big Data and Exascale challenges for Climate Sciences http://convergence.ipsl.fr

Message Flow

IDRIS TGCC XXX CINES IPSL SMTP Relay SMTP Relay SMTP Relay SMTP Relay SMTP Relay msg msg msg msg msg MQ Cluster IPSL IPSL MQ Apps DB’s API json IPSL User @ Browser | Command Line | Desktop

Flow 1: CC.NODE ---> IPSL.SMTP Compute Node @ Computing Centre SMTP simul liblIGCM args args client (fortran) (ksh) msg-batch SMTP @ IPSL

Flow 2: IPSL.SMTP ---> IPSL.MQ SMTP @ IPSL msg-batch MQ Cluster @ IPSL (ampqs + rabbit-mq)

Flow 3: IPSL.MQ <---> IPSL.MQ-APP MQ Cluster @ IPSL (ampqs + rabbit-mq + shovel) tcp/ip (port=5672) msg msg sim-mon metrics esg-f es-doc api smtp img (python) (python) (python) (python) (python) (python) (python) primary secondary MQ APPS @ IPSL

Flow 4: IPSL.MQ-APP ---> Other MQ APPS @ IPSL primary secondary sim-mon metrics esg-f es-doc api smtp cv (python) (python) (python) (python) (python) (python) (python) tcp/ip https wss smtp https FE @ API API SMTP DB’s @ IPSL GitHub browser @ ESG-F @ ES-DOC @ IPSL (Relational, NO-SQL, Graph)

Flow 5: IPSL.API ---> IPSL.FE API @ IPSL (tornado http & ws server) sim-mon (python) https / wss json sim-mon (javascript) FE @ browser (jquery, backbone, lo-dash, bootstrap)

Rabbit MQ « Messaging that just works »

Rabbit MQ - Security — AMPQS = transport level akin to HTTPS — Authentication = SASL PLAIN (other schemas supported) — Authorization = user accounts are allocated permissions to MQ vhost, exchange, queue with regular expressions

Rabbit MQ – Message Structure — Header = AMPQ Basic Properties — Key / Value pairs — Common headers mandated by AMPQ spec — Prodiguer injects custom headers — Body — Text blob — Content type / encoding specified in header — Platform passes decoded messages to consumers

Rabbit MQ - Durability — Ensuring buffered messages aren’t lost if server crashes — RabbitMQ persists messages to disk — Messages marked as persistent and exchanges as durable — Incurs performance overhead

Rabbit MQ - Reliability — Ensuring messages are reliably published / consumed — Publishers: — Receive confirmation when message reaches exchange — If unconfirmed then need failure strategy such as “retry 3 times and then save to local file system” — Consumers: — Acknowledge when unit of work succeeds — Acknowledged messages are removed from queue — Unacknowledged messages require failure strategy, e.g. retry

Rabbit MQ - Routing — MQ producers & consumers bind to exchanges — Producers specify a “.” delimited routing key: — [mode].[user-id].[producer-id].[app-id].[type] — Consumers specify a routing key filter: — e.g. *.*.*.*.0000 = simulation initialisation messages — Routing objective: optimise ratio of queues to workers so that queues are as empty as possible

Summary

Summary - I Secure Lossless Scalable Extensible Open Source Real time Non-intrusive

Summary - II Simulation monitoring & control Data publishing Documentation publishing Simulation metrics publishing HPC diagnostics aggregation Controlled vocabulary management Web Socket / SMS / SMTP push notifications