cylc from the ncas point of view ncas experience with
play

Cylc from the NCAS point of view (NCAS Experience with Rose/Cylc) - PowerPoint PPT Presentation

Oct 26, 2022 •33 likes •223 views

Cylc from the NCAS point of view (NCAS Experience with Rose/Cylc) Rosalyn Hatcher, Annette Osprey, NCAS-CMS (Grenville Lister) Joint final IS-ENES2 workshop on Workflow Solutions in Earth System Modelling and Meta-Data Generation during

  1. Cylc from the NCAS point of view (NCAS Experience with Rose/Cylc) Rosalyn Hatcher, Annette Osprey, NCAS-CMS (Grenville Lister) Joint final IS-ENES2 workshop on Workflow Solutions in Earth System Modelling and Meta-Data Generation during Experiments, Lisbon, Sept 2016

  2. NCAS Rose/Cylc - Outline • NCAS-CMS – who we are • Workflow - Historical Overview • Rose/Cylc – Platforms – Suites – Management – Training – Future directions

  3. NCAS Rose/Cylc – who we are NCAS Computational Modelling Services - http://cms.ncas.ac.uk

  4. NCAS Rose/Cylc - Historical Code repositories trunk mirrored Institutional from MO/IPSL Commits/Extracts compute/storage Working copies Job configuration Job submission UMUI (jobs?) Local to PUMA COMPUTE Mobilis Polaris HPC MONSooN ARCHER (NOC) (Leeds) Wales POST PROCESSING MONSooN ARCHER JASMIN PP PP, RDF LOTUS, MASS cluster, SCI VMs JASMIN VM

  5. UM software (pre vn9.0) PUMA UMUI Database of user jobs FCM Graphical job editor Code manager Compilation and build HPC OASIS Atmosphere model Reconfiguration Coupler Dynamical core Prepares initial model Physics state Diagnostics (STASH) [JULES and/or UKCA] Ocean model Output file tools Input file tools Local / Jasmin Data processing Prepare ancillary data Analysis and visualisation

  6. NCAS Rose/Cylc - Historical • Discovery – ad hoc • Manage code – software engineering tools • Configure/reconfigure experiment – manual process • Manage job submission How this works is highly • Manage job failure/continuation dependent in individual users and frequently involves a good deal on • Manage output manual intervention. • Post process • Archive

  7. NCAS Rose/Cylc - MOSRS M et O ffice S hared R epositories (UM, JULES, etc) Updated every 5 mins another site PUMA Local Mirror Local Mirror Make Make of Shared of Shared code changes code changes Repositories Repositories Extract code into Extract code into Rose suite Rose suite

  8. NCAS Rose/Cylc Code repositories mirrored from Institutional MOSRS Commits/Extracts compute/storage Working copies Job configuration Rose/Cylc Job submission MOSRS Local to PUMA COMPUTE HPC Mobilis MONSooN Polaris ARCHER Wales POST PROCESSING MONSooN ARCHER JASMIN PP PP, RDF LOTUS, MASS cluster, SCI VMs JASMIN VM

  9. UM software (vn10.0 onwards) PUMA MOSRS UMUI Rose Database of user jobs FCM Graphical job editor Cylc / Rose Code manager Job submission Workflow manager Compilation and build HPC OASIS Atmosphere model Reconfiguration Coupler Dynamical core Prepares initial model Physics state Diagnostics (STASH) [JULES and/or UKCA] Ocean model Output file tools Input file tools Local / Jasmin Data processing Prepare ancillary data Analysis and visualisation

  10. NCAS Rose/Cylc - Platforms • PUMA (cylc daemons, polling) • ARCHER MONSooN (MO managed) – Service nodes Lander – RDF Analytics cluster 76 Rose VM Rose/Cylc Cylc VM • JASMIN users HPC – jasmin-xfer1 PP – jasmin-cylc (MO) • JASMIN-Reading (running jules locally) 119 Rose/Cylc users • Polaris (on the way)

  11. NCAS Rose/Cylc - Suites  Many suites! • Initial proliferation (support) • Greater convergence Standard Suites (?) • GA7 – ACSIS/FEBBRAIO • GC3 – HighresMIP • GO5 Suite development in production runs • NEMOVAR Many moving parts/points of failure • UKESM • Nesting

  12. NCAS Rose/Cylc - Suites post processing postproc file conversion move from scratch to RDF remove from scratch checksum pptransfer pull files to jasmin checksum

  13. NEMO data assimilation suite

  15. NCAS Rose/Cylc - Training All but retired our UMUI-based training! • UMUI Conversion course (Sept 2016, Spring 2017) (http://cms.ncas.ac.uk/wiki/UmTraining/RoseSept2016) • 3-day UM Introduction (November and April) – Rose/Cylc based • 5-day UKCA Training (January 2016) – Rose/Cylc based (http://www.ukca.ac.uk/wiki/index.php/UKCA_Training_January_2016)

  16. NCAS Rose/Cylc – Management … Management • Installation/testing of new releases and upgrades Consistency across platforms • UM, GCOM, other installation/testing – Rose-stem Still learning about the capabilities of the system Support • Porting suites • Troubleshooting/debugging • Increased level infrastructure support eg users writing their own guis

  17. NCAS Rose/Cylc - Future • PUMA – central submission hub cloud-based PUMA (JASMIN VM)

  18. NCAS Rose/Cylc - Future • PUMA – central submission hub cloud-based PUMA (JASMIN VM) • UM in the cloud – experiments in AWS ongoing Rose/Cylc control?

  19. NCAS Rose/Cylc - Future • PUMA – central submission hub cloud-based PUMA (JASMIN VM) • UM in the cloud – experiments in AWS ongoing Rose/Cylc control? • MIP data workflow PP LOTUS MONSooN ARCHER VMs RDF -C work GWS MASS RDF ET

Recommend

Payment Processing North Carolina Accounting System (NCAS) Most State agencies use the North

Payment Processing North Carolina Accounting System (NCAS) Most State agencies use the North

Payment Processing North Carolina Accounting System (NCAS) Most State agencies use the North Carolina Accounting System (NCAS) for financial transaction processing. The entities that do not use NCAS are DOT, Employment Security Commission, the

68 views • 3 slides
Field measurements of urban roughness-sublayer turbulence Andreas Christen Department of

Field measurements of urban roughness-sublayer turbulence Andreas Christen Department of

Field measurements of urban roughness-sublayer turbulence Andreas Christen Department of Geography, Atmospheric Science Program NCAS Urban Meteorology Workshop University of British Columbia NCAS Urban Meteorology Workshop 30-31 March 2009,

1.1k views • 54 slides
Governance & SPOR TF ToR An agency of the European Union Revised Network Governance Model

Governance & SPOR TF ToR An agency of the European Union Revised Network Governance Model

Governance & SPOR TF ToR An agency of the European Union Revised Network Governance Model EMA MB HMA EU TMB EMA + All NCAs EMA + Some NCAs EU NDB IT Directors IT Business & IT Advisory - IT Advisory - Data

298 views • 4 slides
Scale and breadth of Cylc usage at the Met Office David Matthews, September 2016 Overview of

Scale and breadth of Cylc usage at the Met Office David Matthews, September 2016 Overview of

Scale and breadth of Cylc usage at the Met Office David Matthews, September 2016 Overview of Cylc usage at the Met Office Where ? (platforms) Who ? (number of uses) Why ? (types of usage) Some history Nov 2011 - Chose Cylc Nov 2012 - System

344 views • 19 slides
Where do you want to study next year? Experience life from a different point of view

Where do you want to study next year? Experience life from a different point of view

Where do you want to study next year? Experience life from a different point of view Broaden your personal and educational perspectives Live in a different geographic region Take courses not offered at home Become more

414 views • 27 slides
1 Key Item 1: Harmonisation Initial Marketing Authorisation Current situation with national

1 Key Item 1: Harmonisation Initial Marketing Authorisation Current situation with national

Introduction The new variation system Commission Regulations (EC) 1084 & 1085/2003 Considerable burden for industry and authorities Impact on NCAs Review project launched in 2006 by EC EC-Objectives: - clearer, simpler,

382 views • 6 slides
http://www.ncas.ac.uk Questions Do we have a simple picture of urban canopy turbulence?

http://www.ncas.ac.uk Questions Do we have a simple picture of urban canopy turbulence?

Probing urban canopy dynamics using direct numerical simulations (DNS) O. Coceal 1a , T.G. Thomas 2 & S.E. Belcher 1 1 Department of Meteorology, University of Reading, UK, 2 School of Engineering Sciences, University of Southampton, UK a Email:

167 views • 14 slides
Experience and Learning Experience and Learning A Psychological Point of View A Psychological

Experience and Learning Experience and Learning A Psychological Point of View A Psychological

Experience and Learning Experience and Learning A Psychological Point of View A Psychological Point of View Gerhard Weber Gerhard Weber Department of Psychology Department of Psychology University of Education Freiburg University of

654 views • 21 slides
Point of View in Writing for Children By Susan Williams Beckhorn When I think of the term, point

Point of View in Writing for Children By Susan Williams Beckhorn When I think of the term, point

Point of View in Writing for Children By Susan Williams Beckhorn When I think of the term, point of view (POV), I imagine a flashlights wedge shaped beam. Everything illuminated by the beam is within the point of view, all else is in the dark

97 views • 6 slides
CO COVID VID-19 19 Catherine O. Egbe, PhD (Specialist Scientist, Alcohol Tobacco and Other

CO COVID VID-19 19 Catherine O. Egbe, PhD (Specialist Scientist, Alcohol Tobacco and Other

THE S THE SCIENCE CIENCE ON SMOKI ON SMOKING AND NG AND CO COVID VID-19 19 Catherine O. Egbe, PhD (Specialist Scientist, Alcohol Tobacco and Other Drug Research Unit, SAMRC) NCAS webinar presentation: 22 April 2020 OUTLINE The human

608 views • 28 slides
Science highlights from the Cape Verde Observatory (CVAO) Lee, J.D. a , Read, K.A. b , Carpenter,

Science highlights from the Cape Verde Observatory (CVAO) Lee, J.D. a , Read, K.A. b , Carpenter,

Science highlights from the Cape Verde Observatory (CVAO) Lee, J.D. a , Read, K.A. b , Carpenter, L.J. a , Lewis, A.C. a , Moller, S.J. a , Mendes, L.M. c , Fleming Z. d , Evans, M. J. e a National Centre for Atmospheric Science (NCAS), University

427 views • 18 slides
Multiprogramming Single $PC Multiple $PCs (CPUs point of view) (process point of view) A

Multiprogramming Single $PC Multiple $PCs (CPUs point of view) (process point of view) A

9/10/2018 Multiprogramming Single $PC Multiple $PCs (CPUs point of view) (process point of view) A CSC 452: Scheduling B D A C B C B Dr. Jonathan Misurda D jmisurda@cs.arizona.edu http://www.u.arizona.edu/~jmisurda D C B A

138 views • 3 slides
A seven-year record (2006-2013) of nonmethane hydrocarbons (NMHCs) in the subtropical marine

A seven-year record (2006-2013) of nonmethane hydrocarbons (NMHCs) in the subtropical marine

A seven-year record (2006-2013) of nonmethane hydrocarbons (NMHCs) in the subtropical marine boundary layer at the Cape Verde Atmospheric Observatory (CVAO) S.Punjabi, K.A. Read, L.J. Carpenter, J.R. Hopkins, and A.C. Lewis - NCAS, University of

400 views • 15 slides
Micro-scale Modelling: examination of two different approaches. Sarah-Jane Lock, Alison Coals,

Micro-scale Modelling: examination of two different approaches. Sarah-Jane Lock, Alison Coals,

Micro-scale Modelling: examination of two different approaches. Sarah-Jane Lock, Alison Coals, Alan Gadian and Stephen Mobbs NCAS, University of Leeds The presentation will examine:- 1. some of the

418 views • 27 slides
Report to Task Force 10 March 2017 0 Outline of session SMS Project Business Cases

Report to Task Force 10 March 2017 0 Outline of session SMS Project Business Cases

Substance Subgroup Report to Task Force 10 March 2017 0 Outline of session SMS Project Business Cases Pilot Survey of NCAs Target Operating Model High Level Business Processes Migration Product-Subgroup

477 views • 45 slides
IDMP IWG SPOR benefits June 2016 1 Key Messages Industry is committed to SPOR and IDMP

IDMP IWG SPOR benefits June 2016 1 Key Messages Industry is committed to SPOR and IDMP

[Topic] IDMP IWG SPOR benefits June 2016 1 Key Messages Industry is committed to SPOR and IDMP Specific business cases bringing joint benefits to both Industry and NCAs will transform this commitment to action: 1. Optimising

499 views • 13 slides
EXPERIENCE PROTOTYPING Point of Views | How Might Wes | Experience Prototypes TEAM MEMBERS Po

EXPERIENCE PROTOTYPING Point of Views | How Might Wes | Experience Prototypes TEAM MEMBERS Po

EXPERIENCE PROTOTYPING Point of Views | How Might Wes | Experience Prototypes TEAM MEMBERS Po Tsui Bonnie Nortz Jenny Kim Grace Hong EDUCATING EMPATHY INITIAL POV We met Sandy, mother, soccer coach, and Intuit employee We were amazed

422 views • 16 slides
Outline: A. Current Capabilities of ePermits B. Applicants Point of View C. PPQ Permit

Outline: A. Current Capabilities of ePermits B. Applicants Point of View C. PPQ Permit

ePermits Update for PPQ State Plant Health Directors and Field Staff March 2006 1 Outline: A. Current Capabilities of ePermits B. Applicants Point of View C. PPQ Permit Staffs Point of View D. Future capabilities of ePermits 2 March

1.2k views • 118 slides
B What it does and how it does it... Takes a customer point of view - outside-in

B What it does and how it does it... Takes a customer point of view - outside-in

Systems Thinking in Streetscene B What it does and how it does it... Takes a customer point of view - outside-in Recognises that every time we / the system deviates from doing what matters to the customer it costs us capacity and/or

671 views • 27 slides
You never really understand a person until you consider things from his point of view . . .

You never really understand a person until you consider things from his point of view . . .

You never really understand a person until you consider things from his point of view . . . until you climb into his skin and walk around in it. _ Atticus Finch from To Kill A Mockingbird by Harper Lee Newark High School English

418 views • 5 slides
Long-term monitoring of VOCs in the free troposphere above the UK Shalini Punjabi a , James R.

Long-term monitoring of VOCs in the free troposphere above the UK Shalini Punjabi a , James R.

Long-term monitoring of VOCs in the free troposphere above the UK Shalini Punjabi a , James R. Hopkins a,b , Alastair Lewis a,b a University of York, York b NCAS, University of York, York sp575@york.ac.uk VOC monitoring : Aims To

418 views • 14 slides
Open Data & Data Management: the INFN experience Marcello Maggi INFN Senior Researcher Istituto

Open Data & Data Management: the INFN experience Marcello Maggi INFN Senior Researcher Istituto

Open Data & Data Management: the INFN experience Marcello Maggi INFN Senior Researcher Istituto Nazionale Fisica Nucleare Bari-Italy A Chaotic view from a scientist point of view The HEP Scientist The Standard Model Of

474 views • 31 slides
The telescopes from the The telescopes from the technological point of view: technological point

The telescopes from the The telescopes from the technological point of view: technological point

The telescopes from the The telescopes from the technological point of view: technological point of view: structures and mirrors for tructures and mirrors for IACTs IACTs Rodolfo Canestrari Rodolfo Canestrari INAF INAF-Astronomical

786 views • 43 slides
I total = k a I ambient + I i ( compute Phong lighting at the vertices h : halfway

I total = k a I ambient + I i ( compute Phong lighting at the vertices h : halfway

Clarify: OpenGL Example University of British Columbia News Clarify: Projective Rendering Pipeline CPSC 314 Computer Graphics coordinate system point of view! coordinate system point of view! midterm is Monday, be on time! object world

618 views • 4 slides

More recommend