Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 - PowerPoint PPT Presentation

Oct 15, 2022 •28 likes •238 views

Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 , Upendra Kumar Devisetty 2 , Nirav Merchant 2 , and Douglas Thain 1 1 University of Notre Dame 2 University of Arizona MAKER : A Bioinformatics Pipeline 2 MAKER In Practice 3

Lessons Learned Moving MAKER from HPC to the Cloud Nick Hazekamp 1 , Upendra Kumar Devisetty 2 , Nirav Merchant 2 , and Douglas Thain 1 1 University of Notre Dame 2 University of Arizona
MAKER : A Bioinformatics Pipeline 2
MAKER In Practice 3
Jetstream Resources Available 4
MAKER Dependencies 5
VC3-Builder 6
VC3-Builder Provided Software Stack 7
MAKER in an HPC Environment 8
MPI Run on Single Node 9
Work Queue 10
WQ-MAKER 11
Work Queue Enabling Multi-instance Computation 12
Providing User Feedback - Debugging 13
Providing User Feedback - Debugging Examples Level Error MAKER Invalid input format Malformed file names containing “|” char IDs too long for BLAST Work Queue Worker times out and disconnects Files fail to transfer No workers connecting VC3-Builder Environment fails to build 14
Providing User Feedback - Runtime Sat Sep 23 15:24:25 2017 :: File Fungal annotated :: 177 Sat Sep 23 15:24:25 2017 :: Failed to annotate :: None ----------------------------------------------------------------- Type Success Failure Abandon Total Tasks 177 12 0 177 ----------------------------------------------------------------- Workers: Joined Removed Idled-Out Lost 10 12 2 2 0 ----------------------------------------------------------------- Work Queue Wall Time: 1d 18:02:15 Cumulative Task Wall Time: 5d 21:40:58 Cumulative Task Good Execute Time: 5d 8:33:32 Work Queue Send Time: 0d 0:11:18 Work Queue Receive Time: 0d 0:58:28 ----------------------------------------------------------------- 15
Providing User Feedback - Performance Task Behavior Master Time Usage Data Transfers 16
Results - Fungal Genome (41MB) 17
Results - Hummingbird Genome subset (900MB) 18
Results - Cactus Genome (1.6Gb) 19
Conclusions Use a tool capable of providing consistent environment on all target platform. Leverage multi-instance computation by coupling distributed communications and MPI execution. Design with feedback in mind to simplify debugging information, give runtime updates, and provide performance analysis. 20
Questions? VC3-Builder CCTools MAKER github.com/vc3-project/vc3-builder ccl.cse.nd.edu yandell-lab.org/software/maker.html Nicholas Hazekamp 21 Email : nhazekam@nd.edu

Recommend

Nico Uys Cloud Business Line Manager 1 Recent SAP on cloud projects Lessons learned

SAP in the cloud, do it right. Nico Uys Cloud Business Line Manager 1 Recent SAP on cloud projects Lessons learned Agenda Automation of the cloud Zag and our services Summary Q & A 2 Interesting Cloud Facts 1.

271 views • 23 slides

Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From

Protg 2006, July 26th, Stanford I R I S A C 1 R E C Saint-Cyr Lessons Learned Lessons Learned From From Lessons Learned Lessons Learned From From Ontology Ontology Design Design Ontology Ontology Design Design Jean Andr B

371 views • 12 slides

Lessons Learned Automating Cloud and Infrastructure Testing

T23 DevOps & The Cloud 2019-05-02 15:00 Lessons Learned Automating Cloud and Infrastructure Testing Presented by:

729 views • 40 slides

Lessons learned from deploying SUSE OpenStack Cloud and Enterprise Storage in the Public Cloud

Lessons learned from deploying SUSE OpenStack Cloud and Enterprise Storage in the Public Cloud TUT1224 Thursday, April 04, 03:15 PM - 04:15 PM | Belmont 1 Friday, April 05, 10:15 AM - 11:15 AM | Belmont 2 Mike Friesenegger Solution Architect

683 views • 35 slides

Security in a cloud context David Crooks, for the EGI CSIRT Lessons learned from recent incidents

Security in a cloud context David Crooks, for the EGI CSIRT Lessons learned from recent incidents www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142 Cloud Security EGI

571 views • 16 slides

C++, Java and .NET: Lessons learned from the Internet Age, and What it Means for the Cloud and

<Insert Picture Here> C++, Java and .NET: Lessons learned from the Internet Age, and What it Means for the Cloud and Emerging Languages Cameron Purdy Vice President, Development Overview <Insert Picture Here> A retrospective of

532 views • 40 slides

Moving Forward: Lessons Learned From The Last 10 Years of Risk Modeling Presentation Agenda 1)

Moving Forward: Lessons Learned From The Last 10 Years of Risk Modeling Presentation Agenda 1) Provide a framework for defining risk 2) Examine the role of theory and personal bias in the modeling process 3) Help outline possible partnerships

558 views • 27 slides

Cloud Computing Lessons Learned & Technical Perspective michal.furmankiewicz@chmurowisko.pl

Cloud Computing Lessons Learned & Technical Perspective michal.furmankiewicz@chmurowisko.pl https://www.linkedin.com/in/mifurm/ @mifurm Micha Furmankiewicz Cloud Architect for last 7 years CEO & co-owner of Chmurowisko / Cloud

834 views • 35 slides

Applying Architectural Patterns for the Cloud: Lessons Learned During Pattern Mining and

Applying Architectural Patterns for the Cloud: Lessons Learned During Pattern Mining and Application Ralph Retter (Daimler TSS GmbH) ralph.retter@daimler.com Christoph Fehling (University of Stuttgart, Germany) Problem We need to do Cloud

282 views • 25 slides

Lessons learned from six years of cloud technology transformation in government David Turner

Lessons learned from six years of cloud technology transformation in government David Turner Managing Director, MSC Digital MSC Digital Independent and vendor agnostic Formed specifically to assist the UK Public Sector - Its

386 views • 19 slides

Data Acquisition Methods: Lessons Learned from Deployment of Cloud HVAC Analytics at UCSB

Data Acquisition Methods: Lessons Learned from Deployment of Cloud HVAC Analytics at UCSB Tuesday, June 27 th 2017 Michael Georgescu (Ecorithm) Jordan Sager (UCSB) California Higher Education & Sustainability Conference (CHESC 2017)

452 views • 16 slides

The 'Cloud Area Padovana': lessons learned a3er two years of a produc:on OpenStack-based IaaS for

The 'Cloud Area Padovana': lessons learned a3er two years of a produc:on OpenStack-based IaaS for the local INFN user community Interna:onal Symposium on Grids and Clouds (ISGC) 2017 Academia Sinica, Taipei, Taiwan, 5-10 March 2017 Marco

295 views • 25 slides

Architecting for the cloud: lessons learned from 100 CloudStack deployments Sheng Liang CTO,

Architecting for the cloud: lessons learned from 100 CloudStack deployments Sheng Liang CTO, Cloud Platforms, Citrix CloudStack History 2008 2009 2010 2012 2011 Sept 2008: Nov 2009: May 2010: July 2011: April 2012: CloudStack

457 views • 34 slides

1 <Insert Picture Here> C++, Java and .NET: Lessons learned from the Internet Age, and

1 <Insert Picture Here> C++, Java and .NET: Lessons learned from the Internet Age, and What it Means for the Cloud and Emerging Languages Cameron Purdy Vice President, Development Overview <Insert Picture Here> A retrospective of

656 views • 41 slides

1. Building Description and Background 2. Fire 3. Inquest 4. Challenges 5. Lessons Learned 6.

917 Dundas Street W, Whitby 2016 OMFPOA Symposium Fire Chief Dave Speed CFPO Nicholas Webb 1. Building Description and Background 2. Fire 3. Inquest 4. Challenges 5. Lessons Learned 6. Moving Forward 1) Building Description and Background

468 views • 15 slides

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background

Institutionalizing Lessons Learned October 25, 2006 Loren Plisco Region II Background SEP 2002 Davis-Besse Lessons Learned Task Force AUG 2004 - Effectiveness Review Lessons Learned Task Force JAN 2005 - EDO Charters

269 views • 22 slides

Migrating HealthCare.gov to Terraform: Lessons Learned Christian Monaghan @monaghan_a_gram

Migrating HealthCare.gov to Terraform: Lessons Learned Christian Monaghan @monaghan_a_gram Cofounder, Nava PBC What is Terraform? A tool for building , changing , and versioning infrastructure Manage cloud providers Infrastructure as Code

958 views • 56 slides

How to run a successful hackathon? Lessons learned from 8 hackathon/bug smash events in China

How to run a successful hackathon? Lessons learned from 8 hackathon/bug smash events in China About us He Ruan Fred Li Jianfeng Ding Tencent Huawei Intel Chief Architect OpenStack Upstream OpenStack Engineering Tencent Cloud TStack Team

380 views • 17 slides

LESSONS FROM A TROUBLE-MAKER INTRODUCTION WHO IS THIS GUY? My name is Devin Director of

INNOVATING INSIDE LARGE ORGANIZATIONS LESSONS FROM A TROUBLE-MAKER INTRODUCTION WHO IS THIS GUY? My name is Devin Director of Data Science, Advance Auto Parts 20 years of experience in a variety of large organizations Grew into

751 views • 38 slides

OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned

OSHA Lessons Learned Adam Fries OSHA Compliance Officer February 13, 2018 OSHA Lessons Learned Jobsite Safety and Health Inspections/Audits better known as; PAPER WORK You hate it, Your safety director requires it, OSHA loves it, Does it

926 views • 35 slides

3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the

3/8/2019 Epidemiology, Risk Factors, and Outcomes of Pediatric PVD: LESSONS learned from the Spanish Registry: Lessons Learned from the Registries o Is It possible (and worthy) to do a national registry? Lessons learned from the o

1.16k views • 43 slides

DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD

DEBUGGING LESSONS LEARNED WHILE DEBUGGING LESSONS LEARNED WHILE FIXING NETBSD FIXING NETBSD ABOUT ME ABOUT ME maya@NetBSD.org coypu@sdf.org NetBSD/pkgsrc for the last 3 years THIS TALK THIS TALK Mix of a bunch of bugs Not solo work

695 views • 31 slides

Lessons Learned in Deploying OpenStack for HPC Users Graham T. Allan Edward Munsell Evan F.

Lessons Learned in Deploying OpenStack for HPC Users Graham T. Allan Edward Munsell Evan F. Bollig Minnesota Supercomputing Institute Stratus: A Private Cloud for HPC Users Project Goals Fill gaps in HPC service offerings HPC-like

580 views • 35 slides

Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda

Lessons Learned A Value Added Product of the Project Life Cycle R Gilman April 19, 2006 Agenda Introduction What are Lessons Learned? Value to the Project Process and Organization Keanes Approach to Lessons Learned Keys to

261 views • 24 slides