Introduction to CONNJUR Workflow Builder and Yes Workflow 2017 - - PowerPoint PPT Presentation

▶
introduction to connjur workflow builder and yes workflow
SMART_READER_LITE
LIVE PREVIEW

Introduction to CONNJUR Workflow Builder and Yes Workflow 2017 - - PowerPoint PPT Presentation

Mar 31, 2023 370 likes •559 views

Introduction to CONNJUR Workflow Builder and Yes Workflow 2017 Summer Workshop: June 29, 2017 Workflows (Wikipedia) u A workflow consists of an orchestrated and repeatable pattern of business activity enabled by the systematic organization of

slide-1
SLIDE 1

Introduction to CONNJUR Workflow Builder and Yes Workflow

2017 Summer Workshop: June 29, 2017

slide-2
SLIDE 2

Workflows (Wikipedia)

u A workflow consists of an orchestrated and repeatable

pattern of business activity enabled by the systematic

  • rganization of resources into processes that transform

materials, provide services, or process information. It can be depicted as a sequence of operations, declared as work of a person or group, an organization of staff, or

  • ne or more simple or complex mechanisms.

u From a more abstract or higher-level perspective,

workflow may be considered a view or representation of real work. The flow being described may refer to a document, service or product that is being transferred from one step to another.

slide-3
SLIDE 3

Workflows (Examples)

u On the first day Frank described an iterative

“workflow” by which a spectroscopist converts Varian/Bruker data into nmrPipe format, resolves ambiguities, performs preliminary processing, resolves phasing, reprocesses, iterate until done.

u Bertram Ludascher: ASAP

u Automate computation u Scalable u Adaptable for reuse u Provenance: capture processing history and data

lineage

slide-4
SLIDE 4

Kepler

Ludäscher, Bertram, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A Lee, Jing Tao, and Yang Zhao. 2006. “Scientific Workflow Management and the Kepler System.” Concurrency and Computation: Practice and Experience 18 (10): 1039–65.

slide-5
SLIDE 5

CONNJUR Workflow Builder

  • M. Fenwick, G. Weatherby, J. Vyas, C. Sesanker, T

. O. Martyn, H. J. Ellis, and M. R. Gryk, (2015) CONNJUR Workflow Builder: a software integration environment for spectral reconstruction. J Biomol NMR, 62, 313-26.

slide-6
SLIDE 6

Provenance Types

(Michael Wilde, Argonne Labs)

u Prospective Provenance: the specification of

the workflows procedure calls and data dependencies (acqu, workflow)

u Retrospective Provenance: the recordings of

when and where each procedure ran, and how each invocation behaved (acqus, reconstruction)

slide-7
SLIDE 7

Yes Workflow

u

Annotation system for Prospective Provenance from scripts

slide-8
SLIDE 8

Reproducibility

Replicability vs. reproducibility — or is it the

  • ther way around?

Reproducibility, replicability, reusability, repeatability

slide-9
SLIDE 9

Reproducibility (Dagstuhl Working Group)

PRIMAD

u

Platform – Portability vs. reproducibility (OS or hardware platforms)

u

Research Objective (goal of computation)

u

Implementation (Fast Fourier Transform)

u

Method (Fourier Transform)

u

Actors (Dagstuhl group defines agent as human)

u

Data (data used in study)

Rauber, A., Braganholo, V., Dittrich, J., et al. (2016). PRIMAD – Information gained by different types of reproducibility. In Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041). Friere, J., Fuhr, N., & Rauber, A. editors. Gryk, M. & Ludäscher, B. (2017). Workflows and provenance: Towards information science solutions for the natural sciences. Library Trends, in press.

slide-10
SLIDE 10

Metadata Definition

u

Definition 1: Data about Data

slide-11
SLIDE 11

Workflow -> Provenance -> Reproducibility

All rely on the capture of metadata

u

Definition 1: Data about Data

slide-12
SLIDE 12

Workflow -> Provenance -> Reproducibility

All rely on the capture of metadata

u

Definition 1: Data about Data

u

Definition 2: Metadata As Surrogate

slide-13
SLIDE 13

Workflow -> Provenance -> Reproducibility

All rely on the capture of metadata

u

Definition 1: Data about Data

u

Definition 2: Metadata As Surrogate

slide-14
SLIDE 14

Yes Workflow

#@BEGIN main #@IN raw_turkey @URI store:shnucks_turkey #@OUT cooked_turkey @URI plate:delicious_turkey #@BEGIN survey_guests #@OUT food_allergies #@END survey_guests #@BEGIN brining #@IN raw_turkey @URI store:shnucks_turkey #@PARAM seasonings #@OUT brined_turkey #@END brining #@BEGIN weighing #@IN brined_turkey #@OUT weighed_turkey #@OUT weight #@END weighing #@BEGIN stuffing #@IN weighed_turkey #@IN stuffing_ingredients #@IN food_allergies #@OUT stuffed_turkey #@END stuffing #@BEGIN baking #@PARAM weight #@PARAM temperature #@PARAM duration #@IN stuffed_turkey #@OUT cooked_turkey @URI plate:delicious_turkey #@END baking #@END main

slide-15
SLIDE 15

Yes Workflow

echo 'Converting from Varian to NMRPipe format' var2pipe -in ./fid \

  • xN

1024

  • yN

128

  • zN

64 \

  • xT

512

  • yT

64

  • zT

32 \

  • xMODE

Complex

  • yMODE

States-TPPI

  • zMODE

Rance-Kay \

  • xSW

12000

  • ySW

6000

  • zSW

2000 \

  • xOBS 599.5694 -yOBS

125.768

  • zOBS 60.7438 \
  • xCAR 4.772
  • yCAR

45

  • zCAR 119 \
  • xLAB 1H
  • yLAB

13C

  • zLAB 15N \
  • ndim 3
  • aq2D

States \

  • out ./data/hnco%03d.pipe -verb -ov

sleep 1 echo 'Transforming 3-dimensional NMR data!' echo 'Processing F3/F1 dimensions first' xyz2pipe -in data/hnco%03d.pipe -x -verb \ | nmrPipe -fn SOL -mode 1 -fl 16 -fs 1 -poly \ | nmrPipe -fn CBF -last 12 \ | nmrPipe -fn SP -off 0.39 -end 0.98 -pow 2 -size 512 -c 0.5 \ | nmrPipe -fn ZF -size 2048 \ | nmrPipe -fn FT -verb \ | nmrPipe -fn PS -p0 0 -p1 0 -di \ | nmrPipe -fn EXT -left -sw -verb \ | nmrPipe -fn TP \ | nmrPipe -fn SP -off 0.39 -end 0.98 -pow 2 -size 64 -c 0.5 \ | nmrPipe -fn ZF -size 256 \ | nmrPipe -fn FT -verb \ | nmrPipe -fn PS -p0 0 -p1 0 -di \ | pipe2xyz -out ft/hnco%03d.ft2 –y xyz2pipe -in ft/hnco%03d.ft2 -z -verb \ | nmrPipe -fn SP -off 0.39 -end 0.98 -pow 2 -size 32 -c 0.5 \ | nmrPipe -fn ZF -size 128 \ | nmrPipe -fn FT -neg -verb \ | nmrPipe -fn PS -p0 0 -p1 0 -di \ | pipe2xyz -out ft/hnco%03d.ft3 -z

Website

slide-16
SLIDE 16

16

How much of the above information is NMR related? How much is related to nmrPipe or the computer we are using?

#! /bin/csh # My Processing Script nmrPipe -in mydata.pipe \ | nmrPipe -fn SOL -mode 1 -fl 16 -fs 1 -poly \ | nmrPipe -fn CBF -last 12 \ | nmrPipe -fn GMB -lb -7 -gb 0.1 -size 512 -c 0.5 \ | nmrPipe -fn ZF -size 2048 \ | nmrPipe -fn FT -verb \ | nmrPipe -fn PS -p0 68.6 -p1 -34.8 -di \ | nmrPipe -fn EXT -left -sw -verb \ | nmrPipe -fn TP \ | nmrPipe -fn LP -fb -ord 30 -x1 2 -xn 128 -pred 64 -fix -fixMode 1 -after \ | nmrPipe -fn SP -off 0.39 -end 0.98 -pow 2 -size 192 -c 0.5 \ | nmrPipe -fn ZF -size 256 \ | nmrPipe -fn FT -verb \ | nmrPipe -fn PS -p0 -9.0 -p1 20.0 -di \

  • out mydata.ft2 -ov

bash vs. csh – the shell wars # = comment which nmrPipe tool? nmrPipe, var2pipe, xyz2pipe, etc. watch for trailing spaces! Syntax? filenames and filesystems

slide-17
SLIDE 17

17

What procedures do we use to massage our data? What procedures do we use to transform our data?

#! /bin/csh # My Processing Script nmrPipe -in mydata.pipe \ | nmrPipe -fn SOL -mode 1 -fl 16 -fs 1 -poly \ | nmrPipe -fn CBF -last 12 \ | nmrPipe -fn GMB -lb -7 -gb 0.1 -size 512 -c 0.5 \ | nmrPipe -fn ZF -size 2048 \ | nmrPipe -fn FT -verb \ | nmrPipe -fn PS -p0 68.6 -p1 -34.8 -di \ | nmrPipe -fn EXT -left -sw -verb \ | nmrPipe -fn TP \ | nmrPipe -fn LP -fb -ord 30 -x1 2 -xn 128 -pred 64 -fix -fixMode 1 -after \ | nmrPipe -fn SP -off 0.39 -end 0.98 -pow 2 -size 192 -c 0.5 \ | nmrPipe -fn ZF -size 256 \ | nmrPipe -fn FT -verb \ | nmrPipe -fn PS -p0 -9.0 -p1 20.0 -di \

  • out mydata.ft2 -ov

Functionality and Function Order!