Virtual Earthquake and seismology Research Community e-science environment in Europe Project 283543 – FP7-INFRASTRUCTURES-2011-2 – www.verce.eu – info@verce.eu DISPEL Tutorial Paul Martin VERCE Training Programme 3rd September 2012 www.verce.eu
Workflow enactment Compute Cluster User Machine DISPEL Tutorial 3rd September 2012 2 / 30
Workflow enactment ADMIRE Gateway VERCE Gateway / Portal Component Registry Data Store Enactment Engine Component Repository DISPEL Tutorial 3rd September 2012 2 / 30
Workflow enactment User Request Dispel Request DISPEL Tutorial 3rd September 2012 2 / 30
Workflow enactment Delegation / Registry Lookup DISPEL Tutorial 3rd September 2012 2 / 30
Workflow enactment Instruction DISPEL Tutorial 3rd September 2012 2 / 30
Workflow enactment Data Request / Submission / Signalling DISPEL Tutorial 3rd September 2012 2 / 30
Workflow enactment Result Notification DISPEL Tutorial 3rd September 2012 2 / 30
An example workflow DISPEL Tutorial 3rd September 2012 3 / 30
A simple script // Import PEs from the local registry. 1 use dispel.db.SQLQuery; 2 use dispel.lang.Results; 3 4 // Create instances of PEs. 5 SQLQuery query = new SQLQuery; 6 Results results = new Results; 7 8 // Construct workflow and feed in data. 9 |-"SELECT * FROM littleblackbook WHERE id < 10"-| => query.expression; 10 |-"uk.org.UoE.dbA"-| => query.resource; 11 |-"10 entries from the little black book"-| => results.name; 12 query.data => results.input; 13 14 // Submit the entire workflow. 15 submit; 16 DISPEL Tutorial 3rd September 2012 4 / 30
Demonstrating stream literals // Import PEs from the local registry. 1 use dispel.db.SQLQuery; 2 3 // Create instances of PEs. 4 SQLQuery query = new SQLQuery; 5 Results results = new Results; 6 7 // Define a set of queries. 8 String[] queries = new String[2]; 9 queries[0] = "SELECT name FROM littleblackbook WHERE id < 10"; 10 queries[1] = "SELECT name FROM littleblackbook WHERE id >= 10 AND id < 20"; 11 queries[2] = "SELECT address FROM littleblackbook WHERE name = ’David Hume’"; 12 13 // Construct workflow and feed in data. 14 |-queries[0], queries[1], queries[2]-| => query.expression; 15 |-"uk.org.UoE.dbA", "uk.org.UoE.dbB", "uk.org.UoE.dbC"-| => query.resource; 16 |-"Results from the little black book"-| => results.name; 17 query.data => results.input; 18 19 // Submit the entire workflow. 20 submit; 21 DISPEL Tutorial 3rd September 2012 5 / 30
Demonstrating stream literals // Import PEs from the local registry. 1 use dispel.db.SQLQuery; 2 3 // Create instances of PEs. 4 SQLQuery query = new SQLQuery; 5 Results results = new Results; 6 7 // Define a set of queries. 8 String[] queries = new String[2]; 9 queries[0] = "SELECT name FROM littleblackbook WHERE id < 10"; 10 queries[1] = "SELECT name FROM littleblackbook WHERE id >= 10 AND id < 20"; 11 queries[2] = "SELECT address FROM littleblackbook WHERE name = ’David Hume’"; 12 13 // Construct workflow and feed in data. 14 |-queries[0], queries[1], queries[2]-| => query.expression; 15 |-repeat 3 of "uk.org.UoE.dbA"-| => query.resource; 16 |-"Results from the little black book"-| => results.name; 17 query.data => results.input; 18 19 // Submit the entire workflow. 20 submit; 21 DISPEL Tutorial 3rd September 2012 6 / 30
A precocious script // Import PEs from the local registry. 1 use dispel.db.SQLQuery; 2 use dispel.tutorial.PrecociousChild; 3 4 // Create instances of PEs. 5 PrecociousChild child = new PrecociousChild; 6 SQLQuery query = new SQLQuery; 7 Results results = new Results; 8 9 // Construct workflow and feed in data. 10 child.output => query.expression; 11 |-repeat enough of "uk.org.UoE.dbA"-| => query.resource; 12 |-"Adult responses"-| => results.name; 13 query.data => results.input; 14 15 // Submit the entire workflow. 16 submit; 17 DISPEL Tutorial 3rd September 2012 7 / 30
PE Specification // Within the Dispel database package. 1 package dispel.db { 2 // Link to database ontology. 3 namespace db "http://www.dispel-lang.org/resource/db"; 4 // Type signature for the SQLQuery PE. 5 Type SQLQuery is PE( <Connection:String::"db:SQLQuery" terminator expression; Connection:String::"db:URI" locator resource> => 6 <Connection:[<rest>]::"db:TupleRowSet" data> ); 7 // Register the PE. 8 register SQLQuery; 9 } 10 11 // Within the package of generically useful PEs. 12 package dispel.core { 13 // A generic interpolation PE with few restrictions. 14 Type Interpolate is PE( Stype Element is Any; <Connection[]:Element inputs> => <Connection:Element output> ); 15 // This PE evenly interpolates in order of input connections. 16 Type OrderedInterpolate is Interpolate 17 with roundrobin inputs, @description = "Interpolates evenly by order of inputs."; 18 19 register Interpolate, OrderedInterpolate; 20 } 21 22 // Within the Dispel filter package. 23 package dispel.filter { 24 Type AbstractFilter is PE( Stype Element is Any; 25 <Connection:Element inputs> => <Connection:Element filtered; Connection:Element unfiltered> ) 26 with @description = "A filter splits input into filtered and unfiltered streams."; 27 28 Type HeadFilter is AbstractFilter 29 with filtered as head, unfiltered as tail, @description = "Diverts the head of a stream."; 30 31 Type ProgrammableIntegerFilter is PE( <Connection:Integer terminator input; Connection:String initiator expression; 32 33 Connection[]:Integer lockstep parameters> => 34 <Connection:Integer filtered; Connection:Integer unfiltered> ) with @description = "Splits the input integer stream into ’filtered’ and ’unfiltered’ streams in" + 35 "accordance with the expression and parameters given."; 36 37 register AbstractFilter, HeadFilter, ProgrammableIntegerFilter; 38 } 39 40 DISPEL Tutorial 3rd September 2012 8 / 30
Constructing new PEs package tutorial.example { 1 // Import existing PE from the registry. 2 use dispel.db.SQLQuery; 3 4 // Define new PE type. 5 Type SQLToTupleList is PE( <Connection expression> => <Connection data> ); 6 7 // Define new PE constructor. 8 PE<SQLToTupleList> lockSQLDataSource(String dataSource) { 9 SQLQuery query = new SQLQuery; 10 |-repeat enough of dataSource-| => query.resource; 11 return PE( <Connection expression = query.expression> => 12 <Connection data = query.data> ); 13 } 14 15 // Create new PEs. 16 PE<SQLToTupleList> TutorialQuery = lockSQLDataSource("uk.org.UoE.dbA"); 17 PE<SQLToTupleList> MirrorQuery = lockSQLDataSource("uk.org.UoE.dbB"); 18 19 // Register new entities. 20 register TutorialQuery, MirrorQuery; 21 } 22 DISPEL Tutorial 3rd September 2012 9 / 30
Using the new PE // Import PEs from the local registry. 1 use tutorial.example.TutorialQuery; 2 3 // Create instances of PEs (no import necessary). 4 TutorialQuery query = new TutorialQuery; 5 Results results = new Results; 6 7 // Connect PEI together to create workflow (no data source needed). 8 |-"SELECT * FROM littleblackbook WHERE id <= 10"-| => query.expression; 9 |-"10 entries from the little black book"-| => results.name; 10 query.data => results.input; 11 12 // Submit workflow. 13 submit results; 14 DISPEL Tutorial 3rd September 2012 10 / 30
Multicasting with composite PEs Query Generator MulticastQuery Results "uk.org.UoE.dbA" "uk.org.UoE.dbB" "uk.org.UoE.dbC" DISPEL Tutorial 3rd September 2012 11 / 30
Multicasting with composite PEs Query Generator SQLQuery List SQLQuery Results "uk.org.UoE.dbA" Intersect "uk.org.UoE.dbB" SQLQuery "uk.org.UoE.dbC" DISPEL Tutorial 3rd September 2012 12 / 30
A PE constructor featuring iteration package tutorial.example { 1 use dispel.db.SQLQuery; 2 use dispel.list.ListIntersect; 3 4 // A PE type which queries multiple sources. 5 Type MulticastQuery is PE( <Connection expression; Connection[] sources> => <Connection data> ); 6 7 // Use parallel SQLQuery instances to corroborate results. 8 PE<MulticastQuery> makeCorroboratedSQLQuery(Integer size) { 9 // Define aliases for workflow inputs in advance. 10 Connection expr; 11 Connection[] srcs = new Connection[size]; 12 // Create instances of internal PEs. 13 SQLQuery[] queries = new SQLQuery[size]; 14 ListIntersect intersect = new ListIntersect with inputs.length = size; 15 16 // Connect SQLQuery instances in parallel. 17 for (Integer i = 0; i < size; i++) { 18 queries[i] = new SQLQuery; 19 expr => queries[i].expression; 20 srcs[i] => queries[i].resource; 21 queries[i].data => intersect.inputs[i]; 22 } 23 24 // Return intersection of query results. 25 return PE( <Connection expression = expr; Connection[] sources = srcs> => 26 <Connection data = intersect.output> ); 27 } 28 29 register MulticastQuery, makeCorroboratedSQLQuery; 30 } 31 DISPEL Tutorial 3rd September 2012 13 / 30
Recommend
More recommend
