hyperloom
play

HyperLoom Stanislav Bhm, Vojt ch Cima https://hyperloom.eu - PowerPoint PPT Presentation

Jan 13, 2023 •218 likes •723 views

HyperLoom Stanislav Bhm, Vojt ch Cima https://hyperloom.eu It4Innovations / ADAS / DEF 10:00-11:30 Intoduction to HyperLoom Hands-on: 'Hello world' 12:30-14:00 Hands-on: Building a real pipeline Best practices

  1. HyperLoom Stanislav Böhm, Vojt ě ch Cima https://hyperloom.eu It4Innovations / ADAS / DEF

  2. 10:00-11:30 • Intoduction to HyperLoom • Hands-on: 'Hello world' 12:30-14:00 • Hands-on: Building a real pipeline • Best practices 14:30-16:00 • Bring your own problems

  3. E CAPE

  4. Chemogenomics gene GSK3B ZZRDAWZXJBVQSO-UYBDAZJANA-N Inactive + = ZZRDINJRCNXHIQ-MCRMGHHCNA-N Inactive ZZRDSHKJUCIYQL-GPQMBLKYNA-N Active ZZRDKBQKAVQCMK-LILDFLRNNA-N Inactive

  5. GSK3B + ZZRDAWZXJBVQSO = inactive GSK3B + ZZRDINJRCNXHIQ = inactive ... GSK3B + ZZRDSHKJUCIYQL = active GSK3B + ZZRDKBQKAVQCMK = inactive labeled dataset

  6. modelling ... M (machine learning) training data (~80%) model

  7. modelling ... M (machine learning) training data (~80%) model validation V validation data (~20%) model accuracy

  8. M ... M TF2 TF1 training data (~80%) training data (~80%) V V VF2 VF1 validation data (~20%) validation data (~20%)

  9. M M M M M V V V V V average accuracies cross-validated accuracy

  10. modelling parameterization P M M M M M V V V V V cross-validated accuracy (for given parametrization)

  11. P1 P2 P3

  12. P1 P2 P3 pick best model best model

  13. best model (but only for a single target!)

  15. HPC Schedulers PBS SLURM Duration of tasks

  16. HPC Schedulers PBS SLURM Work fl ow systems Luigi Air fl ow DAGMAN Duration of tasks

  17. HPC Schedulers PBS SLURM Work fl ow systems Luigi Air fl ow DAGMAN Task-based programming OmpSs StarPU Duration of tasks

  18. HPC Schedulers PBS SLURM Work fl ow systems Luigi Air fl ow DAGMAN Task-based programming OmpSs StarPU Duration of tasks

  19. HPC Schedulers PBS Map Reduce SLURM Spark Hadoop Work fl ow systems Luigi Air fl ow DAGMAN Task-based programming OmpSs StarPU Duration of tasks

  20. Berkeley Dask/ Distributed Ray HPC Schedulers Rain pycompss PBS Map Reduce SLURM Spark Hadoop Work fl ow systems Luigi Air fl ow DAGMAN Task-based programming OmpSs StarPU Duration of tasks

  21. HyperLoom https://code.it4i.cz/ADAS/loom

  22. Architecture worker pipeline client server worker worker

  23. Architecture worker client server worker server schedules tasks worker for execution on workers

  24. Architecture worker client server worker worker workers process tasks

  25. Architecture server returns results to client worker client server worker results worker

  26. Deployment worker pipeline single PBS client server worker job results worker

  27. HyperLoom Checklist Inputs Task Intra-node tasks Output

  28. HyperLoom Checklist Inputs Task Intra-node tasks Output Python code & .py Extern programs ./binary

  29. HyperLoom Checklist Inputs Task Intra-node tasks Output Python code & .py Extern programs ./binary 100k+ Many tasks

  30. HyperLoom Checklist Inputs Task Intra-node tasks Output Python code & .py Extern programs ./binary 100k+ Many tasks Inter-depenedant tasks

  31. HyperLoom Checklist Inputs Task Intra-node tasks Output Python code & .py Extern programs ./binary 100k+ Many tasks Inter-depenedant tasks Heterogenous tasks

  32. Example 1 import loom.client as lc # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

  33. Example 1 import loom.client as lc const const # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) merge # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

  34. Example 1 import loom.client as lc const const # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) merge # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

  35. Example 1 import loom.client as lc const const # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) merge # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

  36. Example 1 import loom.client as lc const const # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) merge # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

  37. Example 2 import loom.client as lc # Pipeline Definition task1 = lc.tasks.run("/bin/hostname") # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather())

  38. Example 3A import loom.client as lc # Pipeline Definition tasks = [lc.tasks.run("/bin/hostname") for i in range(100)] task1 = lc.tasks.merge(tasks) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather())

  39. Example 3A import loom.client as lc # Pipeline Definition tasks = [lc.tasks.run("/bin/hostname") for i in range(100)] task1 = lc.tasks.merge(tasks) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather()) Output: r37u12n989 r36u11n321 r36u11n320 r37u12n989 r37u12n989

  40. Example 3B import loom.client as lc # Pipeline Definition tasks = [lc.tasks.run("/bin/hostname > output", shell=True, outputs=["output"]) for i in range(100)] task1 = lc.tasks.merge(tasks) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather())

  41. Mapping inputs/outputs task_load_data = ... tasks = lc.tasks.run("./my-train --train-data=train.dat --output=model.dat", inputs=[(task_load_data, "train.dat")], outputs=["model.dat"])

  42. Example 4 import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

  43. Example 4 import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

  44. Example 4 import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

  45. Example 4 import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

  46. Example 5 task = ... task.resource_request = lc.tasks.cpus(24) task.checkpoint_path = "/scratch/myproject/task-1"

  47. Progress Lore Loom reporting Scheduling info Node utilization

  48. Hands-on user@loginX.salomon$ cp -r /scratch/temp/hyperloom-tutorial/examples/ex1 . user@loginX.salomon$ cd ex1 user@loginX.salomon$ qsub ex1.pbs

  49. Hands-on user@loginX.salomon$ cp -r /scratch/temp/hyperloom-tutorial/examples/ex1 . user@loginX.salomon$ cd ex1 user@loginX.salomon$ qsub ex1.pbs user@loginX.salomon$ ml load loom user@loginX.salomon$ python3 -m pip install bokeh==0.12.6 --user user@loginX.salomon$ python3 -m loom.lore lore_logs user@loginX.salomon$ firefox output.html # Or download output.html and open it locally

  50. Snailwatch

Recommend

More recommend