makeflow
play

Makeflow Work Local Condor Torque Queue W W Makefile - PowerPoint PPT Presentation

May 27, 2023 •120 likes •627 views

Makeflow Work Local Condor Torque Queue W W Makefile FutureGrid Private Torque W Cluster Cluster W W W Thousands of Workers in a Makeflow Personal Cloud W W W Campus Public Condor Cloud Pool

  3. • • • • • Makeflow Work Local Condor Torque Queue

  4. W W Makefile FutureGrid Private Torque W Cluster Cluster W W W Thousands of Workers in a Makeflow Personal Cloud W W W Campus Public Condor Cloud Pool Provider W W W

  6. • – – – • – – – –

  7. Work Queue Application submit wait C Python Perl Work Queue Library $$$ Master Process Sandbox for Each Task

  11. #include “work_queue.h” struct work_queue *queue; struct work_queue_task *task; queue = work_queue_create( 0 ); work_queue_specify_name( “myproject” ); task = work_queue_task_create(“sim.exe –p 50 in.dat >out.txt”); /// Missing: Specify files needed by the task. work_queue_submit( queue, task ); while(!work_queue_empty(queue)) { task = work_queue_wait( queue, 60 ); if(task) work_queue_task_delete( task ); }

  12. use Work_Queue; $queue = Work_Queue->new( 0 ); $queue->specify_name( “myproject” ); $task = Work_Queue::Task->new(“sim.exe –p 50 in.dat >out.txt”); ### Missing: Specify files needed by the task. $queue->submit( $task ); while(!$queue->empty()) { $task = $queue->wait( 60 ); ### Missing: Do something with the task's results }

  13. from work_queue import * queue = WorkQueue( port = 0 ) queue.specify_name( “myproject” ); task = Task(“sim.exe –p 50 in.dat >out.txt”) ### Missing: Specify files needed by the task. queue.submit( task ) While not queue.empty(): task = queue.wait(60)

  14. calib.dat sim. out.txt exe in.dat work_queue_task_specify_file( task,“in.dat”,”in.dat”, WORK_QUEUE_INPUT, WORK_QUEUE_NOCACHE ); work_queue_task_specify_file( task,“calib.dat”,”calib.dat”, WORK_QUEUE_INPUT, WORK_QUEUE_CACHE ); work_queue_task_specify_file( task,“out.txt”,”out.txt”, WORK_QUEUE_OUTPUT, WORK_QUEUE_NOCACHE ); work_queue_task_specify_file( task,“sim.exe”,”sim.exe”, WORK_QUEUE_INPUT, WORK_QUEUE_CACHE );

  15. calib.dat sim. out.txt exe in.dat $task->specify_input_file(“in.dat”); $task->specify_input_file(“calib.dat”); $task->specify_output_file(“out.txt”); $task->specify_input_file(local_name => “sim.exe”, remote_name => "sim.exe", flags => $Work_Queue::WORK_QUEUE_CACHE );

  16. calib.dat sim. out.txt exe in.dat task.specify_file( “in.dat”, ”in.dat”, WORK_QUEUE_INPUT, cache = False ) task.specify_input_file( “calib.dat” ) task.specify_output_file( “out.txt” ) task.specify_input_file( “sim.exe”, cache = True )

  18. Running a Work Queue Program …

  19. … for Python …

  20. … for Perl …

  23. • • • • •

  25. Work Queue Worker (port 9037) Catalog

  26. Specify Project Names in Work Queue

  27. Start Workers with Project Names

  29. Tag a Work Queue task • – –

  30. Cancel Work Queue task •

  31. Retry “slow” Work Queue tasks • – – • – – – –

  32. Activating fast abort in Work Queue #abort if task exceeds 2.5 * avg execution time

  33. Send intermediate buffer data as input file for Work Queue Task • •

  34. Send intermediate buffer data as input file for Work Queue Task

  35. Work Queue Task Structure • • • • • • • • •

  36. Accessing Work Queue Task structure

  37. Work Queue Statistics • • • – – – – • – • – • …

  38. Accessing Work Queue Statistics •

  39. Work Queue Statistics •

  40. Thinking Even BIGGER

  41. Managing Your Workforce W WQ 200 Pool W W W W WQ 100 Pool W

  42. Multi-Slot Workers 1 core 1 core 1 core 1 core 1 core task task task task task 4 cores specify_cores(4); 512 MB specify_memory(512); work_queue_worker --cores 8 work_queue_worker --memory 1024 (implies 1 task, 1 core)

  43. Using Foremen T T T T T T T T T T T T work_queue_worker --foreman $MASTER $PORT $$$ $$$ Approx X1000 at each fanout. California Chicago

  45. • • Replica • Exchange • Work Queue

  46. Sequence Data SAND SAND Celera filter align Consensu master master s W W W W W W W

More recommend