Jug Tutorial: Coarse-Level Parallel Programming in Python Luis - PowerPoint PPT Presentation

Jug Tutorial: Coarse-Level Parallel Programming in Python Luis Pedro Coelho luis@luispedro.org May 15, 2013 Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (1 / 24) ⋆ ⋆

Problem Brute force a password. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (2 / 24) ⋆ ⋆

Assumptions The crypt module exists with elements: decrypt: decrtypt ciphertext given a password. isgood: test whether this text could be the plaintext. letters: just the letters. Also, we know that the password is five letters. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (3 / 24) ⋆ ⋆

Simple Solution import itertools from crypt import decode, letters, isgood, preprocess ciphertext = file('secret.msg').read() ciphertext = preprocess(ciphertext) for p in itertools.product(letters, repeat=5): text = decode(ciphertext, p) if isgood(text): passwd = "".join(map(chr,p)) print '%s:%s' % (passwd, text) Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (4 / 24) ⋆ ⋆

This solution does not take advantage of multiple processors. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (5 / 24) ⋆ ⋆

Tasks import itertools from crypt import decode, letters, isgood, preprocess ciphertext = file('secret.msg').read() ciphertext = preprocess(ciphertext) def decrypt(ciphertext, p): text = decode(ciphertext, p) if isgood(text): passwd = "".join(map(chr,p)) return (passwd, text) # else: return None results = [] for p in itertools.product(letters, repeat=5): results.append(Task(decrypt,ciphertext,p)) Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (6 / 24) ⋆ ⋆

Python Magic from jug import TaskGenerator import itertools from crypt import decode, letters, isgood, preprocess ciphertext = file('secret.msg').read() ciphertext = preprocess(ciphertext) @TaskGenerator def decrtypt(ciphertext, p): text = decode(ciphertext, p) if isgood(text): passwd = "".join(map(chr,p)) return (passwd, text) # else: return None results = [] for p in itertools.product(letters, repeat=5): results.append(decrypt(ciphertext,p)) Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (7 / 24) ⋆ ⋆

Enter Jug You give it the Jugfile, it runs the tasks for you! Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (8 / 24) ⋆ ⋆

Jug Loop while len(tasks) > 0: ready = [t for t in tasks if can_run(t)] for t in ready: if not is_running(t): t.run() tasks.remove(t) Except jug is much fancier! Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (9 / 24) ⋆ ⋆

Jug Advantages Automatic task-level parallelization with dependency tracking. 1 Remember all intermediate results. 2 Makes writing parallel code look like writing sequential code. 3 Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (10 / 24) ⋆ ⋆

This example is actually not so good. We have 26 5 ≈ 11 M tasks, all of which run very fast. As a rule of thumb, your tasks should take at least a couple of seconds. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (11 / 24) ⋆ ⋆

Solution each task will be: Given a letter, try all passwords beginning with that letter. Now, we have 26 tasks. Much better. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (12 / 24) ⋆ ⋆

@TaskGenerator def decrypt(prefix): res = [] for suffix in product(letters, repeat=5-len(prefix)): passwd = np.concatenate([prefix, suffix]) text = decode(ciphertext, passwd) if isgood(text): passwd = "".join(map(chr, passwd)) res.append( (passwd, text) ) return res @TaskGenerator def join(partials): return list(chain(*partials)) results = join([decrypt([p]) for p in letters]) Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (13 / 24) ⋆ ⋆

Let’s call jug now. (Assuming our code is in a file called jugfile.py) Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (14 / 24) ⋆ ⋆

$jug status Task name Waiting Ready Finished Running ----------------------------------------------- jugfile.join 1 0 0 0 jugfile.decrypt 0 26 0 0 ............................................... Total: 1 26 0 0 Some tasks are ready. None are running. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (15 / 24) ⋆ ⋆

$jug execute & [1] 29501 $jug execute & [2] 29502 Executing in the background. . . Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (16 / 24) ⋆ ⋆

$jug status Task name Waiting Ready Finished Running ----------------------------------------------- jugfile.join 1 0 0 0 jugfile.decrypt 0 24 0 2 ............................................... Total: 1 24 0 2 Two tasks running. Good. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (17 / 24) ⋆ ⋆

Wait a few minutes. . . Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (18 / 24) ⋆ ⋆

$jug status Task name Waiting Ready Finished Running ----------------------------------------------- jugfile.join 1 0 0 0 jugfile.decrypt 0 14 10 2 ............................................... Total: 1 14 10 2 Ten tasks have finished. Notice how the join task must wait for all the others. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (19 / 24) ⋆ ⋆

A few more minutes. . . Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (20 / 24) ⋆ ⋆

$jug status Task name Waiting Ready Finished Running ----------------------------------------------- jugfile.join 0 0 1 0 jugfile.decrypt 0 0 26 0 ............................................... Total: 0 0 27 0 Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (21 / 24) ⋆ ⋆

What Now? All tasks are finished! How do I get to the results? Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (22 / 24) ⋆ ⋆

import jug jug.init('jugfile', 'jugdata') import jugfile results = jug.task.value(jugfile.results) for p,t in results: print "%s\n\n Password was '%s'" % (t,p) jug.init is necessary to initialise the jug backend. Then, the jugfile can be directly imported as a Python module. Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (23 / 24) ⋆ ⋆

For more information http://luispedro.org/software/jug http://github.com/luispedro/jug Luis Pedro Coelho (luis@luispedro.org) Jug Tutorial 0 May 15, 2013 (24 / 24) ⋆ ⋆