Automatic Job Submission Simon Albright Three classes: - PowerPoint PPT Presentation

Automatic Job Submission Simon Albright

Three classes: ● SSHConnection ● Handles sending and receiving data, monitoring job status etc ● Remote_Optimise ● Handles itteration over variables, manipulation of .py scripts etc ● Itterable_Variable ● Contains variable to be edited as a single value or function and sets range of values for each itteration

SSHConnection Build on Paramiko, python module for handling SSH (text interface) and SFTP (file transfer) connections ● ssh = sp.SSHConnection('salbrigh', verb=1, workingDir = remoteDir, localDir = localDir) ● Username = 'salbrigh' ● Verbosity = 1 (Higher levels increase output information, can be set to 0 for *almost* no output) ● LocalDir & WorkingDir sets where all commands and files will be sent by default, can be changed on a line by line basis ● Server, port, response to mission keys have default values but can be changed ● Password handled by “getpass”, prevents text appearing on screen when typing and stores password in RAM.

Password method def set_password(self): if self.verbosity > 0: print("Password for LXPLUS required") print("Password is not stored outside RAM") print("Password will not appear in history") print("Password should only come from interactive source") self.pWord = getpass.getpass()

Connection controls start_session ● Creates initial SSH and SFTP connections ● If password is incorrect gives three attempts to try again before ending program end_session ● Terminates SFTP and SSH connection start_stop_connect ● Closes and then reopens SFTP and SSH connection ● Prevents session from hanging

Job submission Small numbers of jobs ( e.g. <~ 100): ● Pass a single string as bash command with format: 'bsub -q queueName -J jobName1 < submissionScript1 ; bsub -q queueName -J jobName2 < submissionScript2 ; … Medium numbers of jobs (100 < N < 1000): ● Write a script with format: bsub -q queueName -J jobName1 < submissionScript1 bsub -q queueName -J jobName2 < submissionScript2 ... ... … Very large numbers (N > 1000): ● As with Medium numbers but multiple scripts JobID stored in list for identification of running/queueing and completed jobs

(Some) additional commands: ● check_jobs ● Monitoring jobs still in queue ● compress_files ● Compression of data for retrieval ● get_file ● Transfer from server ● put_file ● Transfer to server ● make_file ● Make text file from string (e.g. for submissionScript writing) ● read_file ● Return file contents as string ● Copy ● Copy files on server ● Move ● Move files on server ● Remove ● Delete file on server ● search_replace ● Search for string in file and replace