Building the Easy Button: Automating SAS Program Batch Runs Nancy Brucken inVentiv Health June 28, 2017
Agenda • Motivation • Driver programs • Windows batch files • Building a batch file • Future considerations • Summary 2
Motivation 3
Running a SAS Program 4
Running Multiple SAS Programs Program 1 Program 2 Program 3 5
Running Many SAS Programs • One person − Open program 1 and run − Open program 2 and run − And so on…. − Takes forever! • Multiple people − Each person runs multiple programs − Distributes the work − What if someone isn’t available? − What about dependencies? 6
Problem Scope • Delivery for “medium-sized” clinical study − 30 SDTM domains − 25 analysis datasets − 60 TLF programs 7
Dependencies • One program relies on the output of another program • Example- pharmaceutical industry Analysis Dataset Model (ADaM) ADSL Other Run ADSL, analysis then run variables other dataset 1 record/subject programs Demographics Treatment arm Partial Population flags ADSL Covariates 8
How do I automate? 9
Where’s my Easy button? 10
Driver Programs 11
Driver Program • SAS program that runs other SAS programs %include ‘C:\mysas\adsl.sas’; %include ‘C:\mysas\adae.sas’; %include ‘C:\mysas\adlb.sas’; . . . . . %include ‘C:\mysas\advs.sas’; 12
Creating a Driver Program • Manually type in all of the programs to run Or • Get the computer to do it for you! 13
Program to Build Driver Program filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; infile myfiles truncover; input file_name $32.; file mybatch; put '%include ' file_name ';'; run; 14
Program to Build Driver Program filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; Pipe routes output of infile myfiles truncover; operating system command to SAS input file_name $32.; dataset file mybatch; put '%include ' file_name ';'; run; 15
Program to Build Driver Program filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; /b limits output to just infile myfiles truncover; file names input file_name $32.; file mybatch; put '%include ' file_name ';'; run; 16
Program to Build Driver Program filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; Read in list of file infile myfiles truncover; names input file_name $32.; file mybatch; put '%include ' file_name ';'; run; 17
Program to Build Driver Program filename myfiles pipe 'dir C:\myfiles\*.sas /b'; filename mybatch ‘C:\mysas\mybatch.sas'; data _null_; infile myfiles truncover; input file_name $32.; Build %INCLUDE file mybatch; statements put '%include ' file_name ';'; run; 18
Resulting Program %include adsl.sas ; %include adae.sas ; %include adlb.sas ; %include admh.sas ; %include advs.sas ; 19
Advantages of Driver Programs • Easy to create • Easy to control dependencies − Specify run order • If one program errors out, the entire job stops 20
Disadvantages of Driver Programs • Programs run sequentially, so do not take advantage of multi- server SAS Grid environments • Large batch jobs may take a long time to run − Can you disconnect your PC and still leave the job running? • Datasets, macro variables and options created in one program will carry through into subsequent programs if not explicitly deleted or cleared • If one program errors out, the entire job stops 21
Batch Files 22
Batch Files • Files containing operating system commands • Supported by many operating systems − Unix − MVS − Windows • Operating system-specific 23
Typical Batch File Structure Initialization Commands 24
Initialization Section • Specify folder paths • Direct LOG and LST files • Specify location of SAS config file • Specify location of SAS executable Place at the top of the batch file so they are only referenced once! 25
Sample Initialization Section @echo off set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“ -CONFIG "\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg" 26
Sample Initialization Section set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“ -CONFIG "\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg" Path to program location 27
Sample Initialization Section set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“ -CONFIG "\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg" Path to log file location 28
Sample Initialization Section set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“ -CONFIG "\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg" Path to LST file location 29
Sample Initialization Section set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas=" C:\Program Files\SAS Institute\SAS\V9\sas.exe “ -CONFIG "\\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\sasv9 2_i3spctx.cfg" Location of SAS executable 30
Sample Initialization Section set pth=-sysin H:\data\sas\Client\Project\PGM\Analysis\ set plog=-log H:\data\sas\Client\Project\LOG\Analysis\ set plst=-print H:\data\sas\Client\Project\LOG\Analysis\ set sas="C:\Program Files\SAS Institute\SAS\V9\sas.exe“ -CONFIG " \\sasserver.company.com\apps_vol\Apps\SAS\Config\Citrix\s asv92_i3spctx.cfg " Location of SAS config file 31
Command Section • Commands to run each program • Includes environment variables from initialization section 32
Sample Command Section @echo on %sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst% 33
Sample Command Section %sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst% Command to run SAS, including location of config file 34
Sample Command Section %sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth% CM.SAS %plog% %plst% Path to program location 35
Sample Command Section %sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth% CM.SAS %plog% %plst% Program name 36
Sample Command Section %sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst% Log file location 37
Sample Command Section %sas% %pth%DM.SAS %plog% %plst% %sas% %pth%SV.SAS %plog% %plst% %sas% %pth%AE.SAS %plog% %plst% %sas% %pth%CM.SAS %plog% %plst% LST file location 38
Advantages of Batch Files • Each program runs independently • Datasets, options and macro variables set in one program do not carry over into subsequent programs • If one program errors out, the remaining programs will still run • Under many environments, you can start/schedule a batch job and log off without having to wait until it finishes • Easy to control dependencies 39
Disadvantages of Batch Files • Run sequentially − Do not take advantages of multi-core Grid servers • If one program errors out, the remaining programs will still run • May be more difficult to set up than driver programs − Need to identify locations of SAS executable and config files • Command syntax varies across operating systems 40
SAS Grid Batch Files • Take advantage of SAS Grid multi-core architecture − Parallel processing • Still some control over dependencies 41
Sample Initialization Section @echo off set pth=\\prod-server\client\product\project\Stat\programs\tables\ set plog=-log \\prod- server\client\product\projec\Stat\Programs\Tables\Log set plst=-print \\prod- server\client\product\projec\Stat\Programs\Tables\Lst\ set gridcfg=-gridconfig “\\prod- server\SAS\BatchFile\sasgsub94.cfg" set SAS= C:\progra~1\SASHome94\SASFoundation\9.4\sasgsub.exe %gridcfg% -gridsasopts "'%plog% %plst%'" -gridsubmitpgm 42
Recommend
More recommend
