UNIX Course Part II Working with files Andy Hauser LAFUGA & - PowerPoint PPT Presentation

UNIX Course Part II Working with files Andy Hauser LAFUGA & Chair of Animal Breeding and Husbandry Gene Center Munich LMU June, 2016 1

Recall ls list file, information about files cd change working directory mkdir make directory whoami; id information about user groups; id information about group memberships df -h information about disks man manual pages

What is an Operating System? Command Line Interface Desktop Environment Kernel User ABI/API Kernel Resource Management Kernel Drivers Communication Hardware Keyboard CPU RAM Disks Net Mouse

stdin, stdout, stderr append 2>> create / overwrite 2> stderr (2) stdin (0) command stdout (1) create / overwrite > append >>

Connecting commands with pipes $ command1 > /tmp/file1 $ command2 < /tmp/file1 > /tmp/file2 $ command3 < /tmp/file2 Stdout of one command is connected to stdin of another command with pipe „|“. $ command1 | command2 | command3 Advantages: Less typing • No temporary files • Data is shared in memory (fast!) • Submit together and forget • Commands run in parallel •

echo $ echo foo foo $ echo foo bar foo bar $ echo foo bar Arguments are separated by one or more space foo bar $ echo "foo bar" One argument with spaces must be protected. foo bar $ echo -n "foo bar" -n suppresses the adding of a newline at the end foo bar$ stdout goes to a file, creating /overwriting it $ echo hello > hello $ echo world >> world stdout goes to a file, appending to it

cat - concatenate files $ cat hello hello world $ echo hello > hello $ cat hello hello $ cat < hello $ echo world > world $ cat hello world hello world $ cat < hello < world Not necessarily supported by a shell

Creating files $ touch touch.whatever $ echo abc > abc.txt $ cp abc.txt abc.copy.txt $ ls -l abc.txt -rw-rw-r-- 1 andy staff 4 13 Jun 14:32 abc.txt Note that default permissions are influenced by umask. Removing files $ rm touch.whatever

File Metadata $ ls -l abc.txt -rw-rw-r-- 1 andy staff 4 13 Jun 14:32 abc.txt Size Permissions User Group Filename Creation date Size blocks Bytes - rwx rwx rwx User Group Others Closest kicks in. E.g. rwx—-rwx will disallow any group member but not others

chmod - changing permissions $ ls -l world -rw-rw-r-- 1 andy staff 6 13 Jun 15:06 world $ chmod a-rwx world $ ls -l world ---------- 1 andy staff 6 13 Jun 15:06 world $ chmod o+r world $ ls -l world -------r-- 1 andy staff 6 13 Jun 15:06 world $ chmod ug+rwx world $ ls -l world -rwxrwxr-- 1 andy staff 6 13 Jun 15:06 world $ chmod g-w world $ ls -l world -rwxr-xr-- 1 andy staff 6 13 Jun 15:06 world

copying files $ cp abc.txt abc.copy.txt $ scp abc.txt housedata:abc.backup.txt can copy over SSH $ cp -r foo/ bar/ bar/ needs to exist! $ scp -r foo/ housedata:bar/ bar/ will be created $ rsync -av foo/ housedata:bar/

shell globbing * $ echo * .CFUserTextEncoding .DS_Store .RData .Rapp.history .Rhistory .Rprofile .Trash .Xauth ority .asoundrc .bash_history .cache .config .conkyrc .cordova .cups .cvsrc .devilsp ierc .emai .gem .gitconfig .gnupg .hgrc .ionic .lesshst .lldb .local .npm .plugman . rnd .rstudio- desktop .screenrc .ssh .subversion .toprc .vim .viminfo .vimrc .wmii .xbindkeysrc .x pdfrc .zcompdump .zshrc .zshrc.local AndroidStudioProjects Attachments Bout2 Desktop Documents Downloads HistTexte.pdf Library MA Movies Music Pictures Public URLS2 admix backup brew-install.rb bta4_bending_andy_filtered_10_logl.png bta4_bending_andy_filtered_logl.png bta4_bending_andy_logl.png bta4_bending_logl.png chess config dot emai github ivica lm.RData src titel-small.png tmp wrk $ echo */ .Trash/ .cache/ .config/ .cordova/ .cups/ .emai/ .gem/ .gnupg/ .ionic/ .lldb/ .local / .npm/ .plugman/ .rstudio-desktop/ .ssh/ .subversion/ .vim/ AndroidStudioProjects/ Attachments/ Bout2/ Desktop/ Documents/ Downloads/ Library/ MA/ Movies/ Music/ Pictures/ Public/ admix/ backup/ chess/ dot/ emai/ github/ ivica/ src/ tmp/ wrk/ $ echo .* .CFUserTextEncoding .DS_Store .RData .Rapp.history .Rhistory .Rprofile .Trash .Xauth ority .asoundrc .bash_history .cache .config .conkyrc .cordova .cups .cvsrc .devilsp ierc .emai .gem .gitconfig .gnupg .hgrc .ionic .lesshst .lldb .local .npm .plugman . rnd .rstudio- desktop .screenrc .ssh .subversion .toprc .vim .viminfo .vimrc .wmii .xbindkeysrc .x pdfrc .zcompdump .zshrc .zshrc.local $ echo *.png bta4_bending_andy_filtered_10_logl.png bta4_bending_andy_filtered_logl.png bta4_bending_andy_logl.png bta4_bending_logl.png titel-small.png $ echo bta4_bending_* bta4_bending_andy_filtered_10_logl.png bta4_bending_andy_filtered_logl.png bta4_bending_andy_logl.png bta4_bending_logl.png

more shell globbing $ echo ? ? matches any one character zsh: no matches found: ? $ touch a $ echo ? a $ touch b [] for matching character sets, $ echo ? allowing for ranges like a-z. a b $ echo [a-z] And [^] for matching not that a b character set. $ echo [abc] a b $ echo [^abc] zsh: no matches found: [^abc] $ echo [^b-z]

Download a tab seperated file you are familiar with $ wget ftp://ftp.ensemblgenomes.org/pub/release-29/bacteria//gtf/bacteria_86_collection/ escherichia_coli_gca_000770055/Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf.gz --2016-06-14 21:22:13-- ftp://ftp.ensemblgenomes.org/pub/release-29/bacteria//gtf/ bacteria_86_collection/escherichia_coli_gca_000770055/ Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf.gz => 'Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf.gz' Resolving ftp.ensemblgenomes.org... 193.62.197.94 Connecting to ftp.ensemblgenomes.org|193.62.197.94|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /pub/release-29/bacteria//gtf/bacteria_86_collection/ escherichia_coli_gca_000770055 ... done. ==> SIZE Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf.gz ... 323321 ==> PASV ... done. ==> RETR Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf.gz ... done. Length: 323321 (316K) (unauthoritative) Escherichia_coli_gca_0007700 100%[=============================================>] 315.74K 64.5KB/s in 4.9s 2016-06-14 21:22:19 (64.5 KB/s) - 'Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf.gz' saved [323321] $ gunzip Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf.gz $ ls Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf

head - print the first lines from a file $ head Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf #!genome-build ASM77005v1 #!genome-version GCA_000770055.1 #!genome-date 2014-11 #!genome-build-accession GCA_000770055.1 #!genebuild-last-updated 2014-11 Contig0000020 ena gene 680 1846 . + . gene_id "JQ56_06920"; gene_version "1"; gene_name "nhaA"; gene_source "ena"; gene_biotype "protein_coding"; Contig0000020 ena transcript 680 1846 . + . gene_id "JQ56_06920"; gene_version "1"; transcript_id "KGP19944"; transcript_version "1"; gene_name "nhaA"; gene_source "ena"; gene_biotype "protein_coding"; transcript_name "nhaA-1"; transcript_source "ena"; transcript_biotype "protein_coding"; Contig0000020 ena exon 680 1846 . + . gene_id "JQ56_06920"; gene_version "1"; transcript_id "KGP19944"; transcript_version "1"; exon_number "1"; gene_name "nhaA"; gene_source "ena"; gene_biotype "protein_coding"; transcript_name "nhaA-1"; transcript_source "ena"; transcript_biotype "protein_coding"; exon_id "KGP19944-1"; exon_version "1"; Contig0000020 ena CDS 680 1843 . + 0 gene_id "JQ56_06920"; gene_version "1"; transcript_id "KGP19944"; transcript_version "1"; exon_number "1"; gene_name "nhaA"; gene_source "ena"; gene_biotype "protein_coding"; transcript_name "nhaA-1"; transcript_source "ena"; transcript_biotype "protein_coding"; protein_id "KGP19944"; protein_version "1"; Contig0000020 ena start_codon 680 682 . + 0 gene_id "JQ56_06920"; gene_version "1"; transcript_id "KGP19944"; transcript_version "1"; exon_number "1"; gene_name "nhaA"; gene_source "ena"; gene_biotype "protein_coding"; transcript_name "nhaA-1"; transcript_source "ena"; transcript_biotype "protein_coding"; $ head -5 Escherichia_coli_gca_000770055.GCA_000770055.1.29.gtf #!genome-build ASM77005v1 #!genome-version GCA_000770055.1 #!genome-date 2014-11 #!genome-build-accession GCA_000770055.1 #!genebuild-last-updated 2014-11