sed: Stream Oriented, Non- Interactive, Text Editor ! Line-oriented - PDF document

sed: Stream Oriented, Non- Interactive, Text Editor ! Line-oriented tool for pattern matching and replacement (stream editor) CSCI: 4500/6500 Programming » Looks for patterns one line at a time, like grep Languages » “ Change ” lines of the file (but acts as a filter) – Filter, i.e., does not modify input file » There is an interactive editor ed that accepts the SED & AWK same commands » UNIX philosophy – edit a stream, a stream flowing through a pipe ! Sed is not really a programming language (but AWK is) 1 2 Maria Hybinette, UGA Maria Hybinette, UGA Sed Architecture Input ! Basic Syntax ! [address(es)]s/pattern/replacement/[flags] Script file Address Action Address Action Input Line Address Action Output 3 4 Maria Hybinette, UGA Maria Hybinette, UGA Awful Syntax Command! (function) ! sed [-n] [-e] [ ‘ command ’ ] [file…] ! sed [-n] [-e] [ ‘ command ’ ] [file…] ! Command Details: ! sed [-n] [-f scriptfile] [file…] » -n – supress output of input lines ! s – substitution » -f scriptfile - next argument is a filename – [address(es)]s/pattern/replacement/[flags] containing editing commands – sed s/day/night/ » -e command - the next argument is an editing – flags – example ‘ g ’ for global, ‘ n ’ which command rather than a filename, useful if occurrence of pattern should be replaced multiple commands are specified » d – delete – s the ultimate substitution command : » And more: y-transform, p-print ! sed s/day/night/ < old > new � ! sed s/day/night/ old > new � 5 6 Maria Hybinette, UGA Maria Hybinette, UGA

Constraining matches by More Warm-up Examples addressing ! s/Tom/Dick/2 ! Commands can be constrained to accept only single line addresses or ranges of address (or » Substitutes Dick for the second occurrence of Tom a pattern). in the pattern space ! s/wood/plastic/p » Substitutes plastic for the first occurrence of wood and outputs (prints) pattern space 7 8 Maria Hybinette, UGA Maria Hybinette, UGA Another Example ! Diving In Example: ! sed [-n] [-e] [ ‘ command ’ ] [file…] ! Escape : \ » echo “ The UNIX operating system ” | sed 's/.NI./wonderful &/ ’ ! . On character ! Ouch! » Special replacement/patterns – characters {saffron} cat first.txt » & - replaced by the entire string matched in the regular first:second expression for pattern one:two {saffron} sed 's/\(.*\):\(.*\)/\2:\1/' test1 9 10 Maria Hybinette, UGA Maria Hybinette, UGA Another Example Address Example ! sed [-n] [-e] [ ‘ command ’ ] [file…] ! Address: ! Escape, » delete lines 1-10: sed -e ' 1,10d ‘ sed -e '1,10 ! Marking patterns (up to 9): “ \( “ , \) ” {h70-33-107-14:ingrid:919} sed -e ‘ 5,14d' afile.txt 1 {saffron} cat test1.txt 2 first:second 3 one:two 4 {saffron} sed 's/\(.*\):\(.*\)/\2:\1/' test1 {h70-33-107-14:ingrid:920} second:first two:one 11 12 Maria Hybinette, UGA Maria Hybinette, UGA

More examples Append, Insert, and Change Syntax for these commands is a little ! Convert unix to dos characters. strange because they must be specified sed -e 's/$/\r/' myunix.txt > mydos.txt » sed -e 's/$/\r/' myunix.txt > mydos.txt on multiple lines ! Transform with y (by character position) ! append [address]a\ echo “ maria hybinette maria hybinette ” | sed -e | sed -e ’ y/aie/xyz/ y/aie/xyz/ ’ ! echo text ! s/Tom/Dick/2 ! insert [address]i\ » Substitutes Dick for the second occurrence of Tom in the text pattern space ! change [address(es)]c\ ! s/wood/plastic/p text » Substitutes plastic for the first occurrence of wood and outputs (prints) pattern space ! append/insert for single lines only, not range 13 14 Maria Hybinette, UGA Maria Hybinette, UGA Change Examples Sed Advantages ! Remove mail headers, ! Regular expressions ie; the address specifies /^From: /,/^$/c\ ! Fast a range of lines <Mail Headers Removed> ! Concise beginning with a line that begins with From /^From: /,/^$/{ until the first blank line. s/^From //p c\ » The first example replaces all lines with a <Mail Header Removed> } single occurrence of <Mail Header Removed>. » The second example replaces each line with <Mail Header Removed> 15 16 Maria Hybinette, UGA Maria Hybinette, UGA Sed Drawbacks ! Hard to remember text from one line to another ! Not possible to go backward in the file Awk ! No way to do forward references like /..../+1 Programmable Filters ! No facilities to manipulate numbers ! Cumbersome syntax 18 17 Maria Hybinette, UGA Maria Hybinette, UGA

Why is it called AWK? Awk Introduction ! A general purpose programmable filter that handles text (strings) as easily as numbers » This makes awk one of the most powerful of the Unix utilities ! awk processes fields while sed only processes lines ! nawk (new awk) is the new standard for awk » Designed to facilitate large awk programs Aho Weinberger Kernighan » gawk is a free nawk clone from GNU 19 20 Maria Hybinette, UGA Maria Hybinette, UGA AWK Highlights Awk Input ! A programming language for handling ! awk gets its input from common data manipulation tasks with only » files a few lines of code » redirection and pipes ! awk is a pattern-action language, like sed » directly from standard input ! Looks like C but automatically handles input, field splitting, initialization, and memory management » Built-in string and number data types » No variable type declarations ! awk is a great prototyping language » Start with a few lines and keep adding until it does what you want 21 22 Maria Hybinette, UGA Maria Hybinette, UGA Awk Features over Sed Structure of an AWK Program ! An optional BEGIN segment ! Convenient numeric processing BEGIN {action} pattern ! Variables and control flow in the actions – For processing to execute {action} prior to reading input ! Convenient way of accessing fields within pattern ! pattern - action pairs lines {action} – Processing for input data ! Flexible printing . – For each pattern matched, . ! Built-in arithmetic and string functions the corresponding action is . ! C-like syntax taken pattern ! An optional END segment { action} – Processing after end of input END {action} data 23 24 Maria Hybinette, UGA Maria Hybinette, UGA

Review: What is AWK? Running an AWK Program ! Programming language used for manipulating ! There are several ways to run an Awk data and generating pretty reports. program » Job control too. » awk 'program' input_file(s) – program and input files are provided as command-line arguments » awk 'program' – program is a command-line argument; input is taken from standard input (yes, awk is a filter!) » awk -f program_file input_files – program is read from a file 25 26 Maria Hybinette, UGA Maria Hybinette, UGA Patterns and Actions Pattern-Action Structure ! Every program statement has to have a ! Search a set of files for patterns. pattern or an action or both ! Perform specified actions upon lines or fields » Default pattern is to match all lines that contain instances of patterns. » Default action is to print current record ! Does not alter input files. ! Patterns are simply listed; ! Process one input line at a time » actions are enclosed in { } ! So for this is similar to sed (except fields) ! awk scans a sequence of input lines , or records , one by one, searching for lines that match the pattern » Meaning of match depends on the pattern 27 28 Maria Hybinette, UGA Maria Hybinette, UGA Patterns BEGIN and END patterns ! A selector that determines whether action ! BEGIN and END provide a way to gain control is to be executed before and after processing, for initialization ! pattern can be: and wrap-up. » BEGIN: actions are performed before the first input » the special token BEGIN or END line is read. » regular expression (enclosed with //) » END: actions are done after the last input line has » relational or string match expression been processed. » ! negates the match » arbitrary combination of the above using && || – /NYU/ matches if the string “ NYU ” is in the record – x > 0 matches if the condition is true – /NYU/ && (name == "UNIX Tools") 29 30 Maria Hybinette, UGA Maria Hybinette, UGA

Actions An Example ! Action ls | awk ' » list of one or more C like statements BEGIN { print "List of html files:" } » arithmetic and string expressions and /\.html$/ { print } » assignments and multiple output streams. END { print "There you go!" } ! action is performed on every line that ' matches pattern . » If pattern is not provided, action is performed on every input line List of html files: index.html » If action is not provided, all matching lines are sent to as1.html standard output. as2.html There you go! 31 32 Maria Hybinette, UGA Maria Hybinette, UGA Awk examples Tutorials ! Add up first column, print sum and average ! SED » http://www.grymoire.com/Unix/Sed.html ! {s += $1 } – Great reference card available ! END {print “ sum is ” , s, “ average is ” , s/NR} » http://sed.sourceforge.net/grabbag/tutorials/ ! awk -f awkprogram awkfile ! AWK 33 34 Maria Hybinette, UGA Maria Hybinette, UGA