sed: Stream Oriented, Non- Interactive, Text Editor Line-oriented - PDF document

sed: Stream Oriented, Non- Interactive, Text Editor � � Line-oriented tool for pattern matching and CSCI: 4500/6500 Programming replacement (stream editor) Languages » � Looks for patterns one line at a time, like grep » � “Change” lines of the file (but acts as a filter) – � Filter, i.e., does not modify input file SED & AWK » � There is an interactive editor ed that accepts the same commands � � Not really a programming language (cf. awk) 1 2 Maria Hybinette, UGA Maria Hybinette, UGA Syntax Command! (function) � � sed [-n] [-e] [‘command’] [file…] � � sed [-n] [-e] [‘command’] [file…] � � Commands: : � � sed [-n] [-f scriptfile] [file…] » � s – substition [address]s/pattern/flags » � -n – supress output of input lines » � d – delete » � -f scriptfile - next argument is a filename » � And more: y-transform, p-print containing editing commands � � Example » � -e command - the next argument is an editing echo “UNIX programming” | sed 's/.nc./wonderful &/’ command rather than a filename, useful if multiple commands are specified 3 4 Maria Hybinette, UGA Maria Hybinette, UGA Constraining matches by addressing � � Commands can be constrained to accept only � � Diving In Example: single line addresses or ranges of address (or » � echo “UNIX programming” | sed 's/.nc./wonderful &/' a pattern). 5 6 Maria Hybinette, UGA Maria Hybinette, UGA

Another Example Another Example � � sed [-n] [-e] [‘command’] [file…]\\ � � sed [-n] [-e] [‘command’] [file…]\\ {saffron} cat test1.txt {saffron} cat test1.txt first:second first:second one:two one:two {saffron} sed 's/\(.*\):\(.*\)/\2:\1/' test1 {saffron} sed 's/\(.*\):\(.*\)/\2:\1/' test1 second:first two:one 7 8 Maria Hybinette, UGA Maria Hybinette, UGA More examples Address Example � � Convert unix to dos characters. � � Address: sed -e 's/$/\r/' myunix.txt > mydos.txt !! !! » � sed -e 's/$/\r/' myunix.txt > mydos.txt � � delete lines 1-10: sed -e ' d‘ sed -e '1,10 1,10d‘ � � Transform (by character position) � � echo “maria hybinette” | sed -e ’y/aie/xyz/’ echo “maria hybinette” | sed -e ’y/aie/xyz/’ {h70-33-107-14:ingrid:919} sed -e ‘5,14d' afile.txt � � s/Tom/Dick/2 1 2 » � Substitutes Dick for the second occurrence of Tom in the 3 pattern space 4 � � s/wood/plastic/p {h70-33-107-14:ingrid:920} » � Substitutes plastic for the first occurrence of wood and outputs (prints) pattern space ! 9 10 Maria Hybinette, UGA Maria Hybinette, UGA Append, Insert, and Change Change Examples Syntax for these commands is a little � � Remove mail headers, strange because they must be specified ie; the address specifies on multiple lines /^From: /,/^$/c\ a range of lines <Mail Headers Removed> � � append [address]a\ beginning with a line text that begins with From /^From: /,/^$/{ � � insert [address]i\ until the first blank line. s/^From //p text c\ » � The first example replaces all lines with a <Mail Header Removed> � � change [address(es)]c\ } single occurrence of text <Mail Header Removed>. » � The second example � � append/insert for single lines only, not replaces each line with range <Mail Header Removed> 11 12 Maria Hybinette, UGA Maria Hybinette, UGA

Sed Advantages Sed Drawbacks � � Regular expressions � � Hard to remember text from one line to another � � Fast � � Not possible to go backward in the file � � Concise � � No way to do forward references like /..../+1 � � No facilities to manipulate numbers � � Cumbersome syntax 13 14 Maria Hybinette, UGA Maria Hybinette, UGA Why is it called AWK? Awk Programmable Filters Aho Weinberger Kernighan 15 16 Maria Hybinette, UGA Maria Hybinette, UGA Awk Introduction Awk Input � � A general purpose programmable filter that � � awk gets its input from handles text (strings) as easily as numbers » � files » � This makes awk one of the most powerful of the » � redirection and pipes Unix utilities » � directly from standard input � � awk processes fields while sed only processes lines � � nawk (new awk) is the new standard for awk » � Designed to facilitate large awk programs » � gawk is a free nawk clone from GNU 17 18 Maria Hybinette, UGA Maria Hybinette, UGA

AWK Highlights Awk Features over Sed � � A programming language for handling common data manipulation tasks with only � � Convenient numeric processing a few lines of code � � Variables and control flow in the actions � � awk is a pattern-action language, like sed � � Convenient way of accessing fields within � � Looks like C but automatically handles lines input, field splitting, initialization, and � � Flexible printing memory management � � Built-in arithmetic and string functions » � Built-in string and number data types � � C-like syntax » � No variable type declarations � � awk is a great prototyping language » � Start with a few lines and keep adding until it does what you want 19 20 Maria Hybinette, UGA Maria Hybinette, UGA Structure of an AWK Program Review: What is AWK? � � An optional BEGIN segment BEGIN {action} � � Programming language used for manipulating data and generating pretty reports. pattern – � For processing to execute {action} prior to reading input » � Job control too. pattern � � pattern - action pairs {action} – � Processing for input data . – � For each pattern matched, . the corresponding action is . taken pattern � � An optional END segment { action} – � Processing after end of input END {action} data 21 22 Maria Hybinette, UGA Maria Hybinette, UGA Running an AWK Program Patterns and Actions � � There are several ways to run an Awk � � Search a set of files for patterns. program � � Perform specified actions upon lines or fields that contain instances of patterns. » � awk 'program' input_file(s) � � Does not alter input files. – � program and input files are provided as command-line arguments � � Process one input line at a time » � awk 'program' � � This is similar to sed – � program is a command-line argument; input is taken from standard input (yes, awk is a filter!) » � awk -f program_file input_files – � program is read from a file 23 24 Maria Hybinette, UGA Maria Hybinette, UGA

Pattern-Action Structure Patterns � � Selector that determines whether action is � � Every program statement has to have a to be executed pattern or an action or both � � pattern can be: » � Default pattern is to match all lines » � Default action is to print current record » � the special token BEGIN or END � � Patterns are simply listed; » � regular expression (enclosed with //) » � relational or string match expression » � actions are enclosed in { } » � ! negates the match � � awk scans a sequence of input lines , or records , one by one, searching for lines that » � arbitrary combination of the above using && || match the pattern – � /NYU/ matches if the string “NYU” is in the record – � x > 0 matches if the condition is true » � Meaning of match depends on the pattern – � /NYU/ && (name == "UNIX Tools") 25 26 Maria Hybinette, UGA Maria Hybinette, UGA BEGIN and END patterns Actions � � Action � � BEGIN and END provide a way to gain control » � list of one or more C like statements before and after processing, for initialization » � arithmetic and string expressions and and wrap-up. » � assignments and multiple output streams. » � BEGIN: actions are performed before the first input � � action is performed on every line that line is read. matches pattern . » � END: actions are done after the last input line has » � If pattern is not provided, action is performed on every input been processed. line » � If action is not provided, all matching lines are sent to standard output. 27 28 Maria Hybinette, UGA Maria Hybinette, UGA An Example Awk examples � � Add up first column, print sum and average ls | awk ' BEGIN { print "List of html files:" } � � {s += $1 } /\.html$/ { print } � � END {print “sum is”, s, “average is”, s/NR} END { print "There you go!" } � � awk -f awkprogram awkfile ' List of html files: index.html as1.html as2.html There you go! 29 30 Maria Hybinette, UGA Maria Hybinette, UGA