Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

sed and awk — editing and fields in a text stream

10 min Ubuntu 24.04 · GNU sed 4.9 · GNU awk 5.2

sed edits and deletes lines by pattern, awk splits lines into fields and computes over them, line by line.

Cheat sheet

8 commands Printable version
  • sed -n 's/old/new/p' app.ini preview the changed lines
  • sed -i.bak 's/old/new/g' app.ini edit, keep app.ini.bak
  • sed -n '5,10p' file lines 5 to 10
  • sed '/^ *#/d; /^$/d' file drop comments and empty lines
  • sed '/^ *#/! s/old/new/' file replace outside comments
  • awk '{print $1}' access.log the first field of every line
  • awk -F: '$3 >= 1000 {print $1}' /etc/passwd filter by a field
  • awk '{s += $4} END {print s}' log.txt sum of the fourth field

Overview

sed and awk read text line by line — from a file or a pipeline — and print the result. The file stays as it was until sed gets the -i option.

  • sed is a stream editor: s/pattern/replacement/ changes text, d deletes a line, p prints it.
  • awk splits a line into fields $1, $2… and runs an action for the lines that match a condition.

On the lab machines sed is GNU sed 4.9 and awk is GNU awk 5.2.

The s command

s/pattern/replacement/flags. The pattern is a regular expression; the flags are g — every match in the line (otherwise only the first), p — print the changed line, I — ignore case. Any character after s works as the delimiter: s#/old/path#/new/path# is handier when the pattern contains /.

A dot in a pattern means any character, so an unescaped address matches too much:

text 
student@ubuntu:~$ cat app.ini
# db_host=10.20.0.15:5432 old
db_host=10.20.0.15:5432
name=app
db_host = 10.20.0.15:5432
cache=10.20.0.150:6379
student@ubuntu:~$ sed -n 's/10.20.0.15/X/p' app.ini
# db_host=X:5432 old
db_host=X:5432
db_host = X:5432
cache=X0:6379

10.20.0.150 matched as well. The pattern 10\.20\.0\.15: with escaped dots and a trailing colon matches only the address you mean.

Line addresses

A command can be limited to certain lines:

Address Lines
5 the fifth
5,10 the fifth to the tenth
$ the last
/^db_host/ those matching the pattern
/^ *#/! all except those matching

A ! after an address inverts it. This is how a replacement skips commented lines:

bash 
sed -n '/^ *#/! s/10\.20\.0\.15:/10.30.0.7:/p' app.ini
text 
db_host=10.30.0.7:5432
db_host = 10.30.0.7:5432

Groups and \1

With -E (extended expressions), parentheses ( ) capture part of the match, and \1, \2 insert it into the replacement:

bash 
sed -E 's/^(db_host) *= *.*/\1=10.30.0.7:5432/' app.ini
text 
# db_host=10.20.0.15:5432 old
db_host=10.30.0.7:5432
name=app
db_host=10.30.0.7:5432
cache=10.20.0.150:6379

Both spellings of the key, with and without spaces, now look the same, and the comment is untouched because the pattern starts with ^. Without -E, parentheses are plain characters and \1 has nothing to refer to (see the invalid reference problem below).

awk fields

An awk program is a set of condition {action} pairs. $0 is the whole line, $1… are fields, NF is the number of fields, NR is the line number. A condition compares a field ($4 == "refund"), checks the line number (NR > 1 skips a header) or matches a pattern (/error/). Without a condition the action runs for every line; without an action a matching line is printed whole.

An END {…} block runs after the last line, and an array s[key] collects values by key:

text 
student@ubuntu:~$ cat log.txt
GET /a 200 512
GET /b 404 0
POST /a 200 1024
GET /a 500 128
bash 
awk '{s[$3] += $4} END {for (k in s) print k, s[k]}' log.txt | sort
text 
200 1536
404 0
500 128

The order of for (k in s) is undefined, hence the sort. Fields are split on spaces and tabs; -F sets another separator: awk -F: … for /etc/passwd, awk -F, … for CSV without quotes.

Options

5

The main options of both programs are in the table below. The full lists are in man sed and man gawk.

Option What it does Example
-i SUFFIX sed — edit the file in place; with no argument, keeps no backup. dangerous sed -i.bak 's/old/new/g' file
-n sed — suppress automatic printing; paired with the p command. sed -n '5,10p' file
-F SEP awk — set the field separator instead of the default whitespace. awk -F: '{print $1}' /etc/passwd
-E sed — extended regular expressions (no need to escape groups). sed -E 's/(a|b)+/X/g' file
-v var=val awk — pass a variable value into the program. awk -v n=5 '{print $n}' file

Nothing found. Try a shorter query.

Recipes

5

Preview a replacement, then edit

bash 
sed -n 's/10\.20\.0\.15:/10.30.0.7:/p' app.ini     # what will change
sed -i.bak 's/10\.20\.0\.15:/10.30.0.7:/' app.ini  # edit with a backup
diff app.ini.bak app.ini                          # what changed

diff shows every changed line before and after:

text 
student@ubuntu:~$ sed -i.bak 's/^name=app$/name=web/' app.ini
student@ubuntu:~$ diff app.ini.bak app.ini
3c3
< name=app
---
> name=web

Delete lines

bash 
sed -i '/^ *#/d; /^$/d' app.ini    # comments and empty lines
sed -i '/^cache=/d' app.ini        # the line with a key

Replace a whole line or add one next to it

bash 
sed -i '/^name=/c\name=web' app.ini        # replace the line
sed -i '/^name=/a\env=prod' app.ini        # add a line after it

One edit across many files

bash 
grep -rl '10\.20\.0\.15:' /etc/app.d/      # where it occurs
sed -i.bak 's/10\.20\.0\.15:/10.30.0.7:/' /etc/app.d/*.conf

The list of files first, the edit second: that way you see what will change.

Filter and count in awk

text 
student@ubuntu:~$ awk '$3 >= 500' log.txt
GET /a 500 128
student@ubuntu:~$ awk 'END {print NR}' log.txt
4
student@ubuntu:~$ awk -F: '$3 >= 1000 {print $1}' /etc/passwd
nobody
student

Troubleshooting

7

sed -i changed nothing

The pattern matched no line. Run the same replacement with -n … p: empty output means the pattern is the problem — case, spaces around =, a dot or * without \.

More was replaced than intended

The pattern is wider than needed: an unescaped dot matched any character, there is no ^ at the start, no address excludes comments. Check the sed -n … p output before editing.

unterminated `s' command

sed 's/a/b' is missing the last delimiter. The full command is s/pattern/replacement/.

unknown option to `s'

The pattern or the replacement contains /, and sed took the rest for flags. Change the delimiter: s#/old/path#/new/path#.

invalid reference \1 on `s' command's RHS

The replacement has \1, but the pattern has no group: without -E, parentheses are plain characters. Add -E or write the group as \(…\).

awk: unexpected newline or end of string

A brace or a quote in the program is not closed:

text 
awk: cmd. line:1: {print $1
awk: cmd. line:1:          ^ unexpected newline or end of string

Put the awk program in single quotes, so the shell does not expand $1.

awk prints empty fields

The line is split with the wrong separator: /etc/passwd uses :, CSV uses a comma. Set -F.

Danger zone

2

sed -i without a preview or a backup

sed -i rewrites the file without asking. A wrong pattern applied to many files at once (*.conf) can only be undone from backups: use -i.bak and check the replacement with -n … p before editing.

Editing in place creates a new file

sed -i writes the result to a temporary file and moves it over the old one. A symbolic link turns into a regular file, and a hard link stops pointing at the edited content. For links, use --follow-symlinks or edit the target.

See also

  • regex — pattern syntax for s and addresses.
  • grep — find lines and files before editing.
  • diff-patch — compare a file before and after.
  • xargs — pass a list of files to sed -i.
  • find — pick the files to edit.
  • cut — simple fields without awk.
  • text-tools — sort, uniq and friends for reports.
  • redirection — the pipelines text flows through.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…