Bash: text processing and scripts / Stage 2 of 9

sed and awk: editing and fields of a stream

Not running Starting usually ~20 s

Step 8 of 9 · lab · 25 min

Change an address in every config

The backend has moved: thirty configs with the address written in different ways, and the comments and similar addresses must stay.

Situation

The store runs thirty instances of its application, each with its own settings file in /etc/app.d/ — lines of the form key=value. The orders backend has moved: it was 10.20.0.15:8080, now it is 10.20.0.40:8080. The old server goes off tonight, and by then the instances that talk to it must talk to the new one. The key holding the backend address is upstream.

Different people have edited these configs by hand for years.

Task

Make sure that in every /etc/app.d/*.conf:

each upstream line that pointed to 10.20.0.15:8080 points to 10.20.0.40:8080;
the comments stay as they were, including commented-out settings: your colleagues read the history from them;
the other lines don't change — other keys and the upstream lines that point to other addresses.

There is no need to restart the application, it rereads its settings by itself. Editing thirty files by hand takes long — edit them as a stream, but first look at how upstream is written in different files.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 3

In every /etc/app.d/*.conf the upstream lines that pointed to 10.20.0.15:8080 point to 10.20.0.40:8080 — not done yet
The comments and commented-out settings are as they were — not done yet
The other lines are as they were, including upstream lines with other addresses — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 3

In every /etc/app.d/*.conf the upstream lines that pointed to 10.20.0.15:8080 point to 10.20.0.40:8080 — not done yet
The comments and commented-out settings are as they were — not done yet
The other lines are as they were, including upstream lines with other addresses — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 3

In every /etc/app.d/*.conf the upstream lines that pointed to 10.20.0.15:8080 point to 10.20.0.40:8080 — not done yet
The comments and commented-out settings are as they were — not done yet
The other lines are as they were, including upstream lines with other addresses — not done yet

Text and search · picked for this lab

sed and awk — editing and fields in a text stream

Cheat sheet

$_ — insert into the terminal

sed -n 's/old/new/p' app.ini preview the changed lines
sed -i.bak 's/old/new/g' app.ini edit, keep app.ini.bak
sed -n '5,10p' file lines 5 to 10
sed '/^ *#/d; /^$/d' file drop comments and empty lines
sed '/^ *#/! s/old/new/' file replace outside comments
awk '{print $1}' access.log the first field of every line
awk -F: '$3 >= 1000 {print $1}' /etc/passwd filter by a field
awk '{s += $4} END {print s}' log.txt sum of the fourth field

The main options of both programs are in the table below. The full lists are in man sed and man gawk.

Preview a replacement, then edit

bash

sed -n 's/10\.20\.0\.15:/10.30.0.7:/p' app.ini     # what will change
sed -i.bak 's/10\.20\.0\.15:/10.30.0.7:/' app.ini  # edit with a backup
diff app.ini.bak app.ini                          # what changed

diff shows every changed line before and after:

text

student@ubuntu:~$ sed -i.bak 's/^name=app$/name=web/' app.ini
student@ubuntu:~$ diff app.ini.bak app.ini
3c3
< name=app
---
> name=web

Delete lines

bash

sed -i '/^ *#/d; /^$/d' app.ini    # comments and empty lines
sed -i '/^cache=/d' app.ini        # the line with a key

Replace a whole line or add one next to it

bash

sed -i '/^name=/c\name=web' app.ini        # replace the line
sed -i '/^name=/a\env=prod' app.ini        # add a line after it

One edit across many files

bash

grep -rl '10\.20\.0\.15:' /etc/app.d/      # where it occurs
sed -i.bak 's/10\.20\.0\.15:/10.30.0.7:/' /etc/app.d/*.conf

The list of files first, the edit second: that way you see what will change.

Filter and count in awk

text

student@ubuntu:~$ awk '$3 >= 500' log.txt
GET /a 500 128
student@ubuntu:~$ awk 'END {print NR}' log.txt
4
student@ubuntu:~$ awk -F: '$3 >= 1000 {print $1}' /etc/passwd
nobody
student

sed -i changed nothing

The pattern matched no line. Run the same replacement with -n … p: empty output means the pattern is the problem — case, spaces around =, a dot or * without \.

More was replaced than intended

The pattern is wider than needed: an unescaped dot matched any character, there is no ^ at the start, no address excludes comments. Check the sed -n … p output before editing.

unterminated `s' command

sed 's/a/b' is missing the last delimiter. The full command is s/pattern/replacement/.

unknown option to `s'

The pattern or the replacement contains /, and sed took the rest for flags. Change the delimiter: s#/old/path#/new/path#.

invalid reference \1 on `s' command's RHS

The replacement has \1, but the pattern has no group: without -E, parentheses are plain characters. Add -E or write the group as $…$.

awk: unexpected newline or end of string

A brace or a quote in the program is not closed:

text

awk: cmd. line:1: {print $1
awk: cmd. line:1:          ^ unexpected newline or end of string

Put the awk program in single quotes, so the shell does not expand $1.

awk prints empty fields

The line is split with the wrong separator: /etc/passwd uses :, CSV uses a comma. Set -F.

sed -i without a preview or a backup

sed -i rewrites the file without asking. A wrong pattern applied to many files at once (*.conf) can only be undone from backups: use -i.bak and check the replacement with -n … p before editing.

Editing in place creates a new file

sed -i writes the result to a temporary file and moves it over the old one. A symbolic link turns into a regular file, and a hard link stops pointing at the edited content. For links, use --follow-symlinks or edit the target.

sed and awk: editing and fields of a stream

Situation

Task

Signing in is needed to start the machine

Preview a replacement, then edit

Delete lines

Replace a whole line or add one next to it

One edit across many files

Filter and count in awk

sed -i changed nothing

More was replaced than intended

unterminated `s' command

unknown option to `s'

invalid reference \1 on `s' command's RHS

awk: unexpected newline or end of string

awk prints empty fields

sed -i without a preview or a backup

Editing in place creates a new file