First steps / Stage 12 of 15

Pipelines: an answer from several commands

Not running Starting usually ~20 s

Step 6 of 7 · lab · 20 min

Count the errors in the logs

A summary of the night's error codes from the shop logs: a pipeline into a file, the command's errors into another one, the history appended.

Situation

Last night an online shop kept answering its customers with errors. Its logs live in /var/log/shop, one file per part of the site. In the morning a colleague asks for a summary: which error codes there were and how many times each. "If you can't read one of the logs, that's fine — just keep the message about it separately, and I'll sort out the access."

The colleague keeps the summaries of earlier nights in ~/report/history.txt, one line per night.

Task

Make it so that:

~/report/errors-by-code.txt has, for each code found in ERROR lines, the number of such lines and the code itself — one line per code, separated by a space, the most frequent first;
~/report/failed.txt holds the error messages of the command that read the logs, and nothing else;
a line for this night appears at the end of ~/report/history.txt, in the same form as the earlier ones: 2026-10-19 and the total number of ERROR lines. The earlier lines must stay where they are.

Read the logs as yourself, without sudo: the summary covers what is open to you.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 3

~/report/errors-by-code.txt lists the count and code of each error, most frequent first — not done yet
~/report/failed.txt holds only the command's error messages — not done yet
~/report/history.txt keeps the earlier lines and ends with a line for 2026-10-19 — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 3

~/report/errors-by-code.txt lists the count and code of each error, most frequent first — not done yet
~/report/failed.txt holds only the command's error messages — not done yet
~/report/history.txt keeps the earlier lines and ends with a line for 2026-10-19 — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 3

~/report/errors-by-code.txt lists the count and code of each error, most frequent first — not done yet
~/report/failed.txt holds only the command's error messages — not done yet
~/report/history.txt keeps the earlier lines and ends with a line for 2026-10-19 — not done yet

Basic commands · picked for this lab

Output redirection and pipelines

Cheat sheet

$_ — insert into the terminal

cmd > out.txt output to a file (overwrite)
cmd >> out.txt output appended to a file
cmd 2> err.txt errors to a file
cmd > all.txt 2>&1 output and errors in one file
cmd &> all.txt the same, shorter (bash)
cmd 2>/dev/null throw errors away
cmd < in.txt input from a file
cmd1 | cmd2 cmd1's output into cmd2's input
cmd 2>&1 | less into a pipe together with errors
cmd | tee out.txt both to the screen and to a file

Output and errors in separate files

bash

./backup.sh > backup.log 2> backup.err

The order of 2>&1 matters

bash

cmd > all.txt 2>&1     # both streams into the file
cmd 2>&1 > all.txt     # only output to the file, errors on screen

The shell reads redirections left to right. In the second line 2>&1 copies where stdout pointed at that moment — the screen.

Process output with a chain of commands

bash

sudo grep 'Failed password' /var/log/auth.log | wc -l
ps -eo rss,comm | sort -rn | head -n 5   # five processes by memory
du -sh /var/log/* 2>/dev/null | sort -rh | head

Each stage does one thing: grep selects, sort orders, wc counts, head trims. Commands like these — they read input and print processed text — are called filters.

Write to a file that needs root

bash

echo '127.0.0.1 app.local' | sudo tee -a /etc/hosts

sudo cmd > /etc/file won't work: your shell does the redirection without root rights. tee runs under sudo and writes the file itself.

Multi-line input right into a command

bash

cat > app.conf <<'EOF'
port = 8080
log = /var/log/app.log
EOF

Empty a log without deleting the file

bash

: > /var/log/app.log         # length 0, file and owner stay
sudo truncate -s 0 /var/log/app.log   # the same for a root-owned file

If a process keeps the log open, rm won't free the space, but emptying will.

Find out which stage of a pipeline failed

bash

false | true
echo "${PIPESTATUS[@]}"      # 1 0 — each stage's exit code
set -o pipefail              # in scripts: any failing stage shows

Permission denied with sudo … > file

Your shell opens the file, not sudo. Use | sudo tee file (or tee -a to append).

bash: $F: ambiguous redirect

The file name in the variable has a space, or the variable is empty. Quote the variable — > "$F" — and check that it's set.

bash: out.txt: cannot overwrite existing file

noclobber mode is on (set -o noclobber); it forbids > onto an existing file. Overwrite explicitly with >|, or turn the mode off with set +o noclobber.

Errors didn't end up in the file

> redirects stdout only. Add 2>&1 after > file, or use &>.

grep in a pipeline doesn't find the error text

The error went to stderr, and a pipe passes only stdout. Pass both streams: cmd 2>&1 | grep text.

The file is empty after sort file > file

The shell truncated file for writing before sort read it. Write to a new file and rename it with mv; sort also has -o file for this.

`>` overwrites without asking

One arrow instead of two, and a log or config is replaced by a single line. Always append with >>. An interactive shell can be protected with set -o noclobber.

`2>/dev/null` hides the cause

Suppressing errors is handy for the output of find /, but in scripts and while troubleshooting the errors are the answer. Read the error first, then decide whether to silence it.

Pipelines: an answer from several commands

Situation

Task

Signing in is needed to start the machine

Output and errors in separate files

The order of 2>&1 matters

Process output with a chain of commands

Write to a file that needs root

Multi-line input right into a command

Empty a log without deleting the file

Find out which stage of a pipeline failed

Permission denied with sudo … > file

bash: $F: ambiguous redirect

bash: out.txt: cannot overwrite existing file

Errors didn't end up in the file

grep in a pipeline doesn't find the error text

The file is empty after sort file > file

> overwrites without asking

2>/dev/null hides the cause

`>` overwrites without asking

`2>/dev/null` hides the cause