Bash: text processing and scripts / Stage 2 of 9

sed and awk: editing and fields of a stream

Not running Starting usually ~20 s

Step 7 of 9 · lab · 30 min

A report from the nginx log

The five longest requests and the bytes per response code for one day, from an nginx log cut in two by rotation.

Situation

Your manager wants a report on the shop.lab.test store site for yesterday, 19 October: which requests the site took longest to handle, and how many bytes it sent with each response code. The site runs on nginx; its request log is /var/log/nginx/shop.access.log, and the line format is set in /etc/nginx/conf.d/shop-log.conf.

The manager compares reports from different days, so the report always has the same layout.

Task

Build ~/report.txt out of two sections:

[slowest]
<time> <URL>
…
[bytes]
<code> <bytes>
…

The [slowest] section holds the five longest requests of 19 October: the request time as written in the log and the request URL, separated by a space. The longest one first.
The [bytes] section holds every response code that occurs on 19 October and the sum of bytes sent with that code during the day ($body_bytes_sent), separated by a space. Codes in ascending order.

The day runs from 00:00:00 to 23:59:59 by the log's clock. Only read the logs, don't change them. First look at what you are working with: ls -l /var/log/nginx and head -n 3 of the log. Build each section as a pipeline one link at a time and watch the output on screen; send it to the file once the output is right.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 2

The [slowest] section lists the five longest requests of 19 October — time and URL, the longest first — not done yet
The [bytes] section holds the sum of bytes of 19 October for each response code, codes in ascending order — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 2

The [slowest] section lists the five longest requests of 19 October — time and URL, the longest first — not done yet
The [bytes] section holds the sum of bytes of 19 October for each response code, codes in ascending order — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 2

The [slowest] section lists the five longest requests of 19 October — time and URL, the longest first — not done yet
The [bytes] section holds the sum of bytes of 19 October for each response code, codes in ascending order — not done yet

Text and search · picked for this lab

grep — searching text

Cheat sheet

$_ — insert into the terminal

grep -i word file case-insensitive
grep -rn word dir/ recursive + line numbers
grep -v word file lines without the word
grep -C2 word file ±2 lines of context
grep -rn --include='*.conf' word /etc
grep -oP 'regex' file match only, Perl regex
cmd | grep -F "literal" a literal string

Flag	Meaning
`-i`	case-insensitive
`-r` / `-R`	recursive through directories (R also follows symlinks)
`-n`	show line numbers
`-v`	invert (lines WITHOUT the pattern)
`-c`	count matches
`-l` / `-L`	only filenames with a match / without one
`-w` / `-x`	a whole word / a whole line
`-E` / `-F` / `-P`	extended regex / a fixed string / Perl regex
`-A N`/`-B N`/`-C N`	N lines after / before / around
`-o`	only the matched part
`--include`/`--exclude`	filter filenames with `-r`

Examples

bash

grep -rni "timeout" /etc                    # recursive, case-insensitive, with line numbers
grep -E "warn|error|fatal" app.log          # an alternative (OR)
grep -A3 -B1 "panic" app.log                # context around it
grep -c "GET" access.log                     # how many GET requests
grep -rl "api_key" .                         # which files contain it
grep -rn --include='*.py' "def main" .       # only .py files
grep -rn --exclude-dir={.git,node_modules} "TODO" .
grep -oP '\d+\.\d+\.\d+\.\d+' access.log | sort | uniq -c | sort -rn   # top IPs (-P, -o)
grep -Ff patterns.txt data.txt               # match any line from patterns.txt
ps aux | grep -v grep | grep nginx

grep itself shows up in `ps aux | grep`

A classic. Exclude yourself: grep -v grep, or pgrep, or the grep "[n]ginx" trick.

Special characters in the pattern get treated as regex

A dot, brackets, + are regex. Escape them, or use grep -F (a fixed string): grep -F "1.2.3.4".

`\d`, `\w`, lookahead don't work

Basic/extended grep doesn't know them — you need -P (PCRE): grep -P '\d{3}'.

I expect matches, but grep is silent

Check the case (-i), the right file, and that the data is actually arriving. grep may skip binary files — -a treats them as text.

Too slow over a large tree

Exclude directories (--exclude-dir), or use ripgrep (rg) — noticeably faster.

sed and awk: editing and fields of a stream

Situation

Task

Signing in is needed to start the machine

Examples

grep itself shows up in ps aux | grep

Special characters in the pattern get treated as regex

\d, \w, lookahead don't work

I expect matches, but grep is silent

Too slow over a large tree

grep itself shows up in `ps aux | grep`

`\d`, `\w`, lookahead don't work