Bash: text processing and scripts / Stage 2 of 9 · step 7 of 9

A report from the nginx log

~30 min

Not running Starting usually ~20 s

Situation

Your manager wants a report on the shop.lab.test store site for yesterday, 19 October: which requests the site took longest to handle, and how many bytes it sent with each response code. The site runs on nginx; its request log is /var/log/nginx/shop.access.log, and the line format is set in /etc/nginx/conf.d/shop-log.conf.

The manager compares reports from different days, so the report always has the same layout.

Task

Build ~/report.txt out of two sections:

[slowest]
<time> <URL>
…
[bytes]
<code> <bytes>
…

The [slowest] section holds the five longest requests of 19 October: the request time as written in the log and the request URL, separated by a space. The longest one first.
The [bytes] section holds every response code that occurs on 19 October and the sum of bytes sent with that code during the day ($body_bytes_sent), separated by a space. Codes in ascending order.

The day runs from 00:00:00 to 23:59:59 by the log's clock. Only read the logs, don't change them. First look at what you are working with: ls -l /var/log/nginx and head -n 3 of the log. Build each section as a pipeline one link at a time and watch the output on screen; send it to the file once the output is right.

Lab goals

0 of 2

The [slowest] section lists the five longest requests of 19 October — time and URL, the longest first — not checked yet
The [bytes] section holds the sum of bytes of 19 October for each response code, codes in ascending order — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Text and search · picked for this lab

grep — searching text

Cheat sheet

$_ — insert into the terminal

grep -i word file case-insensitive
grep -rn word dir/ recursive + line numbers
grep -v word file lines without the word
grep -C2 word file ±2 lines of context
grep -rn --include='*.conf' word /etc
grep -oP 'regex' file match only, Perl regex
cmd | grep -F "literal" a literal string

Flag	Meaning
`-i`	case-insensitive
`-r` / `-R`	recursive through directories (R also follows symlinks)
`-n`	show line numbers
`-v`	invert (lines WITHOUT the pattern)
`-c`	count matches
`-l` / `-L`	only filenames with a match / without one
`-w` / `-x`	a whole word / a whole line
`-E` / `-F` / `-P`	extended regex / a fixed string / Perl regex
`-A N`/`-B N`/`-C N`	N lines after / before / around
`-o`	only the matched part
`--include`/`--exclude`	filter filenames with `-r`

Examples

bash

grep -rni "timeout" /etc                    # recursive, case-insensitive, with line numbers
grep -E "warn|error|fatal" app.log          # an alternative (OR)
grep -A3 -B1 "panic" app.log                # context around it
grep -c "GET" access.log                     # how many GET requests
grep -rl "api_key" .                         # which files contain it
grep -rn --include='*.py' "def main" .       # only .py files
grep -rn --exclude-dir={.git,node_modules} "TODO" .
grep -oP '\d+\.\d+\.\d+\.\d+' access.log | sort | uniq -c | sort -rn   # top IPs (-P, -o)
grep -Ff patterns.txt data.txt               # match any line from patterns.txt
ps aux | grep -v grep | grep nginx

grep itself shows up in `ps aux | grep`

A classic. Exclude yourself: grep -v grep, or pgrep, or the grep "[n]ginx" trick.

Special characters in the pattern get treated as regex

A dot, brackets, + are regex. Escape them, or use grep -F (a fixed string): grep -F "1.2.3.4".

`\d`, `\w`, lookahead don't work

Basic/extended grep doesn't know them — you need -P (PCRE): grep -P '\d{3}'.

I expect matches, but grep is silent

Check the case (-i), the right file, and that the data is actually arriving. grep may skip binary files — -a treats them as text.

Too slow over a large tree

Exclude directories (--exclude-dir), or use ripgrep (rg) — noticeably faster.

A report from the nginx log

Situation

Task

Signing in is needed to start the machine

Examples

grep itself shows up in ps aux | grep

Special characters in the pattern get treated as regex

\d, \w, lookahead don't work

I expect matches, but grep is silent

Too slow over a large tree

grep itself shows up in `ps aux | grep`

`\d`, `\w`, lookahead don't work