Bash: text processing and scripts / Stage 5 of 9 · step 6 of 7

Fix the report script

~25 min

Not running Starting usually ~20 s

Situation

Every night the branches send their uploads to ~/import/inbox, and the importer loads them into the database. Before loading it runs a check and goes on only if the check returned 0:

bash report.sh inbox/*.csv && import-uploads inbox/*.csv

A colleague wrote ~/import/report.sh; the contract is written in the script's header. In the morning you look at the output of the night's check: it has more lines than there are uploads, the empty east.csv passed as OK, and the import started even though the output has a FAIL in it.

Task

Fix ~/import/report.sh so that it keeps its contract:

exactly one line on stdout per argument, in the same order: OK <argument> or FAIL <argument>; the argument exactly as passed, even with a space or an asterisk * in the name; nothing on stderr meanwhile;
OK only if it is a regular file (or a link to one), it can be read and it is not empty; otherwise FAIL;
exit status 1 if at least one file got FAIL, wherever it stands, and 0 if all are OK;
without arguments — empty stdout, the line Usage: report.sh FILE... on stderr and exit status 2.

The check runs the script as you — bash ~/import/report.sh … — on its own files, which are not in inbox. The uploads in inbox are for your experiments: fix the script, not them.

Lab goals

0 of 4

One line per argument with the name exactly as passed, even with a space or an asterisk; stderr is empty — not checked yet
OK only for a regular non-empty file that can be read, or for a link to such a file — not checked yet
Exit status 1 if any file fails, wherever it stands; exit status 0 if all pass — not checked yet
Without arguments — empty stdout, a Usage line on stderr and exit status 2 — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Basic commands · picked for this lab

Bash scripts: running, variables, arguments

Cheat sheet

$_ — insert into the terminal

#!/bin/bash # first line: what runs the file
set -euo pipefail stop at the first error
chmod u+x backup.sh allow running it
./backup.sh "my file.txt" run it by path
bash backup.sh run it without the x bit
bash -n backup.sh check syntax without running
name="value"; echo "$name" a variable — in double quotes
for f in "$@"; do echo "$f"; done every argument, one at a time

Conditions and loops

bash

if [ -f "$file" ]; then
  echo "the file exists"
elif [ -d "$file" ]; then
  echo "it is a directory"
else
  echo "no such thing: $file" >&2
fi

for f in /var/log/*.log; do echo "$f"; done
while read -r host; do ping -c1 "$host"; done < hosts.txt

The [ -f ], [ -z ] and -eq checks are in test-conditions.

Default values and parts of a path

bash

dir="${1:-/srv/backup}"     # the first argument or /srv/backup
f=/var/log/app.log
echo "${f##*/}"             # app.log — the name without the directory
echo "${f%.log}"            # /var/log/app — without the extension
echo "${f%/*}"              # /var/log — the directory

Check the syntax without running

text

student@ubuntu:~$ bash -n syn.sh; echo "rc=$?"
syn.sh: line 5: syntax error: unexpected end of file
rc=2

shellcheck goes deeper: quoting, an unchecked cd, $?.

File names with spaces

bash

for f in /srv/data/*.csv; do gzip "$f"; done   # a pattern, not ls
find /srv/data -name '*.csv' -exec gzip {} +   # with subdirectories

A loop over ls or find output inside $( ) splits names on spaces. A pattern and find -exec pass every name whole (see xargs for -print0).

Permission denied

bash: ./backup.sh: Permission denied, code 126: the file lacks the x permission. Run chmod u+x backup.sh or start it with bash backup.sh.

command not found

backup.sh: command not found, code 127: a name without a path is looked up only in the PATH directories. Run ./backup.sh or put the script in ~/bin.

cannot execute: required file not found

bash: ./backup.sh: cannot execute: required file not found — the kernel could not find the interpreter named on the #! line. Either the path has a typo, or the file came from Windows with CRLF line endings: file backup.sh says with CRLF line terminators. Strip the \r: sed -i 's/\r$//' backup.sh.

[[: not found, Syntax error: "(" unexpected

A bash script was run by sh (dash): it was started as sh backup.sh, or its #! line says /bin/sh. Run ./backup.sh with #!/bin/bash, or bash backup.sh.

name: command not found on a line with =

Spaces around =: name = "x". Write name="x".

unbound variable

backup.sh: line 3: name: unbound variable — set -u caught a variable with no value, often a typo. Fix the name or give it a default: "${name:-}".

syntax error: unexpected end of file

A block or a quote is not closed: if without fi, do without done, a stray ". The line number points at the end of the file — look further up.

The script breaks on a name with a space

An unquoted variable: cp $f /backup/ gets two arguments instead of one. Write "$f" and "$@"; shellcheck finds such places (SC2086).

Works by hand, fails on a schedule

Cron and systemd services have a different environment: a short PATH, the directory /, none of the variables from ~/.bashrc. Details are in the service environment.

rm with an empty variable

rm -rf "$dir/"* with an empty dir turns into rm -rf /*. Use set -u or write "${dir:?}" — an empty value stops the script with an error.

cd without a check

cd /srv/tmp; rm -rf ./* deletes the files in the current directory if cd failed. Write cd /srv/tmp || exit 1.

Fix the report script

Situation

Task

Signing in is needed to start the machine

Conditions and loops

Default values and parts of a path

Check the syntax without running

File names with spaces

Permission denied

command not found

cannot execute: required file not found

[[: not found, Syntax error: "(" unexpected

name: command not found on a line with =

unbound variable

syntax error: unexpected end of file

The script breaks on a name with a space

Works by hand, fails on a schedule

rm with an empty variable

cd without a check