Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

Output redirection and pipelines

6 min Ubuntu 24.04 · Bash 5.2

The shell sends a command's input and output to a file or another command — >, >>, 2>&1, /dev/null, the | pipe.

Cheat sheet

10 commands Printable version
  • cmd > out.txt output to a file (overwrite)
  • cmd >> out.txt output appended to a file
  • cmd 2> err.txt errors to a file
  • cmd > all.txt 2>&1 output and errors in one file
  • cmd &> all.txt the same, shorter (bash)
  • cmd 2>/dev/null throw errors away
  • cmd < in.txt input from a file
  • cmd1 | cmd2 cmd1's output into cmd2's input
  • cmd 2>&1 | less into a pipe together with errors
  • cmd | tee out.txt both to the screen and to a file

Overview

Every program starts with three open streams. A stream's number is a file descriptor:

Stream What it is
0 — stdin input: the keyboard by default
1 — stdout normal output: the screen by default
2 — stderr errors and warnings: also the screen

A redirection tells the shell to connect a stream to a file or another command. The shell does it before the command starts, and the command doesn't know about it. That's why errors still show on screen when the output went to a file:

text 
student@ubuntu:~$ ls /etc/hostname /nope > out.txt
ls: cannot access '/nope': No such file or directory
student@ubuntu:~$ cat out.txt
/etc/hostname

2>&1 means "stream 2 goes wherever stream 1 goes right now". A pipe | connects one command's stdout to the next one's stdin; all stages run at the same time.

Recipes

7

Output and errors in separate files

bash 
./backup.sh > backup.log 2> backup.err

The order of 2>&1 matters

bash 
cmd > all.txt 2>&1     # both streams into the file
cmd 2>&1 > all.txt     # only output to the file, errors on screen

The shell reads redirections left to right. In the second line 2>&1 copies where stdout pointed at that moment — the screen.

Process output with a chain of commands

bash 
sudo grep 'Failed password' /var/log/auth.log | wc -l
ps -eo rss,comm | sort -rn | head -n 5   # five processes by memory
du -sh /var/log/* 2>/dev/null | sort -rh | head

Each stage does one thing: grep selects, sort orders, wc counts, head trims. Commands like these — they read input and print processed text — are called filters.

Write to a file that needs root

bash 
echo '127.0.0.1 app.local' | sudo tee -a /etc/hosts

sudo cmd > /etc/file won't work: your shell does the redirection without root rights. tee runs under sudo and writes the file itself.

Multi-line input right into a command

bash 
cat > app.conf <<'EOF'
port = 8080
log = /var/log/app.log
EOF

Empty a log without deleting the file

bash 
: > /var/log/app.log         # length 0, file and owner stay
sudo truncate -s 0 /var/log/app.log   # the same for a root-owned file

If a process keeps the log open, rm won't free the space, but emptying will.

Find out which stage of a pipeline failed

bash 
false | true
echo "${PIPESTATUS[@]}"      # 1 0 — each stage's exit code
set -o pipefail              # in scripts: any failing stage shows

Troubleshooting

6

Permission denied with sudo … > file

Your shell opens the file, not sudo. Use | sudo tee file (or tee -a to append).

bash: $F: ambiguous redirect

The file name in the variable has a space, or the variable is empty. Quote the variable — > "$F" — and check that it's set.

bash: out.txt: cannot overwrite existing file

noclobber mode is on (set -o noclobber); it forbids > onto an existing file. Overwrite explicitly with >|, or turn the mode off with set +o noclobber.

Errors didn't end up in the file

> redirects stdout only. Add 2>&1 after > file, or use &>.

grep in a pipeline doesn't find the error text

The error went to stderr, and a pipe passes only stdout. Pass both streams: cmd 2>&1 | grep text.

The file is empty after sort file > file

The shell truncated file for writing before sort read it. Write to a new file and rename it with mv; sort also has -o file for this.

Danger zone

2

> overwrites without asking

One arrow instead of two, and a log or config is replaced by a single line. Always append with >>. An interactive shell can be protected with set -o noclobber.

2>/dev/null hides the cause

Suppressing errors is handy for the output of find /, but in scripts and while troubleshooting the errors are the answer. Read the error first, then decide whether to silence it.

Terms

7
standard input
The input stream on file descriptor 0; its source may be a terminal, file, or pipe. Also appears in: wc, /proc: processes and the kernel as files
standard output
The main output stream on file descriptor 1; it can be redirected. Also appears in: cat, strace
standard error
The diagnostic stream on file descriptor 2, separate from standard output. Also appears in: Exit codes: $?, && and ||, sysctl, Bash scripts: running, variables, arguments, strace
file descriptor
A non-negative integer a process uses to refer to an open file, pipe, or socket. By convention, 0, 1, and 2 are the standard streams; a process's open descriptors are listed in /proc/<PID>/fd. Also appears in: flock, Log rotation with logrotate, lsof, strace and 1 more
pipeline
A chain of commands joined by |: each command's standard output feeds the next command's standard input, and the commands run concurrently. The pipeline's exit status is that of the last command unless pipefail is set. Also appears in: Exit codes: $?, && and ||, sed and awk, sort, watch and 13 more
/dev/null
A special device file: it accepts any output and keeps nothing, and reading from it gives end of file at once. 2> /dev/null throws error messages away. Also appears in: /proc: processes and the kernel as files, AppArmor, find, HTTP and 4 more
filter
A command that reads a file or standard input, processes the text and prints the result to standard output: grep, head, sort, uniq, wc. Filters are chained into a pipeline. Also appears in: fail2ban, jq, conntrack, ss and 27 more

See also

  • bash — the shell that performs redirections.
  • tee — split a stream: to the screen and a file, sudo included.
  • cat — join files and write short text with <<EOF.
  • grep — the main filter in pipelines.
  • sort, uniq, cut, wc — the filters pipelines are built from.
  • text-tools — tr, paste, column, comm and other pipeline stages.
  • xargs — turn output into arguments for the next command.
  • bash-scripting — set -o pipefail and exit codes in scripts.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…