Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

sort, uniq, cut, wc, tr — text stream processing

5 min GNU coreutils

Small pipeline utilities; uniq only removes adjacent duplicates, so it almost always needs sort before it.

Cheat sheet

6 commands Printable version
  • sort -u file sort + unique
  • sort file | uniq -c | sort -rn line frequency (top)
  • cut -d: -f1 file a column by delimiter
  • tr 'a-z' 'A-Z' < file case conversion
  • column -t -s, file.csv align into a table
  • comm -12 <(sort a) <(sort b) the intersection of two files

Overview

Small utilities that, chained in a pipeline (|), solve almost any text processing task. Often used together with grep and sed/awk.

Options

10

sort without -n compares lines as text: 10 sorts before 9, because the character "1" is less than "9" — numeric order needs -n. uniq only collapses lines that are identical and adjacent, so it almost always needs a sort first. cut splits fields on a tab by default — for CSV or passwd you set the delimiter explicitly with -d.

Option What it does Example
-n sort — compare values numerically, not lexicographically. sort -n sizes.txt
-c uniq — count how many times each line occurred. sort file | uniq -c
-d cut — set the field delimiter instead of the default tab. cut -d: -f1 /etc/passwd
-f N cut — select fields by number, several separated by commas. cut -d: -f1,3 /etc/passwd
-l wc — count lines. wc -l file
-d tr — delete characters from the given set instead of replacing them. tr -d ' \t' < file
-r sort — reverse the sort order. sort -rn sizes.txt
-k N sort — sort by the given field instead of the whole line. sort -k2 table.txt
-u sort — remove duplicates right after sorting. sort -u file
-d uniq — show only the lines that repeat. sort file | uniq -d

Nothing found. Try a shorter query.

Recipes

3

sort / uniq

bash 
sort file                         # sort lines (lexicographically)
sort -n file                      # as numbers
sort -rn file                     # descending (numbers)
sort -h file                      # human-readable sizes (2K, 1G)
sort -k2 file                     # by the 2nd field
sort -t: -k3 -n /etc/passwd       # ":" delimiter, by the 3rd field, numerically
sort -u file                      # sort + remove duplicates
sort -c file                      # check whether it's already sorted

uniq file                         # remove ADJACENT duplicates (needs sort first!)
sort file | uniq -c               # count how many times each line repeats
sort file | uniq -c | sort -rn    # top by frequency
sort file | uniq -d               # only repeated lines ; -u — only unique ones

cut / tr / wc / more

bash 
cut -d: -f1 /etc/passwd           # field 1, ":" delimiter
cut -d: -f1,3 /etc/passwd         # several fields
cut -c1-10 file                   # characters 1–10
tr 'a-z' 'A-Z' < file             # to uppercase
tr -d ' \t' < file                # remove spaces/tabs
tr -s ' ' < file                  # collapse repeated spaces
tr -cd '[:print:]\n' < file       # drop unprintable characters
wc -l file                        # count lines
wc -w file                        # count words
wc -c file                        # count bytes
paste -d, a.txt b.txt             # glue files together by column
join -t, -1 1 -2 1 a.csv b.csv    # a relational join by key (both sorted)
column -t -s, data.csv            # align into a table
comm -12 <(sort a) <(sort b)      # lines common to both files

Parsing logs and tables

bash 
# top-10 IPs in access.log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# unique values of a CSV column
cut -d, -f3 data.csv | sort -u

# the most frequent words
tr -cs '[:alpha:]' '\n' < text | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head

# the difference between two lists (in A, not in B)
comm -23 <(sort A) <(sort B)

# neatly show /etc/passwd as a table
column -t -s: /etc/passwd

Troubleshooting

5

sort doesn't sort numbers by magnitude

By default, sorting is lexicographic: 10 comes before 9. Numeric sort uses -n, by a specific column with -k:

bash 
sort -n sizes.txt          # by numbers
sort -k2 -nr table.txt     # by the second column, descending

uniq didn't remove duplicates

uniq only collapses adjacent identical lines, so it needs sort before it: sort file | uniq -c.

cut doesn't see the columns

The default delimiter is a tab. For spaces and commas you set it explicitly (cut -d, -f3), and for several spaces in a row awk '{print $3}' is more reliable.

wc -l shows one line fewer

A file's last line without a trailing newline isn't counted. This is normal for the output of some programs; check with tail -c1 file | xxd.

Cyrillic breaks in tr

tr works on bytes, not characters: ranges like а-я produce garbage in UTF-8. For multi-byte text, use sed instead — see sed-awk.

See also

  • sort — sort lines: as text, as numbers, by field.
  • uniq — remove or count adjacent repeats.
  • cut — cut fields out by a delimiter.
  • wc — count lines, words and bytes.
  • redirection — the | pipeline and output redirection.
  • grep — Searching lines by pattern; regexes, recursion, context.
  • sed-awk — Stream editing (sed) and column processing (awk).
  • files-basics — Navigation and files — ls, cd, cp, mv, rm, cat, less, head, tail, copy-paste in the terminal.

Found an inaccuracy? Report an error in the article

Practice on this topic

Tools on this topic

Topics
Loading…