Top addresses in a web server log
This answers "who sent the most requests": the address and its request count, largest on top.
uniq — remove or count repeated lines
The uniq command collapses adjacent identical lines and counts them with -c; sort almost always comes before it.
Cheat sheet
6 commands Printable version-
sort file | uniqdrop repeats -
sort file | uniq -chow many times each line occurs -
sort file | uniq -c | sort -rnmost common first -
sort file | uniq -donly the lines that repeat -
sort file | uniq -uonly the lines without a twin -
sort file | uniq -ciignore case
Overview
uniq compares each line with the previous one. Identical lines in a row become one,
and -c prints their count in front. Lines separated by other lines are different as
far as uniq is concerned:
That is why uniq nearly always follows sort: sorting puts equal lines next to
each other. On its own, uniq helps where repeats already come in a row — say, a log
where one error is written many times a second.
The number from uniq -c is right-aligned with spaces. To cut, those spaces are
empty fields, so the count is processed further with sort -rn or awk.
Options
7sort -u gives the same result as sort | uniq, but only uniq -c can count the
repeats. The full list is in man uniq.
| Option | What it does | Example |
|---|---|---|
-c --count |
Prefix each line with the number of adjacent repeats. | sort file | uniq -c |
-d --repeated |
Print one copy of each repeated line only. | sort file | uniq -d |
-u --unique |
Print only lines that are not repeated. | sort file | uniq -u |
-i --ignore-case |
Ignore differences in case. | sort -f file | uniq -ci |
-D --all-repeated |
Print every copy of the repeated lines. | sort hosts.txt | uniq -D |
-f --skip-fields N |
Skip the first N fields when comparing. | uniq -f1 -c events.log |
-w --check-chars N |
Compare only the first N characters of a line. | uniq -w8 -c ids.txt |
Nothing found. Try a shorter query.
Recipes
4How many distinct values
Find duplicates in a list
Compare only part of the line
-f N skips N blank-separated fields; -w N compares the first N characters.
Troubleshooting
4uniq did not remove the repeats
The repeats are not adjacent. Sort before uniq: sort file | uniq.
Lines look identical but are not collapsed
They differ in invisible characters: a trailing space or a \r from a Windows file.
Show them with cat -A file — the line ends with ^M$ or a space before $. Remove
\r with tr -d '\r'.
cut does not pick the number from uniq -c output
The number is padded with spaces, and cut -d' ' sees empty fields. Take it with
awk '{print $1}' or sort by it with sort -rn.
uniq: x: No such file or directory
There is no file at that path. Check the name with ls. The second name in
uniq in.txt out.txt is the file to write, not a second input.
Danger zone
1The second argument is the output file
uniq a.txt b.txt does not compare two files: it reads a.txt and overwrites
b.txt with the result. To compare lists use comm or diff — see text tools
and diff and patch.
See also
- sort — put equal lines together before uniq.
- cut — cut out the field whose repeats you count.
- wc — how many lines are left.
- head — keep the top of a frequency list.
- redirection — how the
|pipeline works. - text-tools — comm, tr and other pipeline links.
Found an inaccuracy? Report an error in the article