Skip to content

uniq

Removes or counts repeated lines that follow each other.

uniq output

Syntax

uniq [OPTION]... [INPUT [OUTPUT]]

Options

Option Description
-c, --count Put the number of occurrences in front of each line
-d, --repeated Only show lines that occur more than once
-u, --unique Only show lines that occur exactly once
-i, --ignore-case Ignore upper and lower case
-f N, --skip-fields=N Ignore the first N fields when comparing

Examples

Count how often each line occurs:

sort servers.txt | uniq -c
      2 db1
      3 web1
      1 web2

Count the HTTP status codes in a log:

cut -d' ' -f9 /var/log/nginx/access.log | sort | uniq -c
      5 200
      1 302
      3 404

Find the IP addresses with the most requests:

cut -d' ' -f1 /var/log/nginx/access.log | sort | uniq -c | sort -rn
      4 198.51.100.23
      2 192.0.2.77
      2 192.0.2.10
      1 192.0.2.55

Only show lines that occur more than once:

sort servers.txt | uniq -d
db1
web1

uniq without sort:

uniq servers.txt
web1
db1
web2
web1
db1
web1

Nothing was removed, because the duplicates in the file are not next to each other.

When do I need it?

  • To count how often something occurs in a log: IP addresses, status codes, URLs, error messages
  • To find duplicates in lists, for example double entries in a user or host list
  • To shorten a list to its unique entries

The classic pipeline

sort | uniq -c | sort -rn is one of the most useful command chains: it counts all values and puts the most frequent ones on top. Add | head for a top 10.

Always sort first

uniq only compares a line with the one directly before it. Duplicates that are spread over the file are only found after sort.

sort · wc · cut