UNIQ(1POSIX) | POSIX Programmer's Manual | UNIQ(1POSIX) |
This manual page is part of the POSIX Programmer's Manual. The Linux implementation of this interface may differ (consult the corresponding Linux manual page for details of Linux behavior), or the interface may not be implemented on Linux.
uniq — report or filter out repeated lines in a file
uniq [-c|-d|-u] [-f fields] [-s char] [input_file [output_file]]
The uniq utility shall read an input file comparing adjacent lines, and write one copy of each input line on the output. The second and succeeding copies of repeated adjacent input lines shall not be written. The trailing <newline> of each line in the input shall be ignored when doing comparisons.
Repeated lines in the input shall not be detected if they are not adjacent.
The uniq utility shall conform to the Base Definitions volume of POSIX.1‐2017, Section 12.2, Utility Syntax Guidelines, except that '+' may be recognized as an option delimiter as well as '-'.
The following options shall be supported:
[[:blank:]]*[^[:blank:]]*
If the fields option-argument specifies more fields than appear on an input line, a null string shall be used for comparison.
The following operands shall be supported:
The standard input shall be used only if no input_file operand is specified or if input_file is '-'. See the INPUT FILES section.
The input file shall be a text file.
The following environment variables shall affect the execution of uniq:
Default.
The standard output shall be used if no output_file operand is specified, and shall be used if the output_file operand is '-' and the implementation treats the '-' as meaning standard output. Otherwise, the standard output shall not be used. See the OUTPUT FILES section.
The standard error shall be used only for diagnostic messages.
If the -c option is specified, the output file shall be empty or each line shall be of the form:
"%d %s", <number of duplicates>, <line>
otherwise, the output file shall be empty or each line shall be of the form:
"%s", <line>
None.
The following exit values shall be returned:
Default.
The following sections are informative.
If the collating sequence of the current locale has a total ordering of all characters, the sort utility can be used to cause repeated lines to be adjacent in the input file. If the collating sequence does not have a total ordering of all characters, the sort utility should still do this but it might not. To ensure that all duplicate lines are eliminated, and have the output sorted according the collating sequence of the current locale, applications should use:
LC_ALL=C sort -u | sort
instead of:
sort | uniq
To remove duplicate lines based on whether they collate equally instead of whether they are identical, applications should use:
sort -u
instead of:
sort | uniq
When using uniq to process pathnames, it is recommended that LC_ALL, or at least LC_CTYPE and LC_COLLATE, are set to POSIX or C in the environment, since pathnames can contain byte sequences that do not form valid characters in some locales, in which case the utility's behavior would be undefined. In the POSIX locale each byte is a valid single-byte character, and therefore this problem is avoided.
The following input file data (but flushed left) was used for a test series on uniq:
#01 foo0 bar0 foo1 bar1 #02 bar0 foo1 bar1 foo1 #03 foo0 bar0 foo1 bar1 #04 #05 foo0 bar0 foo1 bar1 #06 foo0 bar0 foo1 bar1 #07 bar0 foo1 bar1 foo0
What follows is a series of test invocations of the uniq utility that use a mixture of uniq options against the input file data. These tests verify the meaning of adjacent. The uniq utility views the input data as a sequence of strings delimited by '\n'. Accordingly, for the fieldsth member of the sequence, uniq interprets unique or repeated adjacent lines strictly relative to the fields+1th member.
uniq -c -f 1 uniq_0I.t 1 #01 foo0 bar0 foo1 bar1 1 #02 bar0 foo1 bar1 foo1 1 #03 foo0 bar0 foo1 bar1 1 #04 2 #05 foo0 bar0 foo1 bar1 1 #07 bar0 foo1 bar1 foo0
The number '2', prefixing the fifth line of output, signifies that the uniq utility detected a pair of repeated lines. Given the input data, this can only be true when uniq is run using the -f 1 option (which shall cause uniq to ignore the first field on each input line).
uniq -d -f 1 uniq_0I.t #05 foo0 bar0 foo1 bar1
uniq -u -f 1 uniq_0I.t #01 foo0 bar0 foo1 bar1 #02 bar0 foo1 bar1 foo1 #03 foo0 bar0 foo1 bar1 #04 #07 bar0 foo1 bar1 foo0
uniq -d -s 2 uniq_0I.t
In the last example, the uniq utility found no input matching the above criteria.
Some historical implementations have limited lines to be 1080 bytes in length, which does not meet the implied {LINE_MAX} limit.
Earlier versions of this standard allowed the -number and +number options. These options are no longer specified by POSIX.1‐2008 but may be present in some implementations.
None.
comm, sort
The Base Definitions volume of POSIX.1‐2017, Chapter 8, Environment Variables, Section 12.2, Utility Syntax Guidelines
Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1-2017, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, 2018 Edition, Copyright (C) 2018 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Standard is the referee document. The original Standard can be obtained online at http://www.opengroup.org/unix/online.html .
Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html .
2017 | IEEE/The Open Group |