Pipes

pipe
What is a pipe?

It is a way of connecting the output of one command to the input of another command in a shell.

Why would I want to do that?
  • Process Chaining: you can connect simple commands in sequence to perform complex tasks.
  • Efficiency: data is streamed directly between commands without writing to disk.
  • Speed: The command reading from the pipe can start consuming data while the command writing to the pipe is still producing data. There is not need to wait for all data to be written to a temporary file on disk before it can be consumed.
How does a pipe work?
  1. The shell creates a temporary memory buffer to store the data flowing through the pipe.
  2. The standard output (STDOUT) of the first command is redirected to write to the pipe.
  3. The standard input (STDIN) of the second command is redirected to read from the pipe.
  4. The data is deleted from the pipe once it's read by the second command.
{% include "scenarios/info_box.html" with content='You can learn more about shell redirection operators in our guide.' %}
How can I create a pipe?

You can create a pipe by using the pipe operator | between two commands in a shell.

cat /etc/services | wc -l
cat usage

Description: cat concatenates files, but it's often used to print the content of a file.

Usage: cat [OPTIONS] FILE

Example: cat myfile.txt

This example prints the content of the file myfile.txt.

Common options:

  • -A: show all non-printable symbols: tabs, newlines, etc.
wc usage

Description: wc counts the number of lines, words and bytes in a file.

Usage: wc [OPTIONS] FILE

Example: wc -l input.txt

This example counts the number of lines in the file input.txt.

Common options:

  • -l: shows the number of lines.
  • -w: shows the number of words.
  • -c: shows the number of bytes.

In this example, the cat command will output the content of the file /etc/services to the pipe. The wc command will then read that data from the pipe and count the number of lines.

Can I chain multiple commands together using pipes?

Yes, you can chain multiple commands together using pipes to create a pipeline. Each pipe is independent and will send the output of the previous command to the input of the next command.

cat /etc/passwd | cut -d: -f1 | sort
cut usage

Description: cut prints field columns from a file.

Usage: cut [OPTIONS] FILE

Example: cut -d: -f1 /etc/passwd

This example prints the first field (delimited by :) of each line in the file /etc/passwd.

Common options:

  • -d DELIMITER: specify the field delimiter.
  • -f FIELDS: specify the fields to print (as a list of comma-separated field numbers).
sort usage

Description: sort orders the lines of a file (alphabetically by default).

Usage: sort [OPTIONS] FILE

Example: sort myfile.txt

This example sorts the lines of the file myfile.txt alphabetically.

Common options:

  • -r: reverse the order of the sort.
  • -n: sort numerically.

This example shows an alphabetically sorted list of the users in the system:

  1. The cat command will print the content of the file /etc/passwd to the first pipe. This file contains the list of users in the system.
  2. The cut command will then read the data from the first pipe and print the first field of each line (delimited by :) to the second pipe. The first field in each line is the name of the user.
  3. The sort command will then read the data from the second pipe, sort it alphabetically and print it to the terminal. This is the alphabetically-sorted list of usernames.
What happens if the command writing to the pipe terminates?

The command reading from the pipe will get an end-of-file (EOF) and will stop reading from the pipe.

What happens if the command reading from the pipe terminates?

The command writing to the pipe will receive a SIGPIPE signal.

You can learn more about SIGPIPE and other signals in our guide.
What happens with STDERR?

STDERR is not redirected to the pipe by default. You can redirect STDERR to the pipe using:

  • The pipe operator |&
    • ls ./this-directory-does-not-exist |& tr ' ' '\n'
  • The shell redirection operator 2> before the pipe operator |
    • ls ./this-directory-does-not-exist 2>&1 | tr ' ' '\n'

The example above translates all spaces to newlines from the error message sent to STDERR by the ls command.

Where can I find more info about pipes?

Man page:

man 7 pipe

External link: Wikipedia

Back to guides Feedback