Chapter 22. Process Substitution

Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple commands? This is where process substitution comes in.

Process substitution feeds the output of a process (or processes) into the stdin of another process.

Template

Command list enclosed within parentheses

>(command_list)

<(command_list)

Process substitution uses /dev/fd/<n> files to send the results of the process(es) within parentheses to another process. [1]

Caution

There is no space between the the "<" or ">" and the parentheses. Space there would give an error message.

 bash$ echo >(true)
 /dev/fd/63
 
 bash$ echo <(true)
 /dev/fd/63
 	      
Bash creates a pipe with two file descriptors, --fIn and fOut--. The stdin of true connects to fOut (dup2(fOut, 0)), then Bash passes a /dev/fd/fIn argument to echo. On systems lacking /dev/fd/<n> files, Bash may use temporary files. (Thanks, S.C.)

Process substitution can compare the output of two different commands, or even the output of different options to the same command.

 bash$ comm <(ls -l) <(ls -al)
 total 12
-rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
-rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
-rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh
        total 20
        drwxrwxrwx    2 bozo bozo     4096 Mar 10 18:10 .
        drwx------   72 bozo bozo     4096 Mar 10 17:58 ..
        -rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
        -rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
        -rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh

Using process substitution to compare the contents of two directories (to see which filenames are in one, but not the other):
   1 diff <(ls $first_directory) <(ls $second_directory)

Some other usages and uses of process substitution:

   1 read -a list < <( od -Ad -w24 -t u2 /dev/urandom )
   2 #  Read a list of random numbers from /dev/urandom,
   3 #+ process with "od"
   4 #+ and feed into stdin of "read" . . .
   5 
   6 #  From "insertion-sort.bash" example script.
   7 #  Courtesy of JuanJo Ciarlante.

   1 cat <(ls -l)
   2 # Same as     ls -l | cat
   3 
   4 sort -k 9 <(ls -l /bin) <(ls -l /usr/bin) <(ls -l /usr/X11R6/bin)
   5 # Lists all the files in the 3 main 'bin' directories, and sorts by filename.
   6 # Note that three (count 'em) distinct commands are fed to 'sort'.
   7 
   8  
   9 diff <(command1) <(command2)    # Gives difference in command output.
  10 
  11 tar cf >(bzip2 -c > file.tar.bz2) $directory_name
  12 # Calls "tar cf /dev/fd/?? $directory_name", and "bzip2 -c > file.tar.bz2".
  13 #
  14 # Because of the /dev/fd/<n> system feature,
  15 # the pipe between both commands does not need to be named.
  16 #
  17 # This can be emulated.
  18 #
  19 bzip2 -c < pipe > file.tar.bz2&
  20 tar cf pipe $directory_name
  21 rm pipe
  22 #        or
  23 exec 3>&1
  24 tar cf /dev/fd/4 $directory_name 4>&1 >&3 3>&- | bzip2 -c > file.tar.bz2 3>&-
  25 exec 3>&-
  26 
  27 
  28 # Thanks, Stphane Chazelas

A reader sent in the following interesting example of process substitution.

   1 # Script fragment taken from SuSE distribution:
   2 
   3 # --------------------------------------------------------------#
   4 while read  des what mask iface; do
   5 # Some commands ...
   6 done < <(route -n)  
   7 #    ^ ^  First < is redirection, second is process substitution.
   8 
   9 # To test it, let's make it do something.
  10 while read  des what mask iface; do
  11   echo $des $what $mask $iface
  12 done < <(route -n)  
  13 
  14 # Output:
  15 # Kernel IP routing table
  16 # Destination Gateway Genmask Flags Metric Ref Use Iface
  17 # 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
  18 # --------------------------------------------------------------#
  19 
  20 #  As Stphane Chazelas points out,
  21 #+ an easier-to-understand equivalent is:
  22 route -n |
  23   while read des what mask iface; do   # Variables set from output of pipe.
  24     echo $des $what $mask $iface
  25   done  #  This yields the same output as above.
  26         #  However, as Ulrich Gayer points out . . .
  27         #+ this simplified equivalent uses a subshell for the while loop,
  28         #+ and therefore the variables disappear when the pipe terminates.
  29 	
  30 # --------------------------------------------------------------#
  31 	
  32 #  However, Filip Moritz comments that there is a subtle difference
  33 #+ between the above two examples, as the following shows.
  34 
  35 (
  36 route -n | while read x; do ((y++)); done
  37 echo $y # $y is still unset
  38 
  39 while read x; do ((y++)); done < <(route -n)
  40 echo $y # $y has the number of lines of output of route -n
  41 )
  42 
  43 More generally spoken
  44 (
  45 : | x=x
  46 # seems to start a subshell like
  47 : | ( x=x )
  48 # while
  49 x=x < <(:)
  50 # does not
  51 )
  52 
  53 # This is useful, when parsing csv and the like.
  54 # That is, in effect, what the original SuSE code fragment does.

Notes

[1]

This has the same effect as a named pipe (temp file), and, in fact, named pipes were at one time used in process substitution.

    Place for this material granted by linux and technology portal www.net4me.net