Pipes, Pipes and More Pipes

No this post is not about plumbing or smoking. This is about the pipes used in GNU/Linux and other operating systems.

A Couple of Notes:

This article is about using redirection, pipes and named pipes in a command shell on a system running GNU/Linux. If you are not interested in command line use, or if you are not interested in using GNU/Linux, please do not bother to make comments about your lack of interest in such things. Comments about how much better other operating systems are, or how outdated command line mode is, will not be posted. I am familiar with, and use, many operating systems. This will not become a forum for arguing the benefits of other operating system. If you want to post a comment that describes a similar method of piping input or output for other operating systems, feel free. Keep it civil and on topic.

Many of the everyday electronic items we now use are running some form of open source software. This allows the system to be modified as desired, or needed, by the user. Most often I find that these items are running a version of GNU/Linux. Anytime I get a new electronic device, I check to see what operating system is running, and what access methods are available

Back to the Pipes:

There are three primary methods to change the standard input, named stdin, and standard output, named stdout, that are commonly used in GNU/Linux operating systems. These are redirection, pipes and named pipes. Is redirection a pipe or is a pipe redirection? In most articles, they consider a pipe to be redirection. I believe that redirection using the < or > symbols is a one way pipe. You are piping the input or output of the command to something other than stdin or stdout. However, in most cases, you will see the use of < or > referred to as redirection and the use of | or a named pipe FIFO referred to as a pipe. I follow that convention most of the time. However, I call redirection a pipe more often than I call a pipe redirection.

Many text mode commands have default output and input devices. Often the default output device is stdout and this is usually directed to the screen, or a window on the screen. The default input device is usually stdin and is most often the keyboard. Both of these, as well as another device, called standard error or stderr, can be redirected to other devices including files or printers.

In our example below, the file /tmp/pipe1, is a named pipe FIFO. Output to it is from the Cat command using a redirection with the > symbol. The default output device for the cat command is stdout so our use of the redirection simply changes the output device to the file, which happens to be a FIFO. This redirection is the first pipe in the example command line.

The second pipe used in the example below is referred to as, simply enough, a pipe and is designated by the vertical bar symbol. The normal function of a pipe is to connect the stdout of the command to the left, to the stdin of the command to the right. It also causes both commands to run at the same time. However, not all commands, by default, accept their input from stdin or send their output to stdout. Ftpput, as used in the example, is such a command. Ftpput does not have a default input device, it must be given an input file or device. More on that in a minute. We still want the Ftpput command to run at the same time as the cat command. The pipe is being used to make that happen.

As mentioned above, not all programs, or commands, have a default input or output. Some, such as ftpput, must be given a file or device to get input from or send output to. In some cases stdin and stdout can still be used. For example, the ftpput command will accept input from stdin two different ways. First, the - symbol can be used in place of the input file name. The - symbol is used by many programs to represent stdin or stdout, depending on where it is used in the command line. In the example, we could use this command line instead of the one given:

#cat (20130908_1916_R*)|ftpput -v -u(user name) -p(password) (192.168.1.203) ("/Home Public/Shows/Actual Show Name and Episode.mpg") -

And it will work fine. In this case, the output from the cat command is being allowed to go to stdout and the input to ftpput is from stdin. The pipe is creating the link to redirect stdout and stdin. In other cases, stdout can be given as a named file with /dev/stdout and stdin can be given as a named file with /dev/stdin. So the command line:

#cat (20130908_1916_R*)>/dev/stdout|ftpput -v -u(user name) -p(password) (192.168.1.203) ("/Home Public/Shows/Actual Show Name and Episode.mpg") /dev/stdin

Will also work.

With either of these alternate command lines, the shell simply creates a FIFO in a buffer and uses it in the same manner as a named pipe FIFO is used.

The third pipe in the example is the FIFO, /tmp/pipe1, which is known as a named pipe. This is a special type of file. It is shown in the directory listing with a p, for pipe, in the first position of the attribute field, the same place that shows a d for a sub-directory. On some operating systems, the directory listing will have the vertical bar symbol at the end of the file name to indicate that the file is a pipe. Using this file, all data that must, or can, be written to or read from a named file may be piped from one command to another command. This type of pipe has some other unique properties besides the name. The file really only exists in the physical realm as a memory buffer and a name in a directory. A directory listing will always show the file as having zero bytes. This is because the file does not accept input from the command doing the writing, until there is a command running that is reading the file. Each byte written to the file is expected to be read right away by another process. The FIFO will cause blocking on one side if the other side is not running. So the cat command will wait for the ftpput command to start looking for data before it starts writing that data.

You may wonder why a named pipe would be used if the same thing can be accomplished using one of the alternate methods of redirection of stdin and stdout. That brings us to another unique property of named pipes. The process, or command, that is writing to the FIFO and the process that is reading from the FIFO do not have to be running in the same shell, or even on the same machine. The example could be broken up so that the cat command is running in one shell terminal and the ftpput command is running in another shell terminal. And the terminals do not have to be on the same machine as long as both have access to the named pipe file and the operating system on both machines recognize that type of file.

Named pipes are persistent so they remain after the processes that are using them end. So the /tmp/pipe1 file in the example remains even after the Cat and Ftpput commands complete and the Telnet session is closed. This is another feature that can be used by programs. it is possible to have a command watching the FIFO that will perform a function when the FIFO is opened by another program or command for data to be written. For example, I have a script that runs all the time on one of my NAS units. It watches a FIFO and sends the data it receives to a file whose name increments each time the command runs. That program receives data from an automatic backup in a program that could only be given a single fixed name for the backup file. Before I started using the FIFO, I had to run a script that renamed all the prior backup files so the fixed name could be used without overwriting the past backup.

There are some other interesting features provided by named pipes. For example, you can read and write to them from multiple programs at the same time. That allows use of things like the Tee command which would allow writing the output to the screen and a file, or to multiple files, at the same time.

This is not intended to be a full Howto for redirection and pipes. I strongly suggest you explore some of the excellent information available on the Internet to learn more about them and how to use them in scripts and programs. This was just an intro to show how I used all three types to accomplish the goal for this case.

The Details and Example:

A little background on what I needed to do and how I did it for this case. Note that I do not use this command line anymore. I have written a shell script that looks for new files on the FAT32 partition and does the cat and ftpput automatically when a new file appears using a database that I created that contains the date and time the file was copied to the partition and the name of the show as I want it stored on the NAS drive.

One of my recent purchases was a new, old stock, DVR device. It was made in 2008 and connects to a network using a wired or wireless Ethernet connection. I found it was capable of a Telnet connection and was running a Linux Venus operating system, a common system for media devices.

This DVR has the capability to copy recorded programs, which are on a UDF file system, to a hard drive. However, it will only copy to a FAT32 partition and it breaks the program up into multiple files, each 2GB, or less. The files are broken up using a Split command, so to use them on other devices they must be put back together. I wanted to save these recordings on network attached storage units, which are formatted as NTFS or EXT3, so they could be viewed from other attached media devices on the network Both of those file systems handle files much larger than the combined file size of any recording. So I just needed to combine the smaller files into one file on the NAS unit.

The other problem was that the DVR gave the files names based on the date and time they were copied to the FAT32 partition. For example, a 3 hour show would have a name, on the FAT32 drive, like 20130908_1917_Rnn.mpg. The Rnn would be R00, R01 and R02 for the three files the program was split into. I wanted the actual name of the show to be the final name on the NAS.

Checking the available commands on the DVR through a Telnet connection, showed that the only method to get a file to a NAS drive was using a Ftpput command. There was no Mount command that could be used to mount a remote share and there was no standard FTP command that would allow wildcard copies. So the copy to the NAS would need to be a single file at a time. I would then need to combine the files on the NAS drive and rename the resulting file. Without using redirection and pipes, this would require dealing with each bit of data at least three times on the network.

Here was the solution I decided to use instead:

I opened a Telnet session to the DVR. After logging in and changing to the directory where the DVR saved the copied files, I issued the following command in the shell.

Note: This is the example referred to above;

#mknod /tmp/pipe1 p;cat (20130908_1916_R)*>/tmp/pipe1|ftpput -v -u(user name) -p(password) (192.168.1.203) ("/Home Public/Shows/Actual Show Name and Episode.mpg") /tmp/pipe1

The ( and ) symbols in the command line shown are not typed into the actual command line. They are used here to indicate items that would need to be changed to match the actual items on your system. For example, your user name and password on the NAS unit you are copying to. However, the " symbols were needed since the path and file name have spaces in them..

The Mknod command was used to create the named pipe FIFO file /tmp/pipe1. On newer GNU/Linux systems a Mkfifo command is frequently available for this. This system did not offer that command. The p after the file name tells Mknod that the file is going to be a named pipe FIFO. As noted earlier, this file is persistent, so that command does not really have to be run each time. However if the file exists, the command simply presents a message to that effect and the shell continues with the next command.

For those who are not familiar with this, you will notice a semicolon after the Mknod command. The semicolon tells the shell to run the Mknod command and, when it finishes, to run the Cat command. This is in contrast to the vertical bar symbol used between the Cat command and the Ftpput command which tells the shell to run both commands at the same time.

Next, a Cat command is used to concatenate the multiple files into a single file using redirection to the FIFO /tmp/pipe1. The Cat command simply reads the file(s), in order, and creates an output which is normally directed to stdout. Redirection of stdout to a file is done using the > symbol. It can also be redirected using a standard pipe with the | symbol. For example "cat sometext.txt|more" would cause Cat to read the file sometext.txt and send it to the More command so it is displayed one page at a time in the console.

The Cat command and the Ftpput command are tied together with a standard pipe so they will run at the same time. Doing this causes the Cat command to open the FIFO for writing and the Ftpput command to open the FIFO for reading at the same time. As data is written to the FIFO by Cat, it is read by Ftpput and sent to the remote file system as a single file. When the Cat command completes, it closes the FIFO for writing. The Ftpput command will see that the FIFO has been closed and, once all data has been processed, will close the file for reading and exit.