The io library interface

Introduction to the io library

Similarly, a UNIX program provides information to external sinks through file descriptors.

The io (input/output) library manages file descriptors. It can

• create descriptors talking to disk files, other programs, etc.;
• receive (``read'') data from a descriptor, if the descriptor has data to provide;
• provide (``write'') data to a descriptor, if the descriptor has space for data;
• if no descriptors are ready for reading or writing, pause until the situation changes; and
• identify the descriptors that are ready for reading or writing.

For example, a program that wants to read a file from disk can use io_readfile() to create a descriptor for the file, io_tryread() repeatedly to receive information from the file, and then io_close() to eliminate the descriptor.

The following program uses the io library to copy standard input (descriptor 0) to standard output (descriptor 1), with a 65536-byte buffer in the middle:

     #include "io.h"
     
     char buf[65536];
     int64 readpos = 0;
     int64 writepos = 0;
     int flageof = 0;
     
     int main()
     {
       int64 r;
     
       if (!io_fd(0)) return 111;
       if (!io_fd(1)) return 111;
     
       for (;;) {
         if (flageof && writepos >= readpos) return 0;
     
         if (!flageof && readpos < sizeof buf) {
           r = io_tryread(0,buf + readpos,sizeof buf - readpos);
           if (r <= -2) return 111; /* read error other than EAGAIN */
           if (r == 0) flageof = 1;
           if (r > 0) readpos += r;
         }
     
         if (writepos < readpos) {
           r = io_trywrite(1,buf + writepos,readpos - writepos);
           if (r <= -2) return 111; /* write error other than EAGAIN */
           if (r > 0) {
             writepos += r;
             if (writepos == readpos) readpos = writepos = 0;
	     /* if writepos is big, might want to left-shift buffer here */
           }
         }
       }
     }

If descriptor 0 is unreadable (or there is no space in the buffer for new data) and descriptor 1 is unwritable (or the buffer is empty), the above program busy-loops: it calls io_tryread() and io_trywrite() repeatedly, making no progress until the situation changes. Here is a smarter program that instead goes to sleep until the situation changes, leaving the CPU free for other programs:

     #include "io.h"
     
     char buf[65536];
     int64 readpos = 0;
     int64 writepos = 0;
     int flageof = 0;
     
     int main()
     {
       int64 r;
     
       if (!io_fd(0)) return 111;
       if (!io_fd(1)) return 111;
     
       for (;;) {
         if (flageof && writepos >= readpos) return 0;
     
         if (!flageof && readpos < sizeof buf) io_wantread(0);
         if (writepos < readpos) io_wantwrite(1);
         io_wait();
         if (!flageof && readpos < sizeof buf) io_dontwantread(0);
         if (writepos < readpos) io_dontwantwrite(1);
     
         if (!flageof && readpos < sizeof buf) {
           r = io_tryread(0,buf + readpos,sizeof buf - readpos);
           if (r <= -2) return 111; /* read error other than EAGAIN */
           if (r == 0) flageof = 1;
           if (r > 0) readpos += r;
         }
     
         if (writepos < readpos) {
           r = io_trywrite(1,buf + writepos,readpos - writepos);
           if (r <= -2) return 111; /* write error other than EAGAIN */
           if (r > 0) {
             writepos += r;
             if (writepos == readpos) readpos = writepos = 0;
	     /* if writepos is big, might want to left-shift buffer here */
           }
         }
       }
     }

Creating descriptors

     #include "io.h"
     char s[];
     int64 d;
     io_readfile(&d,s);

If something goes wrong, io_readfile sets errno to indicate the error, and returns 0; it does not create a new descriptor, and it does not touch d.

     #include "io.h"
     char s[];
     int64 d;
     io_createfile(&d,s);

If something goes wrong, io_createfile sets errno to indicate the error, and returns 0; it does not create a new descriptor, and it does not touch d. (However, it may have truncated or created the file.)

     #include "io.h"
     int64 d[2];
     io_pipe(d);

A pipe is typically stored in an 8192-byte memory buffer; the exact number depends on the UNIX kernel. Bytes are written to the end of the buffer and read from the beginning of the buffer. Once a byte has been read, it is eliminated from the buffer, making space for another byte to be written; readers cannot ``rewind'' a pipe to read old data. Once 8192 bytes have been written to the buffer, the pipe will not be ready for further writing until some of the bytes have been read. Once all the bytes written have been read, the pipe will not be ready for further reading until more bytes are written.

io_pipe sets d[0] to the number of a new descriptor reading from the pipe, and sets d[1] to the number of a new descriptor writing to the pipe. It then returns 1 to indicate success. If something goes wrong, io_pipe returns 0, setting errno to indicate the error; in this case it frees any memory that it allocated for the new pipe, and it leaves d alone.

     #include "io.h"
     int64 d;
     io_close(d);

io_close has no return value; it always succeeds in deallocating the memory used for the descriptor. If d is not the number of a descriptor, io_close has no effect.

Reading data

     #include "io.h"
     int64 d;
     char buf[];
     int64 len;
     int64 result;
     result = io_tryread(d,buf,len);

• io_tryread returns an integer between 1 and len: This number of bytes was available for immediate reading; the bytes were read into the beginning of buf. Note that this number can be, and often is, smaller than len; you must not assume that io_tryread always succeeds in reading exactly len bytes.
• io_tryread returns 0: No bytes were read, because the descriptor is at end of file. For example, this descriptor has reached the end of a disk file, or is reading an empty pipe that has been closed by all writers.
• io_tryread returns -1, setting errno to EAGAIN: No bytes were read, because the descriptor is not ready. For example, the descriptor is reading an empty pipe that could still be written to.
• io_tryread returns -3, setting errno to something other than EAGAIN: No bytes were read, because the read attempt encountered a persistent error, such as a serious disk failure (EIO), an unreachable network (ENETUNREACH), or an invalid descriptor number (EBADF).

     #include "io.h"
     int64 d;
     char buf[];
     int64 len;
     int64 result;
     result = io_waitread(d,buf,len);

• io_waitread returns an integer between 1 and len: This number of bytes was available for reading; the bytes were read into the beginning of buf. Note that this number can be, and often is, smaller than len; you must not assume that io_waitread always succeeds in reading exactly len bytes.
• io_waitread returns 0: No bytes were read, because the descriptor is at end of file. For example, this descriptor has reached the end of a disk file, or is reading an empty pipe that has been closed by all writers.
• io_waitread returns -3, setting errno to something other than EAGAIN: No bytes were read, because the read attempt encountered a persistent error, such as a serious disk failure (EIO), an unreachable network (ENETUNREACH), or an invalid descriptor number (EBADF).

Writing data

     #include "io.h"
     int64 d;
     const char buf[];
     int64 len;
     int64 result;
     result = io_trywrite(d,buf,len);

• io_trywrite returns an integer between 1 and len: This number of bytes was immediately written from the beginning of buf. Note that this number can be, and often is, smaller than len; you must not assume that io_trywrite always succeeds in writing exactly len bytes.
• io_trywrite returns -1, setting errno to EAGAIN: No bytes were written, because the descriptor is not ready. For example, the descriptor is writing to a full pipe that could still be read.
• io_trywrite returns -3, setting errno to something other than EAGAIN: No bytes were written, because the write attempt encountered a persistent error, such as a serious disk failure (EIO), a pipe that can no longer be read (EPIPE), an unreachable network (ENETUNREACH), or an invalid descriptor number (EBADF).

Once upon a time, many UNIX programs neglected to check the success of their writes. They would often encounter EPIPE, and would blithely continue writing, rather than exiting with an appropriate exit code. The UNIX kernel developers decided to send a SIGPIPE signal, which terminates the process by default, along with returning EPIPE. This papers over the problem without fixing it: the same programs ignore other errors such as EIO. One hopes that the programs have been fixed by now; kernels nevertheless continue to generate the SIGPIPE signal. The first time io_trywrite or io_waitwrite is called, it arranges for SIGPIPE to be ignored. (Technically, for SIGPIPE to be caught by an empty signal handler, so this doesn't affect child processes.) Do not use SIGPIPE elsewhere in the program.

     #include "io.h"
     int64 d;
     const char buf[];
     int64 len;
     int64 result;
     result = io_waitwrite(d,buf,len);

• io_waitwrite returns len: All the bytes were written.
• io_waitwrite returns something between 0 and len-1, setting errno to something other than EAGAIN: After this number of bytes was written, no further bytes were written, because the write attempt encountered a persistent error, such as a serious disk failure (EIO), a pipe that can no longer be read (EPIPE), an unreachable network (ENETUNREACH), or an invalid descriptor number (EBADF).

Setting time limits on descriptors

     #include "io.h"
     int64 d;
     tai6464 t;
     io_timeout(d,t);

io_timeout(d,t) sets the timeout for descriptor d to t.

io_timeout has no return value; it always succeeds. (Space to store the timeout was already allocated as part of the descriptor.) It has no effect if d is not the number of a descriptor.

     #include "io.h"
     int64 d;
     char buf[];
     int64 len;
     int64 result;
     result = io_tryreadtimeout(d,buf,len);

• io_tryread returns -1 (the descriptor is not ready for reading), and
• the descriptor has a timeout, and
• the read attempt was after the descriptor's timeout,

     #include "io.h"
     int64 d;
     const char buf[];
     int64 len;
     int64 result;
     result = io_trywritetimeout(d,buf,len);

• io_trywrite returns -1 (the descriptor is not ready for writing), and
• the descriptor has a timeout, and
• the write attempt was after the descriptor's timeout,

Waiting for ready descriptors

     #include "io.h"
     int64 d;
     io_wantread(d);
     io_wantwrite(d);
     io_dontwantread(d);
     io_dontwantwrite(d);

Similar comments apply to io_wantwrite and io_dontwantwrite.

These functions have no return value; they always succeed. (Space to store the numbers was already allocated as part of the descriptor.) They have no effect if d is not the number of a descriptor.

You do not have to indicate interest in a descriptor before using the descriptor. The importance of io_wantread and io_wantwrite is their interaction with io_wait, io_waituntil, and io_check.

     #include "io.h"
     tai6464 t;
     io_wait();
     io_waituntil(t);
     io_check();

io_wait pays attention to timeouts: if a descriptor reaches its timeout, and the program is interested in reading or writing that descriptor, io_wait will return promptly.

Under some circumstances, io_wait will return even though no interesting descriptors are ready. Do not assume that a descriptor is ready merely because io_wait has returned.

io_wait is not interrupted by the delivery of a signal. Programs that expect interruption are unreliable: they will block if the same signal is delivered a moment before io_wait. The correct way to handle signals is with the self-pipe trick.

io_waituntil(t) is like io_wait() but does not wait (noticeably) past time t. io_check() is like io_wait() but does not wait at all; its importance is its interaction with io_canread and io_canwrite.

     #include "io.h"
     int64 d;
     d = io_canread();
     d = io_canwrite();

io_waituntil and io_check interact with io_canread in the same way that io_wait does. io_wait, io_waituntil, and io_check share one list of descriptors.

Do not assume that data can actually be read merely because io_canread has pointed to a descriptor. Data that was readable a moment ago could already have been read by another program. Even worse, the low-level UNIX routines used by io_wait could have failed to allocate memory; in this case, io_canread has no choice but to indicate that all descriptors are ready. You must check the results of io_tryread.

Similar comments apply to io_canwrite.

You do not need to use io_canread and io_canwrite. You can call io_wait, try io_tryread and io_trywrite for every descriptor, and repeat. However, if you have thousands of descriptors of which only a few are ready, most of the effort of trying descriptors will be wasted. It is faster to focus on the descriptors indicated by io_canread and io_canwrite.

(The io_wait implementation and underlying UNIX kernel routines might have their own speed problems with thousands of descriptors, but this problem can be solved without any change in programs that use io_wait.)

Using descriptors created in other ways

     #include "io.h"
     int64 d;
     io_fd(d);

• When a program starts, it typically ``inherits'' (is started with) UNIX descriptors numbered 0 (stdin: standard input), 1 (stdout: standard output), and 2 (stderr: standard error). More sophisticated UNIX programming often involves more startup descriptors. In contrast, the io library starts with an empty list. The startup descriptors should be passed to io_fd.
• Various low-level mechanisms outside the io library, such as the UNIX socket function, create UNIX descriptors that aren't on the io list. These descriptors should be passed to io_closeonexec, io_nonblock, and io_fd.

     #include "io.h"
     int64 d;
     io_nonblock(d);

Actually, current UNIX kernels do not support non-blocking descriptors; they support non-blocking open files. Furthermore, many programs will break if they encounter non-blocking mode. This means that you must not use io_nonblock for a descriptor inherited from another program.

io_nonblock has no return value; it always succeeds. If d is not the number of a UNIX descriptor, io_nonblock has no effect.

If io_fd is given a descriptor in blocking mode, io_tryread and io_trywrite go through the following contortions to avoid blocking:

1. Stop if poll says that the descriptor is not ready. Otherwise there's a good chance, but not a guarantee: even if poll says the descriptor is ready, the descriptor might not be ready a moment later. (Furthermore, poll can fail on some systems.)
2. Catch SIGALRM. SIGALRM must not be blocked, and must not be used elsewhere in the program.
3. Set an interval timer so that any blocking call will be interrupted by SIGALRM within 10 milliseconds. (Current UNIX kernels do not allow any shorter interval.) Of course, this may still mean a 10-millisecond delay.

Future versions of the io library may use kernel extensions to avoid these contortions.

     #include "io.h"
     int64 d;
     io_closeonexec(d);

io_closeonexec has no return value; it always succeeds. If d is not the number of a UNIX descriptor, io_closeonexec has no effect.