D. J. Bernstein
Data structures and program structures
Rebuilding target files when source files have changed

Rebuilding target files atomically

A typical Makefile, using programs such as gcc in the simplest way, ends up building each target file in blocks: the first part of the file is created, then the second part of the file is created, etc. Consequences:

Several programs (gcc, for example, and some versions of make) retroactively remove the target file if they notice anything going wrong. But there's no guarantee that they'll notice. In particular, if the power suddenly goes out, the target file won't be removed. Furthermore, retroactive removal doesn't do anything to solve the problem of a file being incorrect while it's being rebuilt.

There is, fortunately, a better way to rebuild files under UNIX. The new data is saved under a new filename, not touching the existing file. Once the new file is complete, the old file is atomically replaced with the new file.

To make this as easy as possible, redo starts each build script with stdout writing to a new file. If the script finishes successfully, redo atomically replaces the old file with the new file. For example, bar.do saying

     redo-ifchange foo
     tr x y < foo
is analogous to
     bar: foo
             tr x y < foo > bar
in Makefile, but it is actually as safe as
     bar: foo
             tr x y < foo > bar---redoing
             fsync bar---redoing
             mv bar---redoing bar
with bar updated atomically.

(There are two ways to deal with programs like gcc that insist on writing to a file instead of to stdout. The first is to specify the filename /dev/fd/1, which is supported by most UNIX kernels. The second is to specify the filename $3, which is provided by redo.)

Furthermore, redo (unlike make) doesn't use the newness of a target file as a reason to consider the target file up to date. The target file isn't considered up to date until redo finishes and (atomically) records what happened in .redo.

When redo is asked to create a file that it hasn't heard of before, it presumes that the file is a source file if it exists, or a target file otherwise. In the second case (new target), redo immediately saves this decision to disk, so that its subsequent creation of the target doesn't change the decision.