D. J. Bernstein

Protocol design

So you want to design a protocol? I'll let you in on an evil secret: most implementors will never read your spec.

Instead, they'll look at some common examples and write the most obvious code that works for those examples.

Maybe you don't like this. Maybe you don't think implementors should be passing off prototypes as finished products. Tough luck. Do you want to complain, or do you want to design a protocol that works?

Here are a few case studies.

Lines terminated by CR LF

In many Internet protocols, the client sends a sequence of commands, each command terminated by CR LF.

What's the most obvious code to read a line terminated by CR LF? Depends on the programming environment. A typical UNIX programmer would use C's fgets() to read one line into a fixed-size buffer, since this is the most obvious way to handle common examples. Unfortunately, if the line is longer than the buffer size chosen arbitrarily by the programmer, or contains an intermediate LF before the CR LF, it will be split into two lines, destroying synchronization between client and server and sometimes corrupting data.

What happens next? Probably the programmer will replace the CR LF with NUL, by writing NUL at position strlen() - 2, and will then use C's string functions for parsing. Unfortunately, if the line contains NUL, it will be truncated; if the line started with an LF, the program may corrupt data or crash.

These reliability problems happen in real programs all the time. Yet CR LF continues to show up in new protocols. In Chris Newman's ``protocol design principles'' it's a requirement.

0-terminated lines: It's a much better idea to use NUL as a line terminator. This isn't supported by C's I/O library, so programmers will end up writing a few lines of code to read a NUL-terminated string, but that code will be much more reliable than the fgets() code described above. There won't be any more problems with LF and NUL. On the other hand, the programmer will probably still use a fixed-size buffer, splitting or truncating long lines (or perhaps crashing).

Counted lines: My best recommendation is netstrings, consisting of a decimal byte count followed by that number of bytes. Since the byte count is up front, there's a chance that the programmer will allocate the right amount of memory beforehand, handling strings of any length without trouble. He'll probably also use fread() or read(), so he'll accept arbitrary characters.

Case independence

Practically every piece of readable text in Internet protocols is case-independent. For example, SMTP servers must accept MAIL, mail, mAiL, etc.

In the real world, MAIL is usually sent in upper case. New clients use upper case since that's what everyone else is doing.

Consequently, many server implementors use strcmp(,"MAIL"). It's obvious code, and it handles the common examples. But it doesn't work with the occasional client that uses lowercase.

Case dependence: The problems caused by case independence are trivial to fix: require lowercase. Case independence is a complete waste of time. RFC 1958, section 4.3, is garbage.

In general, in the absence of overriding concerns such as efficiency, there should be only one way to encode each object.