The Network · Chapter 13
How do they speak?

Chapter 12 delivered one complete message when its channel recovered. A real transport can supply an ordered byte stream, but the application still needs to decide where its messages end and what they mean. Two machines holding an open stream are like two people on a phone line who have not agreed on a language: the line works and they still cannot talk. This chapter builds the agreement. It is the top of the network’s data climb, and it gives the bytes a meaning; one rung remains after it, privacy.

The agreement is a protocol, and the one built here has the text shape of HTTP/1.x, the protocol the web is made of. Three ideas carry it. A message has a fixed structure: a start line, some headers, then a body. The length of the body is written down, so the receiver knows where it ends inside a stream that never stops. And a server is then nothing more than a function from a request to a response.

13.1 The shape of a message

A request and a response are the same shape. Each begins with a start line, then a run of headers, which are simple name and value pairs, then a blank line, then the body. Only the start line differs: a request leads with a method and a path, a response with a status and a reason. Figure 13.2 lays the two side by side.

The blank line matters more than it looks. It marks where the head ends and the body begins. But it cannot mark where the body ends, because the body is raw bytes and may contain a blank line of its own. That is what Content-Length is for: it states the body’s length up front, so the receiver counts out exactly that many bytes and stops. Parsing a message is undoing the structure with that one number in hand.

Our parser receives one already-buffered message. It requires the blank-line separator, a nonnegative decimal length, and at least that many body bytes; otherwise it raises an error. Any bytes after the declared body are ignored. Reading from a connection until the whole message has arrived is a separate job, not implemented here.¹

Example 13-1. Reading a message off the wire

def _decode(message: bytes):
    """Parse one buffered message, requiring its separator and declared body.

    A nonnegative decimal Content-Length selects the body, which may contain the
    gap. Bytes after that body are ignored; stream buffering is the caller's job.
    """
    head, separator, rest = message.partition(_GAP)1
    if not separator:
        raise ValueError("message is missing the header/body separator")
    start_line, *header_lines = head.split(_CRLF)
    headers = {}
    for line in header_lines:
        name, _, value = line.decode("ascii").partition(": ")
        headers[name] = value
    declared = headers.pop("Content-Length", "")2
    if not declared.isascii() or not declared.isdecimal():
        raise ValueError("Content-Length must be a nonnegative decimal integer")
    length = int(declared)
    if len(rest) < length:
        raise ValueError("incomplete body: fewer bytes than Content-Length")
    return start_line.decode("ascii"), headers, rest[:length]3

• the blank line splits the head from the body; everything before it is the start line and headers
• Content-Length is removed from the headers and checked before use: it must contain decimal digits and fit within the buffered body
• the body is taken by that length, not by searching for a delimiter, so a body may safely contain the blank line itself

13.2 A server is a function

With requests and responses in hand, the server is almost anticlimactic. It is a function: a request goes in, a response comes out. Nothing is hidden in between. For this small protocol the function is a lookup. It keeps a table of paths and the bytes each one serves, and it answers a request by finding the path.

Example 13-2. The server: a request in, a response out

def serve(request: Request, routes: dict) -> Response:
    """The server: a function from a request to a response. Look the path up in
    the routes; answer 200 with its body, or 404 if there is no such path."""
    if request.path in routes:1
        return Response(200, "OK", {}, routes[request.path])2
    return Response(404, "Not Found", {},
                    b"no such path: " + request.path.encode("ascii"))

• the whole decision: is this path one the server knows?
• a known path is answered 200 OK with its bytes; an unknown one falls through to 404 Not Found, the status every reader of the web has met

A real web server adds methods beyond fetching, headers that negotiate format and caching, and handlers that compute a page rather than look one up. None of that changes the request/response idea. HTTP/1.x uses this textual envelope; HTTP/2 uses binary framing, so the wire representation is not universal. A handler can still be understood as a function from a request to a response.

13.3 The protocol, run

Running ftb.protocol sends a request through the protocol and reads the reply, then sends the very same request again over the lossy channel from Chapter 12.

the client sends a request:
    GET /hello HTTP/1.0
    Content-Length: 0

the server parses it, routes on the path, and replies:
    HTTP/1.0 200 OK
    Content-Length: 23
    
    Hello from the server.

the client reads: 200 OK, 23 bytes of body
a request for /missing is answered: 404 Not Found

the same request, carried over the transport layer's lossy channel:
  arrived intact: True
  same answer: 200 OK

The client encodes a request for /hello; the server parses it, looks the path up, and replies 200 OK with the page; the client reads the status and the body back. A request for a path the server does not have is answered 404. The last lines are the point of the whole network part: the same request, carried over a channel that drops and reorders segments, arrives identical and earns the same answer. The protocol does not know the channel is unreliable, because this successful transport call returned the complete request.

Figure 13.3 shows the composition this program actually executes. It encodes and parses request/response bytes, and its second request goes through the transport model. That model’s lossy channel stands in for the lower network; no routing, ARP, physical signaling, or link-framing code is called. The first round trip encodes and parses the response locally. The replay inspects the server’s response object directly; neither response goes through transport.

The interactive version needs the in-browser runtime.

Everything here crossed in the clear; the next chapter asks what a private conversation requires.

Exercises

• Add a new path to the server, and request it.
• Request a path the server does not have, and confirm the not-found reply.