Using the “103 Early Hints” Status Code in Go Applications

103 is a new experimental HTTP status code defined in RFC 8297. It’s an informational status that can be sent by a server before the main HTTP response. Used in conjunction with the Link HTTP header and the preload relation, 103 gives the client the opportunity to fetch resources (assets, images, related API documents…) related to the explicitly requested one, as early as possible, and while the server is preparing the main response. Early Hints look like that:

HTTP/1.1 103 Early Hints
Link: </style.css>; rel=preload; as=style
Link: </script.js>; rel=preload; as=script

HTTP/1.1 200 OK
Date: Fri, 26 May 2017 10:02:11 GMT
Content-Length: 1234
Content-Type: text/html; charset=utf-8
Link: </style.css>; rel=preload; as=style
Link: </script.js>; rel=preload; as=script

[… rest of the response body is omitted from the example …]

Early Hints in the Wild

The 103 Early Hints status code could be a good alternative to HTTP/2 Server Push, which will be removed from Chrome and discouraged in the spec. It adds 1 RTT compared to Server Push, but – among other advantages – it allows better caching and is (theoretically) easier to implement. Chrome and Fastly are running an experiment to measure the potential benefit of this new status code. But for this experiment to be successful, we need compatible servers… and servers are waiting for compatible browsers before implementing this new status code. We’re in a typical “the chicken or the egg” situation.

Go and Early Hints

As you may know, I’m very interested in the topic of resource preloading applied to web APIs. I created the Vulcain protocol, which is an alternative to (some features of) GraphQL. It allows designing fast and idiomatic client-driven APIs, strictly following the REST architectural style. Vulcain (the protocol) supports Early Hints since day one, and has been designed with compatibility with this status code in mind. We will publish a new revision of Vulcain taking into account the twilight of Server Push soon, and I’ll present what this changes for the protocol in depth during AFUP Day 2021.

However, the Vulcain Gateway Server (the reference implementation), doesn’t support the new status code yet. So servers using this component cannot participate in the experiment.

As the Mercure.rocks hub, the Vulcain Gateway Server is now available as a module for the brilliant Caddy Web Server. This has been possible because both Caddy and the library implementing Vulcain are written in Go. Unfortunately, the standard library of Go doesn’t support the 103 status code yet. This prevents using Early Hints with Caddy and Vulcain.

To move forward, I submitted to the Go project patches implementing the RFC for HTTP/1.1 and for HTTP/2. They aren’t merged yet, but as Go 1.16 has been released yesterday, they may land soon in the development branch. In the meantime, it’s already possible to use this feature in your own Go programs, and it’s what we’ll see in the rest of this article!

The Go toolchain (especially the gc compiler) has an interesting characteristic: it creates statically-linked binaries by default. This means that once compiled with a development version of Go supporting the new feature, your standalone binaries can be deployed without requiring any change to your servers.

First be sure that the current stable version of Go is installed on your system. Because the Go toolchain itself is written in Go, we need Go to compile Go. It’s called the bootstrapping process (another instance of the “chicken or the egg” problem).

Then, clone my fork of Go and checkout the branch containing the required changes for HTTP/1.1:

git clone https://github.com/dunglas/go.git dunglas-go
cd dunglas-go
git checkout feat/http-103-status-code

This branch contains the patch for HTTP/1.1 but not for HTTP/2. The implementation of HTTP/2 of Go is stored in a separated module: x/net/http2. Before creating our custom build of Go, we need to retrieve the patched version of x/net/http2 and to bundle it in the standard library.

Start by replacing the x/net package by my fork of it, and update the vendored dependencies:

export GOROOT=$(pwd)
cd src/
go mod edit -replace="golang.org/x/net=github.com/dunglas/[email protected]"
go mod vendor
unset GOROOT

Then, install the bundle command, and use it to bundle the net/http module:

go get golang.org/x/tools/cmd/bundle
cd net/http/
$(go env GOPATH)/bin/bundle -o=h2_bundle.go -dst net/http -prefix=http2 -tags='!nethttpomithttp2' golang.org/x/net/http2

The file h2_bundle.go has been replaced by a bundle containing our patch!

Finally, go back in the src/ directory and build our 103-enabled Go version:

cd ../../
./make.bash

Our enhanced Go compiler is ready!

Sample Program

Implementing the example provided in the RFC is straightforward:

// main.go
package main

import (
    "io"
    "log"
    "net/http"
)

func main() {
    helloHandler := func(w http.ResponseWriter, req *http.Request) {
        w.Header().Add("Link", "</style.css>; rel=preload; as=style")
        w.Header().Add("Link", "</script.js>; rel=preload; as=script")

        w.WriteHeader(103)

        // do your heavy tasks such as DB or remote APIs calls here

        w.WriteHeader(200)

        io.WriteString(w, "<!doctype html>\n[... rest of the response body is omitted from the example ...]")
    }

    http.HandleFunc("/hello", helloHandler)
    log.Fatal(http.ListenAndServeTLS(":443", "cert.pem", "key.pem", nil))
}

To generate a locally-trusted certificate TLS certificate (necessary to use HTTP/2), I recommend the mkcert command:

mkcert -cert-file ./cert.pem -key-file ./key.pem localhost

Use the custom build of Go we created to compile the program, and start it:

/path/to/dunglas-go/bin/go build main.go
./main main

Alternatively, execute go run main.go to compile and start the program with just one command. Use curl to check if it works properly.

With HTTP/1.1:

curl -v --http1.1 https://localhost/hello

And with HTTP/2:

curl -v https://localhost/hello

You should see something like that:

[snip]
< HTTP/2 103
< link: </style.css>; rel=preload; as=style
< link: </script.js>; rel=preload; as=script
< HTTP/2 200
< link: </style.css>; rel=preload; as=style
< link: </script.js>; rel=preload; as=script
< content-type: text/html; charset=utf-8
< content-length: 79
< date: Sat, 13 Feb 2021 09:47:27 GMT
<
<!doctype html>
[snip]

Conditionally Sending Early Hints

Calling http.ResponseWriter.WriteHeader() several times will cause an error with the vanilla Go runtime. To be able to compile the same program with the stable compiler and with the patched one, you can wrap the call to http.ResponseWriter.WriteHeader(103) in a condition:

package main

import (
    "io"
    "log"
    "net/http"
    "runtime"
    "strings"
)

func main() {
    helloHandler := func(w http.ResponseWriter, req *http.Request) {
        w.Header().Add("Link", "</style.css>; rel=preload; as=style")
        w.Header().Add("Link", "</script.js>; rel=preload; as=script")

        if strings.HasPrefix(runtime.Version(), "devel") {
            // skip if compiled with a stable version of Go
            w.WriteHeader(103)
        }

        w.WriteHeader(200)

        io.WriteString(w, "<!doctype html>\n[... rest of the response body is omitted from the example ...]")
    }

    http.HandleFunc("/hello", helloHandler)

    log.Fatal(http.ListenAndServeTLS(":443", "cert.pem", "key.pem", nil))
}

103 Early Hints and HTTP/3

HTTP/3 is the upcoming version of the king of web protocols. It should become an RFC soon. HTTP/3 is already enabled by default in Safari and is available under a flag in Firefox and Chrome.

An experimental implementation of HTTP/3 for Go (which is used by Caddy and Vulcain) is available, but it doesn’t support the 103 status code either. So I also opened a Pull Request to add support for this status code to quic-go! To use the 103 status code with HTTP/3, try this patch.

And voilà! You’re ready to create programs supporting this new status code, and you can participate in this experiment to make the web faster and greener! If you do so, let me know on Twitter! The Vulcain Gateway Server will soon be updated to support the 103 status code using the approach we’ve seen in this article. And if you encounter any bug while playing with these patches, please report them!

If you liked this article, and want to contribute to my research work about web APIs, HTTP and Go, consider sponsoring me!

The Mercure.rocks Hub is now based on Caddy Web Server

I’m very happy to announce the immediate availability of the Mercure.rocks Hub version 0.11! The Mercure.rocks Hub is a free software implementing the Mercure specification, an open protocol for fast, reliable and battery-efficient in-browser real-time communications.

Version 0.11 is a major milestone for the project!

As you may know, the Mercure.rocks Hub is written in Go. Until now, its built-in web server was home-made. Although this ad hoc server was doing its job, it was suffering from some limitations: few settings, no HTTP/3 support, not much debugging tools…

For this version 0.11, we worked hard to provide an easy-to-use standalone Go module, which can be used to add support for the Mercure protocol to any Go project.

This refactoring also allowed us to write a Mercure module for the popular Caddy web server! Then, we migrated the standalone Mercure.rocks Hub itself to a build of Caddy including the Mercure module.

Using Caddy instead of the home-made server adds a lot of new features to the Mercure.rocks Hub, and unlocks a broad range of new usages. For instance, it’s now possible to use the Mercure.rocks Hub as a production-grade reverse proxy for your website or API, that adds the Mercure well-known URL (/.well-known/mercure). So the Mercure well-known URL is on the same domain as your website, and you don’t need to deal with CORS anymore!

All features provided by Caddy are also supported by this custom build including but not limited to HTTP/3 and h2c support, advanced compression, detailed Prometheus metrics (with additional Mercure-specific metrics) or a built-in profiler (/debug/pprof/).

Check out the project’s website to discover all the features supported by Caddy web server.

It’s also possible to create custom Caddy builds including the Mercure module as well as other modules such as the Vulcain module for Caddy or the brand-new Caddy HTTP Cache module, which we co-maintain with the awesome Caddy team. Consequently, the Symfony Docker and the API Platform projects already migrated to Caddy with the Mercure module!

Before upgrading to version 0.11, be sure to migrate your configuration to the new Caddyfile format.

To ease the migration, we still provide binaries and Docker images including the legacy home-made server. These legacy builds are compatible with the old configuration format, and they are prefixed with “legacy-“.

A big thanks to Márk Sági-Kazár, Tamás Szigeti and to the Caddy team for their contributions and for their incredible help!

Please try Mercure 0.11, and report any problem!

A Structured HTTP Fields Parser and Serializer for the Go Programming Language

“Structured Field Values for HTTP” is an upcoming RFC defining a set of well-defined data types to use in HTTP headers and trailers. This new format will improve the interoperability and the safety of HTTP by allowing to create generic parsers and serializers suitable for all HTTP headers (currently, most headers need a custom parser) and could also allow to improve the performance of the web. Mark Nottingham, one of the authors of this RFC, has published a very interesting article explaining these aspects in depth.

Headers and trailers using structured fields look like this:

Example-Item: token; param1=?0; param2="a-string"
Example-List: token, "string", ?1; parameter, (42, 42.0)
Example-Dict: foo1=bar, foo2="baz"; param, foo3=(?1 10.1)

Custom HTTP headers (and trailers) can start using structured values right now and most upcoming web standards including the new security headers proposed by the Chrome team are embracing them.

The next version of Vulcain, a popular proposal specification of mine relying on HTTP/2 (and HTTP/3) Server Push to allow creating fast and idiomatic client-driven web APIs, will also leverage Structured Fields Values!

Problem: the reference implementation of Vulcain is written in Go… but until now Go had no parser for Structured Field Values. So I wrote one!

Here comes httpsfv, a Structured Field Values parser and serializer for the Go programming language! The library implements entirely the latest version of the Internet-Draft (19), is fully documented, is tested with the official test suite (and many additional test cases), is quite fast (benchmark included in the repository) and is free as in beer, and as in speech (BSD-3-Clause License)!

As the Structured Field Values spec will become a RFC soon and is planned to be used for most new HTTP headers, I also opened a Pull Request on the Go repository to include this parser (and serializer) directly in the standard library of the language. If this PR is merged, it will be possible to deal with SFV without requiring any third-party package.

This PR also provides convenient helper methods to parse and serialize structured headers directly from Header instances:

h := http.Header{}

// Parsing
i := h.GetItem("Foo-Item")
l := h.GetList("Foo-List")
d := h.GetDictionary("Foo-Dictionary")

// Serializing
d := sfv.NewDictionary()
d.Add("b", sfv.NewItem(false))

bar := sfv.NewItem(sfv.Token("bar"))
bar.Params.Add("baz", 42)
d.Add("a", bar)

h.SetStructured("My-Header", d)

Examples and the documentation are available on the dedicated GitHub repository. Give it a try, and if you want to support this project, also give it a star!

Vulcain: HTTP/2 Server Push
 and the rise of client-driven REST APIs

Watch a longer video (English)!
Watch a French version.

Over the years, several formats have been created to fix performance bottlenecks of web APIs: the n+1 problem, over fetching, under fetching…
The current hipster solution for these problems is to replace the conceptual model of HTTP (resource-oriented), by the one of GraphQL.

It’s a smart network hack for HTTP/1… But a hack that comes with (too) many drawbacks when it comes to HTTP cache, logs, security…
Fortunately, thanks to the new features introduced in HTTP/2 and HTTP/3, it’s now possible to create REST APIs fixing these problems with ease and class.

Vulcain is a brand new Internet Draft allowing to create fast, idiomatic and client-driven REST APIs.
To do so, it relies on the Server Push feature introduced by HTTP/2+ and on the hypermedia capabilities of the HTTP protocol.

Better, Vulcain comes with an open-source reverse proxy that you can put on top of any existing web API to instantly turn it into a Vulcain-compatible one!

HATEOAS is back, and it’s for the best!

Stack2Slack: a Slack bot written in Go to monitor StackOverflow tags

Screenshot of Stack to Slack

At Les-Tilleuls.coop, we use Slack to centralize our communications and notifications. And, as the maintainers of the API Platform framework, we also do our best to help the community on StackOverflow when we have some free time. Until recently we were just checking periodically the StackOverflow website for new questions. But because all our notifications (GitHub, Travis, Twitter…) except the one from StackOverflow were centralized on Slack, it wasn’t optimal.

So I created a tiny Slack bot (just 150 LOC) to monitor StackOverflow (or any other site from the StackExchange galaxy) and automatically post new questions in dedicated Slack channels.

This bot is written in Go, the source code (MIT licensed) as well as binaries are available on GitHub. A Docker image is also available to easily run the daemon locally or on your servers.

To install the bot, start by registering a new Slack bot, the run the daemon. For instance, using Docker:

docker run -e DEBUG=1 -e SLACK_API_TOKEN=<your-API-token> -e TAG_TO_CHANNEL='{“stackoverflow-tag”: “slack-channel”}’ dunglas/stack2slack

You can monitor as many SO tags as you want, and map and choose in which Slack channel questions must be posted.

Last but not least, if you rely on Kubernetes to manage your servers (or, as we do, use Google Container Engine), you can install Stack to Slack in a single command thanks to the provided Helm chart:

  1. Clone the Git repository: git clone [email protected]:dunglas/stack2slack.git
  2. Install the provided chart: helm install  –set slackApiToken=<your-API-token>  –set tagToChannel.stackExchangeTag=slackChannel  ./chart/stack2slack

If you like this bot, give it a star on GitHub. If you’re looking for skilled Go developers, or bot and API experts, drop us a mail!