A Structured HTTP Fields Parser and Serializer for the Go Programming Language

“Structured Field Values for HTTP” is an upcoming RFC defining a set of well-defined data types to use in HTTP headers and trailers. This new format will improve the interoperability and the safety of HTTP by allowing to create generic parsers and serializers suitable for all HTTP headers (currently, most headers need a custom parser) and could also allow to improve the performance of the web. Mark Nottingham, one of the authors of this RFC, has published a very interesting article explaining these aspects in depth.

Headers and trailers using structured fields look like this:

Example-Item: token; param1=?0; param2="a-string"
Example-List: token, "string", ?1; parameter, (42, 42.0)
Example-Dict: foo1=bar, foo2="baz"; param, foo3=(?1 10.1)

Custom HTTP headers (and trailers) can start using structured values right now and most upcoming web standards including the new security headers proposed by the Chrome team are embracing them.

The next version of Vulcain, a popular proposal specification of mine relying on HTTP/2 (and HTTP/3) Server Push to allow creating fast and idiomatic client-driven web APIs, will also leverage Structured Fields Values!

Problem: the reference implementation of Vulcain is written in Go… but until now Go had no parser for Structured Field Values. So I wrote one!

Here comes httpsfv, a Structured Field Values parser and serializer for the Go programming language! The library implements entirely the latest version of the Internet-Draft (19), is fully documented, is tested with the official test suite (and many additional test cases), is quite fast (benchmark included in the repository) and is free as in beer, and as in speech (BSD-3-Clause License)!

As the Structured Field Values spec will become a RFC soon and is planned to be used for most new HTTP headers, I also opened a Pull Request on the Go repository to include this parser (and serializer) directly in the standard library of the language. If this PR is merged, it will be possible to deal with SFV without requiring any third-party package.

This PR also provides convenient helper methods to parse and serialize structured headers directly from Header instances:

h := http.Header{}

// Parsing
i := h.GetItem("Foo-Item")
l := h.GetList("Foo-List")
d := h.GetDictionary("Foo-Dictionary")

// Serializing
d := sfv.NewDictionary()
d.Add("b", sfv.NewItem(false))

bar := sfv.NewItem(sfv.Token("bar"))
bar.Params.Add("baz", 42)
d.Add("a", bar)

h.SetStructured("My-Header", d)

Examples and the documentation are available on the dedicated GitHub repository. Give it a try, and if you want to support this project, also give it a star!

Say Hello to Mercure 0.10!

I’m very excited to announce the immediate availability of the version 6 of the Mercure Internet Draft as well as of the version 0.10 of the reference implementation!

Mercure is a real-time protocol built on top of Server-sent Events and leveraging HTTP/2+. It allows to push messages to JavaScript webapps, mobile apps or IoT devices with ease. Client-side, it relies on the native EventSource class and doesn’t require any JS dependency. Server-side, it allows to use a Hub to handle persistent connections and dispatch messages sent, for instance, from your REST or GraphQL APIs. An open source Hub designed for performance and high loads is developed in parallel to the specification. Get started with Mercure.

Since the initial release of Mercure, we gather feedback from the ever-growing community of users and contributors. This allowed us to identify the pain points, some limitations and new use cases. It took months of work to get there, but these new versions of the protocol and of the FOSS hub (written in Go) fix all the limitations and design issues we identified so far. They are by far the best and the fastest versions ever released! Let’s discover these changes and new features.

Protocol Changes

Authorization Mechanism

One of the key features of Mercure is its authorization mechanism. It allows to send private updates, which will be received only by subscribers having the appropriate credentials.

Until the last version of the protocol, this mechanism was using the concept of targets, and was hard to understand for newcomers.
In the 6th version of the protocol, the authorization mechanism has been dramatically simplified, while staying as powerful as before.

The authorization mechanism has been totally revamped and the concept of targets is gone. Now, to send a private update, the publisher must only mark it as private when sending the POST request to the hub using the new private parameter.

On the other hand, to receive a private update, a subscriber must present a JWT to the hub containing a list of topic selectors in the mercure.subscribe claim of the token.

A private update will be received by a subscriber only if:

  • the topic of the update is contained in the list of topic selectors
  • the topic matches an URI template present in the list
  • the topic selectors list contains the special value *, which allows to receive all updates.

More advanced use cases such as cherry-picking which updates will be received by a subscriber are also supported thanks to existing features such as alternate topics.

Also, the new concept of target selectors can also be used to subscribe to (public or private) updates, making the whole protocol more consistent.

Learn more about topic selectors and the authorization mechanism.

Presence API

One of the most requested feature was the ability to retrieve the list of currently connected subscribers. This is now possible thanks to a new web API exposed by the hub! It’s also possible for a subscriber to receive Mercure updates when other subscribers connect or disconnect.

Hubs can now expose endpoints returning the list of connected users as JSON-LD documents:

  • /.well-known/subscriptions returns the list of all active subscriptions
  • /.well-known/subscriptions/{topic} returns the list of active subscriptions for a specific topic
  • /.well-known/subscriptions/{topic}/{subscriber} returns details for a single subscription

Also, after having fetched the initial list of connected users, clients can subscribe (using the standard subscription mechanism) to connection status updates by using these URIs as topics.

To access to this presence API, subscribers must be authenticated and have topic selectors matched by these URIs.

Finally, arbitrary data (such as the subscriber’s username or IP address) can be attached to a subscription and retrieved by other subscribers using the new mercure.payload JWT claim.

The debug UI shipped with the reference hub now supports this new feature:

Learn more about active Mercure subscriptions.

Event Sourcing

The native features of Mercure, including its built-in history and its state reconciliation mechanism make it a convenient solution to implement Event Sourcing.

The latest version of the protocol introduces changes making dedicated to this kind of usages.

First, it is now possible to set the value of the Last-Event-ID HTTP header (or of the query parameter with the same name) to earliest to retrieve the all past events. This allows to use the Mercure hub similarly to Apache Kafka, but directly from the browser (or any other kind of HTTP client).

Additionally, when a Last-Event-ID header or parameter is set, the hub will now always return a Last-Event-ID header in the HTTP response. If it matches the requested one, it means that all requested events have been returned, otherwise, it means that the requested event may have been deleted by the hub (if it stores a limited number of events for instance), or that this event ID doesn’t exist.

Learn more about Event Sourcing and state reconciliation capabilities of Mercure.

IRIs, Strings, and Fragment Identifiers

The protocol has been made more consistent by allowing to use IRIs (recommended) or plain strings (discouraged but sometimes practical) for every identifiers: topics, event IDs, subscriber IDs…

Event IDs starting by the # character (URI fragment) are now reserved for generation by the hub. This allows a hub to use offsets generated by systems such as Apache Kafka, Apache Pulsar or Redis Streams as update identifiers while keeping these identifiers globally unique (because they are relative to the Hub URI).

Changes in the Reference Implementation

All the features described previously have already been implemented in the reference implementation (the open source hub written in Go). But that’s not all! The hub has also been dramatically improved during the past few months.

New Engine

Until version 0.10, the reference hub was using an engine working like this: when a new update was posted by a publisher, a new light-weight process (goroutine) was started and the update was pushed to every active subscriber in a loop. If a Last-Event-ID header (or parameter) was passed by a subscriber to request past events, these past events were retrieved from the persistence layer and dispatched from another goroutine.

However, this approach had several drawbacks:

  • If a subscriber was slow to receive an update, it was slowing down the main loop, and so increasing the latency also for other subscribers.
  • If a Last-Event-ID was passed, new events could be received before events coming from the history (for example, when a new event was sent from the main goroutine while another goroutine was still retrieving past events from the persistence layer). The order in which events were received wasn’t 100% guaranteed.

Mercure 0.10 contains a brand new engine that fixes these issues! This new engine leverages thoroughly the iconic features of the Go programming language: channels and goroutines. Now, two buffers are assigned to each subscribers: the first one stores events coming from the history while the second one stores “live” events. When a new event is published, the main goroutine pushes this event in the “live” buffers without waiting for events to be dispatched. Then a dedicated goroutine per subscriber pops events from the buffers associated with it. So every subscriber can receive the events as its own pace, without impacting the rest of the system.

This buffer system also fixes the order issue: goroutines dedicated to specific subscribers wait for all updates coming from the history to be dispatched before starting to dispatch updates stored in the “live” buffer.

Last but not least, this new system allows to disconnect subscribers that are too slow to receive the updates. A new config option, dispatch_timeout, has been introduced to configure the maximum time that a subscriber can take to receive an update before being disconnected to free resources.

If you’re interested in the internals, take a look at this piece of Go code!

A big thanks to Dani Kaplan for his precious help testing and debugging this new engine under high loads!

Prometheus and Health Checks Support

Prometheus is a very popular monitoring system and time series database for Cloud Native applications. Jérémy Decool made a series of great contributions that add support for Prometheus to the Mercure hub. Metrics include (among other things) the number of currently connected subscribers, the number of dispatched updates, the number of served HTTP request segregated by status code, the memory usage, the current number of processes and the number of open file descriptors!

In addition, a new URL (/healthz) is now available to check if the hub is up and running without polluting the HTTP logs with useless entries. This particularly useful for Kubernetes’ liveness and readiness probes.

Version Flag

Jérémy Decool also contributed a new --version flag allowing to know what exact version of the Hub is being used. It looks obvious and priceless, but it was a missing feature. Thanks Jérémy for adding it!

Examples and Demos

We updated the documentation as well as all existing examples (in various programming languages) to be in sync with the latest version of the spec. I also added examples and demos illustrating new features, including a chat written in JavaScript and Python:

This demonstration chat is now available online, try it!

Community

Mercure is becoming more and more popular! It recently reached 2k stars on GitHub and integrations with existing ecosystems are now plenty. For this new version, we worked with the community to keep libraries and other Mercure-related projects in sync with the latest changes in the spec. Most of them are already compatible including (but not limited to):

For some others (for instance, the Java library and the Python library), Pull Requests are open. Finally, some tools still need to be updated including the awesome alternative Hub written in Node created by Nicolas Coutin and the Yii framework integration. Any help is very welcome!

To keep up with Mercure news, you can now follow our brand new Twitter account.

Finally, I’m very happy to announce that Les-Tilleuls.coop now provides official training for Mercure, in English and in French!

Improving Libraries Used by Mercure

One of the most powerful feature of Mercure is the ability to use URI templates to subscribe to topics matching patterns. Under the hood, the reference implementation of Mercure use the uritemplate Go library.

The performance of the Mercure hub highly depends of the performance of this library. So I worked with Kohei Yoshida (the author of this lib) to improve its performance, add benchmarks and fix some bugs! This also dramatically improved the performance of the hub (and of other projects using this library). Thank you Kohei for your quick reviews and releases!

The development of the new version of the hub also allowed me to find an issue in the Go standard library, and to open a Pull Request to fix it.

Hosted and High Availability Versions

Managed and On Premise High Availability (cluster) versions of the Mercure hub are available for a while now. They are already compatible with Mercure 0.10 and are now out of beta!

I want to thank very much our first customers, who trusted us, helped us to battle-test our hosting platform and gave us incomparable feedback allowing us to improve both the open protocol, the Managed and On Premise versions and the Open Source hub! They also allow us to invest more in the protocol’s development and in the Open Source version! So thank you again.

I also want to thank very much my GitHub sponsors, who allow me to work always more on my free software projects, including Mercure but also Vulcain, API Platform and the Symfony components I maintain.

Try and Star Mercure!

If you aren’t using Mercure yet, you can install a hub in a few seconds, or use the online demo, give it a try! If you like the project, show your support by starring it on GitHub!

Vulcain: HTTP/2 Server Push
 and the rise of client-driven REST APIs

Over the years, several formats have been created to fix performance bottlenecks of web APIs: the n+1 problem, over fetching, under fetching…
The current hipster solution for these problems is to replace the conceptual model of HTTP (resource-oriented), by the one of GraphQL.

It’s a smart network hack for HTTP/1… But a hack that comes with (too) many drawbacks when it comes to HTTP cache, logs, security…
Fortunately, thanks to the new features introduced in HTTP/2 and HTTP/3, it’s now possible to create REST APIs fixing these problems with ease and class.

Vulcain is a brand new Internet Draft allowing to create fast, idiomatic and client-driven REST APIs.
To do so, it relies on the Server Push feature introduced by HTTP/2+ and on the hypermedia capabilities of the HTTP protocol.

Better, Vulcain comes with an open source reverse proxy that you can put on top of any existing web API to instantly turn it into a Vulcain-compatible one!

HATEOAS is back, and it’s for the best!

Forum PHP 2019: Mercure – Real-Time for PHP Made Easy

Yesterday at Forum PHP 2019 I presented how easy it is to create real-time apps using PHP (among other languages) and the Mercure protocol.

I also introduced the shiny and new Mercure website (designed by Laury S.)!

A special thanks to Eric Comellas who jumped on stage to explain how iGraal uses Mercure on a large scale to serve simultaneously more than 100,000 of their customers!

Upcoming conferences: AFUP Day, Web2Day, API Platform Meetup and more!

During the spring I’ll speak at several tech events about my projects Mercure (Go), API Platform (PHP, server-side and JS, client-side) and Symfony Panther (PHP/WebDriver):

AFUP Day, May 17th in Lille

Discover Symfony Panther, a brand new end-to-end testing tool using real web browsers for Symfony and PHP.

API Platform pre-Web2day meetup, June 3th in Nantes

Topic to be announced!

Web2Day, May 6th in Nantes

Mercure: UIs always synchronized with data in database

My company Les-Tilleuls.coop will have a booth during every of these events. Come, and let’s have a chat!
Also, I’ll announce my participation to some more events soon, stay tuned!

Stack2Slack: a Slack bot written in Go to monitor StackOverflow tags

Screenshot of Stack to Slack

At Les-Tilleuls.coop, we use Slack to centralize our communications and notifications. And, as the maintainers of the API Platform framework, we also do our best to help the community on StackOverflow when we have some free time. Until recently we were just checking periodically the StackOverflow website for new questions. But because all our notifications (GitHub, Travis, Twitter…) except the one from StackOverflow were centralized on Slack, it wasn’t optimal.

So I created a tiny Slack bot (just 150 LOC) to monitor StackOverflow (or any other site from the StackExchange galaxy) and automatically post new questions in dedicated Slack channels.

This bot is written in Go, the source code (MIT licensed) as well as binaries are available on GitHub. A Docker image is also available to easily run the daemon locally or on your servers.

To install the bot, start by registering a new Slack bot, the run the daemon. For instance, using Docker:

docker run -e DEBUG=1 -e SLACK_API_TOKEN=<your-API-token> -e TAG_TO_CHANNEL='{“stackoverflow-tag”: “slack-channel”}’ dunglas/stack2slack

You can monitor as many SO tags as you want, and map and choose in which Slack channel questions must be posted.

Last but not least, if you rely on Kubernetes to manage your servers (or, as we do, use Google Container Engine), you can install Stack to Slack in a single command thanks to the provided Helm chart:

  1. Clone the Git repository: git clone [email protected]:dunglas/stack2slack.git
  2. Install the provided chart: helm install  –set slackApiToken=<your-API-token>  –set tagToChannel.stackExchangeTag=slackChannel  ./chart/stack2slack

If you like this bot, give it a star on GitHub. If you’re looking for skilled Go developers, or bot and API experts, drop us a mail!