Category / Programmation

by Kévin Dunglas

Webperf: PHP after Server Push

Here are the slide deck from the talk I gave this morning for AFUP Day 2021! And here is the video (in French):

Google recently announced that it will remove Server Push support from its flagship browser. Server Push is a technology that is part of the HTTP/2 and HTTP/3 standards. Server Push is designed to improve the performance of websites and web applications. Server Push is widely implemented in the PHP ecosystem: it is natively supported by most web servers as well as popular tools such as Symfony and API Platform. It is also the core of the Vulcan specification which allows creating very powerful and easy to cache web APIs.

The engineers at Google propose to use three technologies to replace Server Push, which they believe would be easier to use and implement and would allow performance gains almost similar to those of Server Push. These technologies are the Preload links, the HTTP return code “103 Early Hints” and the WebTransport JavaScript API. The PHP ecosystem and Vulcain already support the first two. The third one could eventually – if adopted by browsers and web servers – provide a modern (though low-level) alternative to WebSockets.

In this presentation, we will discover the use cases of each of these technologies, we will compare them with Server Push, and we will see how to use them in PHP (both client and server side). We will then see how Vulcain takes advantage of them.

by Kévin Dunglas

Symfony UX Turbo: Do You Still Need JavaScript?!

Hotwire Turbo is a tiny library recently introduced by DHH (the creator of Ruby on Rails) allowing to have the speed of Single-Page Apps without having to write any JavaScript!

As part of the Symfony UX initiative, I’m very excited to announce the immediate availability of Symfony UX Turbo: the official integration of Turbo in Symfony. With Symfony UX Turbo, you can get rid of JavaScript and enjoy using Twig again!

In this slide deck, we’ll discover how the library works, how to leverage it to enhance your Twig templates and how to add real-time features to your websites with the native support of the Mercure protocol!

This talk has been presented during the SymfonyWorld 2021 and SymfonyLive Paris 2021 conferences.

If you like this new approach, consider sponsoring me on GitHub and giving a star to the project!

Screen recordings:

by Kévin Dunglas

Using the “103 Early Hints” Status Code in Go Applications

103 is a new experimental HTTP status code defined in RFC 8297. It’s an informational status that can be sent by a server before the main HTTP response. Used in conjunction with the Link HTTP header and the preload relation, 103 gives the client the opportunity to fetch resources (assets, images, related API documents…) related to the explicitly requested one, as early as possible, and while the server is preparing the main response. Early Hints look like that:

HTTP/1.1 103 Early Hints
Link: </style.css>; rel=preload; as=style
Link: </script.js>; rel=preload; as=script

HTTP/1.1 200 OK
Date: Fri, 26 May 2017 10:02:11 GMT
Content-Length: 1234
Content-Type: text/html; charset=utf-8
Link: </style.css>; rel=preload; as=style
Link: </script.js>; rel=preload; as=script

[… rest of the response body is omitted from the example …]

Early Hints in the Wild

The 103 Early Hints status code could be a good alternative to HTTP/2 Server Push, which will be removed from Chrome and discouraged in the spec. It adds 1 RTT compared to Server Push, but – among other advantages – it allows better caching and is (theoretically) easier to implement. Chrome and Fastly are running an experiment to measure the potential benefit of this new status code. But for this experiment to be successful, we need compatible servers… and servers are waiting for compatible browsers before implementing this new status code. We’re in a typical “the chicken or the egg” situation.

Go and Early Hints

As you may know, I’m very interested in the topic of resource preloading applied to web APIs. I created the Vulcain protocol, which is an alternative to (some features of) GraphQL. It allows designing fast and idiomatic client-driven APIs, strictly following the REST architectural style. Vulcain (the protocol) supports Early Hints since day one, and has been designed with compatibility with this status code in mind. We will publish a new revision of Vulcain taking into account the twilight of Server Push soon, and I’ll present what this changes for the protocol in depth during AFUP Day 2021.

However, the Vulcain Gateway Server (the reference implementation), doesn’t support the new status code yet. So servers using this component cannot participate in the experiment.

As the Mercure.rocks hub, the Vulcain Gateway Server is now available as a module for the brilliant Caddy Web Server. This has been possible because both Caddy and the library implementing Vulcain are written in Go. Unfortunately, the standard library of Go doesn’t support the 103 status code yet. This prevents using Early Hints with Caddy and Vulcain.

To move forward, I submitted to the Go project patches implementing the RFC for HTTP/1.1 and for HTTP/2. They aren’t merged yet, but as Go 1.16 has been released yesterday, they may land soon in the development branch. In the meantime, it’s already possible to use this feature in your own Go programs, and it’s what we’ll see in the rest of this article!

The Go toolchain (especially the gc compiler) has an interesting characteristic: it creates statically-linked binaries by default. This means that once compiled with a development version of Go supporting the new feature, your standalone binaries can be deployed without requiring any change to your servers.

First be sure that the current stable version of Go is installed on your system. Because the Go toolchain itself is written in Go, we need Go to compile Go. It’s called the bootstrapping process (another instance of the “chicken or the egg” problem).

Then, clone my fork of Go and checkout the branch containing the required changes for HTTP/1.1:

git clone https://github.com/dunglas/go.git dunglas-go
cd dunglas-go
git checkout feat/http-103-status-code

This branch contains the patch for HTTP/1.1 but not for HTTP/2. The implementation of HTTP/2 of Go is stored in a separated module: x/net/http2. Before creating our custom build of Go, we need to retrieve the patched version of x/net/http2 and to bundle it in the standard library.

Start by replacing the x/net package by my fork of it, and update the vendored dependencies:

export GOROOT=$(pwd)
cd src/
go mod edit -replace="golang.org/x/net=github.com/dunglas/[email protected]"
go mod vendor
unset GOROOT

Then, install the bundle command, and use it to bundle the net/http module:

go get golang.org/x/tools/cmd/bundle
cd net/http/
$(go env GOPATH)/bin/bundle -o=h2_bundle.go -dst net/http -prefix=http2 -tags='!nethttpomithttp2' golang.org/x/net/http2

The file h2_bundle.go has been replaced by a bundle containing our patch!

Finally, go back in the src/ directory and build our 103-enabled Go version:

cd ../../
./make.bash

Our enhanced Go compiler is ready!

Sample Program

Implementing the example provided in the RFC is straightforward:

// main.go
package main

import (
    "io"
    "log"
    "net/http"
)

func main() {
    helloHandler := func(w http.ResponseWriter, req *http.Request) {
        w.Header().Add("Link", "</style.css>; rel=preload; as=style")
        w.Header().Add("Link", "</script.js>; rel=preload; as=script")

        w.WriteHeader(103)

        // do your heavy tasks such as DB or remote APIs calls here

        w.WriteHeader(200)

        io.WriteString(w, "<!doctype html>\n[... rest of the response body is omitted from the example ...]")
    }

    http.HandleFunc("/hello", helloHandler)
    log.Fatal(http.ListenAndServeTLS(":443", "cert.pem", "key.pem", nil))
}

To generate a locally-trusted certificate TLS certificate (necessary to use HTTP/2), I recommend the mkcert command:

mkcert -cert-file ./cert.pem -key-file ./key.pem localhost

Use the custom build of Go we created to compile the program, and start it:

/path/to/dunglas-go/bin/go build main.go
./main main

Alternatively, execute go run main.go to compile and start the program with just one command. Use curl to check if it works properly.

With HTTP/1.1:

curl -v --http1.1 https://localhost/hello

And with HTTP/2:

curl -v https://localhost/hello

You should see something like that:

[snip]
< HTTP/2 103
< link: </style.css>; rel=preload; as=style
< link: </script.js>; rel=preload; as=script
< HTTP/2 200
< link: </style.css>; rel=preload; as=style
< link: </script.js>; rel=preload; as=script
< content-type: text/html; charset=utf-8
< content-length: 79
< date: Sat, 13 Feb 2021 09:47:27 GMT
<
<!doctype html>
[snip]

Conditionally Sending Early Hints

Calling http.ResponseWriter.WriteHeader() several times will cause an error with the vanilla Go runtime. To be able to compile the same program with the stable compiler and with the patched one, you can wrap the call to http.ResponseWriter.WriteHeader(103) in a condition:

package main

import (
    "io"
    "log"
    "net/http"
    "runtime"
    "strings"
)

func main() {
    helloHandler := func(w http.ResponseWriter, req *http.Request) {
        w.Header().Add("Link", "</style.css>; rel=preload; as=style")
        w.Header().Add("Link", "</script.js>; rel=preload; as=script")

        if strings.HasPrefix(runtime.Version(), "devel") {
            // skip if compiled with a stable version of Go
            w.WriteHeader(103)
        }

        w.WriteHeader(200)

        io.WriteString(w, "<!doctype html>\n[... rest of the response body is omitted from the example ...]")
    }

    http.HandleFunc("/hello", helloHandler)

    log.Fatal(http.ListenAndServeTLS(":443", "cert.pem", "key.pem", nil))
}

103 Early Hints and HTTP/3

HTTP/3 is the upcoming version of the king of web protocols. It should become an RFC soon. HTTP/3 is already enabled by default in Safari and is available under a flag in Firefox and Chrome.

An experimental implementation of HTTP/3 for Go (which is used by Caddy and Vulcain) is available, but it doesn’t support the 103 status code either. So I also opened a Pull Request to add support for this status code to quic-go! To use the 103 status code with HTTP/3, try this patch.

And voilà! You’re ready to create programs supporting this new status code, and you can participate in this experiment to make the web faster and greener! If you do so, let me know on Twitter! The Vulcain Gateway Server will soon be updated to support the 103 status code using the approach we’ve seen in this article. And if you encounter any bug while playing with these patches, please report them!

If you liked this article, and want to contribute to my research work about web APIs, HTTP and Go, consider sponsoring me!

by Kévin Dunglas

Schema Generator 3: A Step Towards Redecentralizing the Web!

I just released API Platform Schema Generator version 3! It’s a command-line tool, part of the API Platform framework, that instantly generates a complete PHP data model (classes, enumerations, ORM mapping, validation rules, web API definitions, PHPDoc…) from RDF vocabularies and ontologies. RDF vocabularies are commonly used to define web-scale (Schema.org), industry-wide (The NASA Air Traffic Management Ontology, The Automotive Ontology, AddictO Vocab) or organization-wide (EU Vocabularies, Volkswagen Vehicles Ontology) concepts and relationships.

New Features

Vocabulary support in versions 1 and 2 of the tool was limited. Only Schema.org was 100% supported. Version 3 is now compatible with most existing vocabularies. To do so, we have implemented support for the Web Ontology Language (OWL) and for XML Schema types.

Also, the generated code now leverages the new features introduced in recent PHP versions, especially the improvements made to the type system: typed properties, nullable type declarations…

Towards a Decentralized Web

As you probably know, I’m very concerned about the centralization of the web:

The web is being weaponized by Tech Giants and governments. What used to be a free and autonomous space, an emancipation instrument where marginalized groups had a voice, is becoming a mass surveillance tool where censorship rules. We must act to redecentralize, just as the original was designed. Following Trump’s ban (which was entirely justified), even the creator of Twitter comes to this conclusion:

Among the various RDF vocabularies, one could be the foundation for this redecentralized web: ActivityStreams. It’s the vocabulary of ActivityPub, a W3C-backed social networking protocol used to build decentralized and federated social media. ActivityPub is already implemented by popular social media software including Mastodon, PeerTube, Mobilizon and WriteFreely. Servers implementing the ActivityPub protocol are interoperable and federated in what we call the Fediverse.

And here we’re back to API Platform Schema Generator v3: this new version has been created especially to bootstrap PHP applications using ActivityStreams! Generating a set of PHP classes implementing the vocabulary is as easy as cherry-picking the types and properties you want in a config file!

And that’s not all. I also created a new experimental component implementing the whole ActivityPub protocol: API Platform ActivityPub!

Thanks to this component, connecting a Symfony or an API Platform application (including existing ones) to the Fediverse is just a matter of installing a bundle! As all other API Platform components, API Platform ActivityPub can also be used as a standalone PHP library in projects not using Symfony.

PHP was the language of choice to build the original (at that time decentralized) web. Over the years, PHP dramatically improved, removing most footguns and becoming one of the fastest programming languages. But PHP is still easy to learn, and usable by hobbyists. I hope that it will become the language of choice to build the redecentralized web, and that API Platform will help to achieve this goal!

Links

Sponsor Me!

To allow me to work more on API Platform, my other free software projects and my experiments to redecentralize the web, consider sponsoring me! (Yes I know, GitHub is a proprietary service and belongs to a Big Tech, but we all have our contradictions, and I hope to be able to switch to radicle at some point.)

Mister Gotcha

API Platform 2.6
by Kévin Dunglas

API Platform 2.6: PHP 8 support, Next.js and Nuxt.js app generator, Caddy server, ActivityPub and much more!

During my talk at SymfonyWorld, I announced the immediate availability of API Platform 2.6! Try it now!

Version 2.6 is the result of more than one year of work and hundreds of commits. It contains a huge list of new features, checkout the change logs (core library, distribution) for the full list. In this blog post, I’ll present the major new features! You can also take a look at the slide deck I presented (go on SpeakerDeck for the transcript):

See also the French translation of this article.

PHP 8 Support

PHP 8 was released one week ago, and it comes with a huge list of new features. Among these features, attributes are a game changer for API Platform, as it relied a lot on annotations.

Starting with API Platform 2.6 you can use every annotation classes as PHP 8 attributes:

<?php

namespace App\Entity;

use ApiPlatform\Core\Annotation\ApiResource;

#[ApiResource]
class Book
{
    public function __construct(
        public string $id,
        public string $title
    ) {}
}

API Platform is now also fully compatible with PHP 8 (all deprecations have been fixed), and PHP 8 is now the default in the API Platform Distribution!

Next.js and Nuxt.js Support

API Platform comes with a very practical tool to scaffold client applications: the client generator. This generator uses the Hydra or OpenAPI (experimental) documentation of the API to generate apps using various client-side technologies. It works with any API documented using these formats, not only with APIs using the server-side component of API Platform!

Generating TypeScript interfaces as well as of Progressive Web Apps built with React Native, React/Redux, Vue.js, Quasar and Vuetify was already supported. In API Platform 2.6, this tool gained support for Next.js!

Next.js is the reference framework in the React ecosystem. It is very easy to use, supports Server-Side Rendering out of the box, has a simple yet powerful page-based routing system and supports a wide range of features.

We also updated the API Platform distribution to use Next.js by default. A Docker container and a Docker Compose definition will start a skeleton of a Next.js app automatically and you will be able to scaffold your pages and forms by running a single command!

API Platform Admin is now also integrated in this Next.js app, on the /admin route.

You prefer using Vue instead of React? We’ve got you covered! In API Platform 2.6 we also introduced a Nuxt.js generator. Nuxt.js is similar to Next.js, but using Vue! And, as API Platform, it has been created by French folks!

Simplifying the API Platform Distribution with Caddy Server

In API Platform 2.6, the distribution has been totally revamped. API Platform supports many advanced features out of the box, including the Mercure and Vulcain protocols. To support these features, API Platform came with many Docker containers:

  • PHP FPM
  • NGINX
  • A container to generate TLS certificates
  • Vulcain reverse proxy
  • Mercure Hub
  • Postgres
  • A Node.js container for the client generator
  • A Node.js container for the admin app

In API Platform 2.6, we simplified everything! First, we switched from NGINX to the awesome Caddy web server. Caddy is fast, fully-featured and extensible. For instance, it generates TLS certificates automatically both in production and on local development environment and natively supports PHP FPM. We also created Caddy modules for Mercure and for Vulcain! We’re also working with the Caddy team to create an HTTP cache module that will replace Varnish.

As explained in the previous section, we also replaced Create React App by Next.js and merged the containers for the client and the admin.

The result is a very simplified setup:

  • PHP FPM
  • Caddy web server
  • Postgres (optional)
  • Next.js container (optional)

We also added support for a production mode to the Docker Compose definition included in the distribution.

Better Configuration

Until now, it wasn’t possible to define global configuration defaults applied to all resources and all operations. You had to copy your configuration in all resource definitions. Starting with API Platform 2.6, you can define a global default configuration:

# config/packages/api_platform.yaml
api_platform:
    defaults:
        itemOperations: ["get"]
        collectionOperations: ["get"]
        mercure: true

With the previous config, write routes are disabled for the whole API, and Mercure support (async API) is enabled for all resources!

Access Control for Properties

API Platform supports resource-based access control. In 2.6, we introduced support for property-based access control:

<?php

namespace App\Entity;

use ApiPlatform\Core\Annotation\ApiResource;
use ApiPlatform\Core\Annotation\ApiProperty;

#[ApiResource]
class Book
{
    #[ApiProperty(security: "is_granted('ROLE_ADMIN')"]
    public $adminOnlyProperty; 
}

Thanks to this rule, the adminOnlyProperty will be serialized only for users having the admin role! Other users will not be able to see it.

Schema Generator v3

Read the dedicated article about this new version of API Platform Schema Generator.

ActivityPub Component

As you may know, I’m very worried by the current state of the web, and I think that we must fight to decentralize the web again.

ActivityPub is a specification edited by the W3C unlocking interoperability between social networking applications. It’s part of the solution to fight global censorship and to move out of the walled gardens of big tech companies. ActivityPub is already implemented by many software including Mastodon (Twitter-like), PeerTube (YouTube-like), PixelFed (Instagram-like), Mobilizon (Meetup-like)… This ecosystem is called the Fediverse.

In API Platform 2.6, we introduced a new experimental component allowing to add support for ActivityPub in any API Platform or Symfony application! It’s now way easier than before to build federated apps and to grow the Fediverse!

Community

The biggest strength of API Platform is its community of users and contributors! To celebrate our vibrant community, we added new pages to the website dedicated to the community. Each contributor now has a dedicated page including a summary of their contributions. Discover our awesome community!

And to thank contributors, we’re also offering nice t-shirts to them as part of the Hacktoberfest initiative!

API Platform 3

The next version of API Platform will be API Platform 3! We’ll release API Platform 2.7 at the same time. API Platform 3 will basically be API Platform 2.7 minus the backward compatibility layer with the deprecated code.

API Platform 3 will promote subresources as first-class citizens! They will support write operations. The overall configuration is also being revamped.

Also, to reduce the maintenance burden, API Platform 3 will only support PHP 8 and Symfony 5.2+! Be prepared.

Misc

They are also loads of new features in API Platform 2.6 not listed in this blog post: our GraphQL subsystem gained a lot of new features (including support for subscriptions and page-based pagination), the MongoDB bridge now supports the Mercure protocol, the OpenAPI builder has been totally rewritten, composed identifiers are easier to handle… Read the change logs (core library, distribution) to discover all of them!

I want to thank all contributors having contributed code and documentation for API Platform 2.6! Nothing would be possible without you.

Please try this new version and report any bugs!

If you have not yet given a GitHub star to the project, please do, it helps a lot to promote it.

If you like my work on the project, please also consider sponsoring me!

by Kévin Dunglas

The Mercure.rocks Hub is now based on Caddy Web Server

I’m very happy to announce the immediate availability of the Mercure.rocks Hub version 0.11! The Mercure.rocks Hub is a free software implementing the Mercure specification, an open protocol for fast, reliable and battery-efficient in-browser real-time communications.

Version 0.11 is a major milestone for the project!

As you may know, the Mercure.rocks Hub is written in Go. Until now, its built-in web server was home-made. Although this ad hoc server was doing its job, it was suffering from some limitations: few settings, no HTTP/3 support, not much debugging tools…

For this version 0.11, we worked hard to provide an easy-to-use standalone Go module, which can be used to add support for the Mercure protocol to any Go project.

This refactoring also allowed us to write a Mercure module for the popular Caddy web server! Then, we migrated the standalone Mercure.rocks Hub itself to a build of Caddy including the Mercure module.

Using Caddy instead of the home-made server adds a lot of new features to the Mercure.rocks Hub, and unlocks a broad range of new usages. For instance, it’s now possible to use the Mercure.rocks Hub as a production-grade reverse proxy for your website or API, that adds the Mercure well-known URL (/.well-known/mercure). So the Mercure well-known URL is on the same domain as your website, and you don’t need to deal with CORS anymore!

All features provided by Caddy are also supported by this custom build including but not limited to HTTP/3 and h2c support, advanced compression, detailed Prometheus metrics (with additional Mercure-specific metrics) or a built-in profiler (/debug/pprof/).

Check out the project’s website to discover all the features supported by Caddy web server.

It’s also possible to create custom Caddy builds including the Mercure module as well as other modules such as the Vulcain module for Caddy or the brand-new Caddy HTTP Cache module, which we co-maintain with the awesome Caddy team. Consequently, the Symfony Docker and the API Platform projects already migrated to Caddy with the Mercure module!

Before upgrading to version 0.11, be sure to migrate your configuration to the new Caddyfile format.

To ease the migration, we still provide binaries and Docker images including the legacy home-made server. These legacy builds are compatible with the old configuration format, and they are prefixed with “legacy-“.

A big thanks to Márk Sági-Kazár, Tamás Szigeti and to the Caddy team for their contributions and for their incredible help!

Please try Mercure 0.11, and report any problem!

by Kévin Dunglas

A Structured HTTP Fields Parser and Serializer for the Go Programming Language

“Structured Field Values for HTTP” is an upcoming RFC defining a set of well-defined data types to use in HTTP headers and trailers. This new format will improve the interoperability and the safety of HTTP by allowing to create generic parsers and serializers suitable for all HTTP headers (currently, most headers need a custom parser) and could also allow to improve the performance of the web. Mark Nottingham, one of the authors of this RFC, has published a very interesting article explaining these aspects in depth.

Headers and trailers using structured fields look like this:

Example-Item: token; param1=?0; param2="a-string"
Example-List: token, "string", ?1; parameter, (42, 42.0)
Example-Dict: foo1=bar, foo2="baz"; param, foo3=(?1 10.1)

Custom HTTP headers (and trailers) can start using structured values right now and most upcoming web standards including the new security headers proposed by the Chrome team are embracing them.

The next version of Vulcain, a popular proposal specification of mine relying on HTTP/2 (and HTTP/3) Server Push to allow creating fast and idiomatic client-driven web APIs, will also leverage Structured Fields Values!

Problem: the reference implementation of Vulcain is written in Go… but until now Go had no parser for Structured Field Values. So I wrote one!

Here comes httpsfv, a Structured Field Values parser and serializer for the Go programming language! The library implements entirely the latest version of the Internet-Draft (19), is fully documented, is tested with the official test suite (and many additional test cases), is quite fast (benchmark included in the repository) and is free as in beer, and as in speech (BSD-3-Clause License)!

As the Structured Field Values spec will become a RFC soon and is planned to be used for most new HTTP headers, I also opened a Pull Request on the Go repository to include this parser (and serializer) directly in the standard library of the language. If this PR is merged, it will be possible to deal with SFV without requiring any third-party package.

This PR also provides convenient helper methods to parse and serialize structured headers directly from Header instances:

h := http.Header{}

// Parsing
i := h.GetItem("Foo-Item")
l := h.GetList("Foo-List")
d := h.GetDictionary("Foo-Dictionary")

// Serializing
d := sfv.NewDictionary()
d.Add("b", sfv.NewItem(false))

bar := sfv.NewItem(sfv.Token("bar"))
bar.Params.Add("baz", 42)
d.Add("a", bar)

h.SetStructured("My-Header", d)

Examples and the documentation are available on the dedicated GitHub repository. Give it a try, and if you want to support this project, also give it a star!

by Kévin Dunglas

Save the Web: Decentralize!

The web was designed to serve humanity, to enable knowledge sharing, for friendship among people.
But data and services centralization by the Tech Giants have completely perverted it. Centralization causes major threats to our societies: mass surveillance, censorship, vote rigging, opponents arrests , advertising tracking.

The situation is such that the founding fathers of the web, including its creator Tim Berners-Lee, multiply the calls to save it.

The solution: decentralize it again, give users control over their data, fight against GAFAM monopolies, rebuild an independent and non-commercial web based on free software.

Developers, journalists, lawyers, artists and activists are organizing to bring out the “Dweb”, the decentralized and distributed web (again). The Dweb is a corpus of standards (usually published by the W3C) such as ActivityPub, JSON-LD, RDF or HTTP as well as tools including Solid, Mastodon, PeerTube or Mobilizon which aim to put the Web at the service of common good.

As web developers, it is our responsibility to build and promote this new web.

The good news is that the PHP ecosystem now has the tools to easily create decentralized web applications based on these standards. During this conference, we will discover why the Dweb is a crucial issue for the future of our societies, how it works technically, and how we implement it with API Platform!

To do so, I just published a new API Platform component: API Platform ActivityPub! This new libraries component allows:

  • Build apps supporting ActivityPub in minutes!
  • Add ActivityPub support to any existingAPI Platform or Symfony app
  • 100% customizable
  • Compatible with all API Platform features
  • State of the art PHP / API Platform

Check it out!

by Kévin Dunglas

Say Hello to Mercure 0.10!

I’m very excited to announce the immediate availability of the version 6 of the Mercure Internet Draft as well as of the version 0.10 of the reference implementation!

Mercure is a real-time protocol built on top of Server-sent Events and leveraging HTTP/2+. It allows to push messages to JavaScript webapps, mobile apps or IoT devices with ease. Client-side, it relies on the native EventSource class and doesn’t require any JS dependency. Server-side, it allows to use a Hub to handle persistent connections and dispatch messages sent, for instance, from your REST or GraphQL APIs. An open source Hub designed for performance and high loads is developed in parallel to the specification. Get started with Mercure.

Since the initial release of Mercure, we gather feedback from the ever-growing community of users and contributors. This allowed us to identify the pain points, some limitations and new use cases. It took months of work to get there, but these new versions of the protocol and of the FOSS hub (written in Go) fix all the limitations and design issues we identified so far. They are by far the best and the fastest versions ever released! Let’s discover these changes and new features.

Protocol Changes

Authorization Mechanism

One of the key features of Mercure is its authorization mechanism. It allows to send private updates, which will be received only by subscribers having the appropriate credentials.

Until the last version of the protocol, this mechanism was using the concept of targets, and was hard to understand for newcomers.
In the 6th version of the protocol, the authorization mechanism has been dramatically simplified, while staying as powerful as before.

The authorization mechanism has been totally revamped and the concept of targets is gone. Now, to send a private update, the publisher must only mark it as private when sending the POST request to the hub using the new private parameter.

On the other hand, to receive a private update, a subscriber must present a JWT to the hub containing a list of topic selectors in the mercure.subscribe claim of the token.

A private update will be received by a subscriber only if:

  • the topic of the update is contained in the list of topic selectors
  • the topic matches an URI template present in the list
  • the topic selectors list contains the special value *, which allows to receive all updates.

More advanced use cases such as cherry-picking which updates will be received by a subscriber are also supported thanks to existing features such as alternate topics.

Also, the new concept of target selectors can also be used to subscribe to (public or private) updates, making the whole protocol more consistent.

Learn more about topic selectors and the authorization mechanism.

Presence API

One of the most requested feature was the ability to retrieve the list of currently connected subscribers. This is now possible thanks to a new web API exposed by the hub! It’s also possible for a subscriber to receive Mercure updates when other subscribers connect or disconnect.

Hubs can now expose endpoints returning the list of connected users as JSON-LD documents:

  • /.well-known/subscriptions returns the list of all active subscriptions
  • /.well-known/subscriptions/{topic} returns the list of active subscriptions for a specific topic
  • /.well-known/subscriptions/{topic}/{subscriber} returns details for a single subscription

Also, after having fetched the initial list of connected users, clients can subscribe (using the standard subscription mechanism) to connection status updates by using these URIs as topics.

To access to this presence API, subscribers must be authenticated and have topic selectors matched by these URIs.

Finally, arbitrary data (such as the subscriber’s username or IP address) can be attached to a subscription and retrieved by other subscribers using the new mercure.payload JWT claim.

The debug UI shipped with the reference hub now supports this new feature:

Learn more about active Mercure subscriptions.

Event Sourcing

The native features of Mercure, including its built-in history and its state reconciliation mechanism make it a convenient solution to implement Event Sourcing.

The latest version of the protocol introduces changes making dedicated to this kind of usages.

First, it is now possible to set the value of the Last-Event-ID HTTP header (or of the query parameter with the same name) to earliest to retrieve the all past events. This allows to use the Mercure hub similarly to Apache Kafka, but directly from the browser (or any other kind of HTTP client).

Additionally, when a Last-Event-ID header or parameter is set, the hub will now always return a Last-Event-ID header in the HTTP response. If it matches the requested one, it means that all requested events have been returned, otherwise, it means that the requested event may have been deleted by the hub (if it stores a limited number of events for instance), or that this event ID doesn’t exist.

Learn more about Event Sourcing and state reconciliation capabilities of Mercure.

IRIs, Strings, and Fragment Identifiers

The protocol has been made more consistent by allowing to use IRIs (recommended) or plain strings (discouraged but sometimes practical) for every identifiers: topics, event IDs, subscriber IDs…

Event IDs starting by the # character (URI fragment) are now reserved for generation by the hub. This allows a hub to use offsets generated by systems such as Apache Kafka, Apache Pulsar or Redis Streams as update identifiers while keeping these identifiers globally unique (because they are relative to the Hub URI).

Changes in the Reference Implementation

All the features described previously have already been implemented in the reference implementation (the open source hub written in Go). But that’s not all! The hub has also been dramatically improved during the past few months.

New Engine

Until version 0.10, the reference hub was using an engine working like this: when a new update was posted by a publisher, a new light-weight process (goroutine) was started and the update was pushed to every active subscriber in a loop. If a Last-Event-ID header (or parameter) was passed by a subscriber to request past events, these past events were retrieved from the persistence layer and dispatched from another goroutine.

However, this approach had several drawbacks:

  • If a subscriber was slow to receive an update, it was slowing down the main loop, and so increasing the latency also for other subscribers.
  • If a Last-Event-ID was passed, new events could be received before events coming from the history (for example, when a new event was sent from the main goroutine while another goroutine was still retrieving past events from the persistence layer). The order in which events were received wasn’t 100% guaranteed.

Mercure 0.10 contains a brand new engine that fixes these issues! This new engine leverages thoroughly the iconic features of the Go programming language: channels and goroutines. Now, two buffers are assigned to each subscribers: the first one stores events coming from the history while the second one stores “live” events. When a new event is published, the main goroutine pushes this event in the “live” buffers without waiting for events to be dispatched. Then a dedicated goroutine per subscriber pops events from the buffers associated with it. So every subscriber can receive the events as its own pace, without impacting the rest of the system.

This buffer system also fixes the order issue: goroutines dedicated to specific subscribers wait for all updates coming from the history to be dispatched before starting to dispatch updates stored in the “live” buffer.

Last but not least, this new system allows to disconnect subscribers that are too slow to receive the updates. A new config option, dispatch_timeout, has been introduced to configure the maximum time that a subscriber can take to receive an update before being disconnected to free resources.

If you’re interested in the internals, take a look at this piece of Go code!

A big thanks to Dani Kaplan for his precious help testing and debugging this new engine under high loads!

Prometheus and Health Checks Support

Prometheus is a very popular monitoring system and time series database for Cloud Native applications. Jérémy Decool made a series of great contributions that add support for Prometheus to the Mercure hub. Metrics include (among other things) the number of currently connected subscribers, the number of dispatched updates, the number of served HTTP request segregated by status code, the memory usage, the current number of processes and the number of open file descriptors!

In addition, a new URL (/healthz) is now available to check if the hub is up and running without polluting the HTTP logs with useless entries. This particularly useful for Kubernetes’ liveness and readiness probes.

Version Flag

Jérémy Decool also contributed a new --version flag allowing to know what exact version of the Hub is being used. It looks obvious and priceless, but it was a missing feature. Thanks Jérémy for adding it!

Examples and Demos

We updated the documentation as well as all existing examples (in various programming languages) to be in sync with the latest version of the spec. I also added examples and demos illustrating new features, including a chat written in JavaScript and Python:

This demonstration chat is now available online, try it!

Community

Mercure is becoming more and more popular! It recently reached 2k stars on GitHub and integrations with existing ecosystems are now plenty. For this new version, we worked with the community to keep libraries and other Mercure-related projects in sync with the latest changes in the spec. Most of them are already compatible including (but not limited to):

For some others (for instance, the Java library and the Python library), Pull Requests are open. Finally, some tools still need to be updated including the awesome alternative Hub written in Node created by Nicolas Coutin and the Yii framework integration. Any help is very welcome!

To keep up with Mercure news, you can now follow our brand new Twitter account.

Finally, I’m very happy to announce that Les-Tilleuls.coop now provides official training for Mercure, in English and in French!

Improving Libraries Used by Mercure

One of the most powerful feature of Mercure is the ability to use URI templates to subscribe to topics matching patterns. Under the hood, the reference implementation of Mercure use the uritemplate Go library.

The performance of the Mercure hub highly depends of the performance of this library. So I worked with Kohei Yoshida (the author of this lib) to improve its performance, add benchmarks and fix some bugs! This also dramatically improved the performance of the hub (and of other projects using this library). Thank you Kohei for your quick reviews and releases!

The development of the new version of the hub also allowed me to find an issue in the Go standard library, and to open a Pull Request to fix it.

Hosted and High Availability Versions

Managed and On Premise High Availability (cluster) versions of the Mercure hub are available for a while now. They are already compatible with Mercure 0.10 and are now out of beta!

I want to thank very much our first customers, who trusted us, helped us to battle-test our hosting platform and gave us incomparable feedback allowing us to improve both the open protocol, the Managed and On Premise versions and the Open Source hub! They also allow us to invest more in the protocol’s development and in the Open Source version! So thank you again.

I also want to thank very much my GitHub sponsors, who allow me to work always more on my free software projects, including Mercure but also Vulcain, API Platform and the Symfony components I maintain.

Try and Star Mercure!

If you aren’t using Mercure yet, you can install a hub in a few seconds, or use the online demo, give it a try! If you like the project, show your support by starring it on GitHub!

by Kévin Dunglas

Vulcain: HTTP/2 Server Push
 and the rise of client-driven REST APIs

Over the years, several formats have been created to fix performance bottlenecks of web APIs: the n+1 problem, over fetching, under fetching…
The current hipster solution for these problems is to replace the conceptual model of HTTP (resource-oriented), by the one of GraphQL.

It’s a smart network hack for HTTP/1… But a hack that comes with (too) many drawbacks when it comes to HTTP cache, logs, security…
Fortunately, thanks to the new features introduced in HTTP/2 and HTTP/3, it’s now possible to create REST APIs fixing these problems with ease and class.

Vulcain is a brand new Internet Draft allowing to create fast, idiomatic and client-driven REST APIs.
To do so, it relies on the Server Push feature introduced by HTTP/2+ and on the hypermedia capabilities of the HTTP protocol.

Better, Vulcain comes with an open source reverse proxy that you can put on top of any existing web API to instantly turn it into a Vulcain-compatible one!

HATEOAS is back, and it’s for the best!