Mercure: Real-Time APIs for Serverless and Beyond

Here is the slide deck I presented during API Days SF 2019:

Mercure is a protocol allowing to push data updates to web browsers and other HTTP clients in a convenient, fast, reliable and battery-efficient way. It is especially useful to publish real-time updates of resources served through web APIs, to reactive web and mobile apps. The protocol is designed for serverless, HTTP/2+, hypermedia and GraphQL, and is fully-featured: auto-discoverable, authorization, re-connection, state reconciliation…

Using Next.js and Material UI Together

Next.js is a convenient and powerful framework for React. Its main benefit over using React directly is its transparent support for Server-Side Rendering.
Material UI is a very popular set of React components implementing Google’s Material Design guidelines.

Both libraries are impressive, but there are some tricks to know to make them playing well together.

Bootstrapping

Setting up MUI in a Next project requires some non-trivial tweaks to Next’s initialization process. Conveniently, Material UI provides a skeleton containing a working Next.js project with Material UI already properly configured. It’s the easiest way to kickstart a new project using both tools, don’t miss it!

# Download the skeleton
$ curl https://codeload.github.com/mui-org/material-ui/tar.gz/master | tar -xz --strip=2  material-ui-master/examples/nextjs
$ cd nextjs
# Install the deps
$ yarn install
# Start the project
$ yarn dev

To learn how to integrate Material UI in an existing project, take a look to pages/_document.js and pages/_app.js they contain most the wiring logic.

Forms

Material UI is especially useful because of the large set of form components it provides. But handling forms with React (and so with Next) is tedious and verbose. My colleague Morgan Auchedé recently told me about Formik. Formik is a tiny yet super powerful library allowing to easily create forms with React. And good news: it plays very well with Next! Here is how a basic login form looks when using Formik:

import React from 'react';
import { Formik, Form, Field, ErrorMessage } from 'formik';

export default MyForm = () => (
    <Formik
      initialValues={{ email: '', password: '' }}
      validate={values => {
        // Your client-side validation logic
      }}
      onSubmit={(values, { setSubmitting }) => {
        // Call your API
      }}
    >
      {({ isSubmitting }) => (
        <Form>
          <Field type="email" name="email" />
          <ErrorMessage name="email" />
          <Field type="password" name="password" />
          <ErrorMessage name="password" />
          <button type="submit" disabled={isSubmitting}>
            Submit
          </button>
        </Form>
      )}
    </Formik>
);

Nice! However, when switching to Material UI inputs, the high level helper components provided by Formik become almost useless. Our forms are verbose again. Fortunately, a small library intuitively named formik-material-ui makes it easy to bridge both libraries! Here is the same form as before (including error handling), but rendered using Material UI components:

import React from 'react';
import { Formik, Form, Field } from 'formik';
import { TextField } from 'formik-material-ui';
import Button from "@material-ui/core/Button";

export default MyForm = () => (
    <Formik
      initialValues={{ email: '', password: '' }}
      validate={values => {
        // Your client-side validation logic
      }}
      onSubmit={(values, { setSubmitting }) => {
        // Call your API
      }}
    >
      {({ isSubmitting }) => (
        <Form>
          <Field type="email" name="email" component="TextField" />
          <Field type="password" name="password" component="TextField" />
          <Button
           type="submit"
           fullWidth
           variant="contained"
           color="primary"
           disabled={isSubmitting}
         >
           Submit
         </Button>
        </Form>
      )}
    </Formik>
);

This form is even less verbose, and is now looking good!

Buttons and Routing

Next.js comes with a nice routing system working transparently regardless if the app is executed client-side or server-side. It’s one of the biggest strength of the framework. However, in Material UI, the Button component is often used to trigger navigation between pages, and using buttons with the Router isn’t very intuitive. Still, it’s easy to do:

import React from "react";
import Link from "next/link";
import Button from "@material-ui/core/Button";

export default MyLink = () => (
  <Link href="/pricing" passHref>
    <Button component="a">Managed version</Button>
  </Link>
);

First, we set the component prop of Button to a. It tells Material UI to use an anchor for this button, instead of a… button by default. Then, we set the passHref prop of the Link element, it hints the Router to pass the href prop to the child component, even if doesn’t look like an anchor. Actually (because of the component prop we set earlier), the grandchild will be an anchor, and Material UI will forward the href prop to it! The rendered a element now has a proper href attribute, both client-side and in the server-side generated HTML. Good SEO, for free!

The same trick can be used with the Typography component:

import React from "react";
import Link from "next/link";
import Button from "@material-ui/core/Typography";

export default MyLink = () => (
  <Link href="/pricing" passHref>
    <Typography variant="caption" component="a">Managed version</Typography>
  </Link>
);

This time, we created a link looking like a caption!

That’s all for today. Have fun with Next and Material UI! For more tricks about JavaScript (among various other technologies), follow me on Twitter!

Upcoming conferences: AFUP Day, Web2Day, API Platform Meetup and more!

During the spring I’ll speak at several tech events about my projects Mercure (Go), API Platform (PHP, server-side and JS, client-side) and Symfony Panther (PHP/WebDriver):

AFUP Day, May 17th in Lille

Discover Symfony Panther, a brand new end-to-end testing tool using real web browsers for Symfony and PHP.

API Platform pre-Web2day meetup, June 3th in Nantes

Topic to be announced!

Web2Day, May 6th in Nantes

Mercure: UIs always synchronized with data in database

My company Les-Tilleuls.coop will have a booth during every of these events. Come, and let’s have a chat!
Also, I’ll announce my participation to some more events soon, stay tuned!

[SymfonyLive Paris slides] Symfony on steroids
: Vue.js, Mercure, Panther

Thanks to the new capabilities of the web platform (web components, Progressive Web Apps…) and the rise of modern JS libraries (Vue, React, Angular) almost all modern Symfony applications must leverage the frontend ecosystem.
Symfony 4 embed many gems that make it easy to integrate modern JavaScript within the framework, including the first component entirely written in JS: Webpack Encore.

In Symfony 4.2, another component that is super convenient for apps containing JS code has been released: Panther, a PHP library compatible with BrowserKit, that drives real web browsers to create end-to-end (E2E) tests with ease.
During this talk, I’ll show you how to cleanly integrate modern JavaScript code with Symfony and Twig and how to test such applications using Panther.

The examples will use VueJS, because it’s probably the easiest JS framework to get started with as a PHP developer, but all the tips and tricks will be applicable with other libraries such as React or Angular.

Finally, we’ll add some real time capabilities to our app using Mercure.rocks

Symfony and API Platform get “push” and real-time capabilities (Mercure protocol)

Mercure.rocks is a brand new protocol allowing to push data updates to web browsers and other HTTP clients in a convenient, fast, reliable and battery-efficient way. It is especially useful to publish real-time updates of resources served through web APIs, to reactive web and mobile apps.

Both Symfony and API Platform now have an official support for this protocol!

From the ground, Mercure has been designed to work with technologies not able to maintain persistent connections. It’s especially relevant in serverless environments, but is also convenient when using PHP or FastCGI scripts.

Mercure is basically a higher-level replacement for WebSocket. Unlike WebSocket, it is compatible with HTTP/2 and HTTP/3.
It has been designed with hypermedia APIs in mind, is auto-discoverable through the Web Linking RFC and is also compatible with GraphQL.
It natively supports authorization, reconnection in case of network issue (with refetching of missed events), subscribing to several topics, topics patterns (using templated URIs)…

Because it is built on top of Server-sent Events and plain old HTTP requests, it is already compatible with all modern browsers, and requires 0 client-side dependencies.

The protocol is open (available as an Internet Draft), and a reference open source implementation of the server written in Go is available.—

NPM dependency hell: comparison with Symfony, Laravel and API Platform

You may have noticed the recent fuss about the compromise of event-stream, a popular NPM package:

event-stream is a transitive dependency of many popular JavaScript projects including Vue, Angular, Gatsby and VSCode (some of them are using a version that isn’t affected by the attack).

This attack raised, again, the problem of the JS dependency cascade: when you install a major project, it comes with hundreds of tiny libraries, sometimes not maintained, and sometimes coming from untrusted sources.

Some claimed that the problem wasn’t specific to the JavaScript ecosystem, and that projects in other languages such as Rails and Symfony were also suffering from a similar dependency hell.
A Twitter poll created by Rafael Dohms highlights that most developers believe that it’s only a matter of luck if this issue has affected the JavaScript ecosystem, and not the PHP one:

Regarding Symfony, as a maintainer I have the feeling that the Symfony Core Team (carefully) adds dependencies only when strictly necessary. However I had no metrics to prove it. So I checked. Then I compared with other PHP frameworks I’m interested in: Laravel and API Platform.

Symfony

When installing Symfony 4.1 using the official skeleton, only 20 packages are downloaded (19 when excluding dev dependencies). 16 are directly maintained by the Symfony project, and the 4 others are interfaces from the PHP Framework Interoperability Group:

$ composer create-project symfony/skeleton nb-deps

$ composer info | wc -l

20

$ composer info | cut -d/ -f1 | uniq

psr

symfony

Even when installing the website skeleton, that comes with all features provided by the framework and third party (trusted) packages such as the Doctrine ORM, the Twig templating library or Swift Mailer, only 94 packages (75 when excluding dev dependencies), from 17 different organisations (14 without dev deps) are installed:

$ composer create-project symfony/website-skeleton nb-deps

$ composer info | wc -l

94

$ composer info | cut -d/ -f1 | uniq

doctrine

easycorp

egulias

facebook

fig
jdorn

monolog

nikic

ocramius

phpdocumentor

psr

sensio
swiftmailer

symfony

twig

webmozart

zendframework

Among these vendors, 7 are directly maintained by members (or former members) of the Symfony Core Team (easycorp, monolog, sensio, swiftmailer, symfony, twig, webmozart) and 2 are from the FIG. All these few libraries, except maybe jdorn/sql-formatter, are actively maintained, by prominent and well known members of the PHP community.

API Platform

A minimal installation of the server part (the one written in PHP) of the API Platform framework contains only 27 packages (26 without dev deps) from 5 vendors.

$ composer create-project symfony/skeleton nb-deps

$ composer require api-platform/core

$ composer info | wc -l

27

$ composer info | cut -d/ -f1 | uniq

api-platform

doctrine

psr

symfony

willdurand

When installing the API pack, that provides all features you can expect from an advanced web API (hypermedia support, automatic persistence with the Doctrine ORM support, automatic generation of human-readable documentation, CORS support, authorisation rules…), 57 packages (56 without dev deps) from 10 vendors are shipped.

$ composer create-project symfony/skeleton nb-deps

$ composer require api

$ composer info | wc -l

57

$ composer info | cut -d/ -f1 | uniq
api-platform

doctrine

jdorn

nelmio

phpdocumentor

psr

symfony

twig

webmozart

willdurand

On the other hand, the API Platform distribution comes with (optional) frontend tools: an admin and a Progressive Web Apps and mobile apps generator. They are built-on top of React, so even if the JavaScript libraries directly maintained by the API Platform team depend of just a few carefully selected libraries… these dependencies are typical JavaScript ones, and come with hundreds of dependencies (see after).

Laravel

The default installation of Laravel (that is somewhat similar in term of features to the Symfony website skeleton) comes with 72 packages (39 when excluding dev deps) from 35 vendors (21 when excluding dev deps).

$ laravel new nb-deps

$ composer info | wc -l
72

$ composer info | cut -d/ -f1 | uniq
dnoegel
doctrine
dragonmantank
egulias
erusev
fideloper
jakub-onderka
laravel
league
monolog
nesbot
nikic
opis
paragonie
psr
psy
ramsey
swiftmailer
symfony
tijsverkoyen
vlucas

The number of maintainers you have to trust when using Laravel is just a bit larger than when using Symfony, but again it’s still a bunch of people that are well known in the PHP community, and who may fit in a single room.

In comparison, a default installation of React using Create React App – that is more similar to the minimal Symfony skeleton than to fully-featured frameworks such as the Symfony website skeleton or Laravel – comes with 809 packages, most of them being maintained by different teams or individuals.

Another main difference is that most JS libraries are distributed as compiled and minified builds. Therefore it’s very difficult to guarantee that what is shipped and executed behaves exactly the same way than the code in the sources. In PHP, the source of the libraries are used directly, without intermediate obfuscation.  PHP builds are easy to reproduce, it helps a lot when auditing.

Of course, it doesn’t mean that major PHP frameworks are immune to this kind of attacks and – as any IT project – they also have their own security issues. However, the amount of third party code installed and the chain of trust you have to rely on is more under control than in the JS world.

By the way, you’d better actively monitor the security vulnerabilities that may affect your PHP projects.

Panther: test your Symfony apps with real web browsers

From a few lines of jQuery to modern React/Vue… PWA, Symfony apps always contain JavaScript code. Unfortunately, the SF functional test helper and the Goutte web scrapping lib aren’t able to execute JS code. It means that they cannot assert on client-side generated HTML, test if an element is visible or not, deal with alert boxes or detect if a bug occurs only with a specific browser.

Panther is a brand new e2e testing and web scrapping library written in PHP that drives real browsers thanks to the WebDriver protocol from the W3C. It implements the exact same API than the BrowserKit component. It means than existing SF tests and Goutte scripts work without modification… but this time JS will be executed! The tool also brings new capabilities including taking screenshots, or executing custom JS in the execution context of the web page. To install Panther you only have one command to run, and it doesn’t need any dependency to install Panther. Let’s meet the feline!

During the first presentation of this talk, Panther reached 1,000 stars on GitHub!
To celebrate, I unveiled a set of new features including some brand new database testing helpers inspired from Laravel.

API Platform 2.3: Major Perf Improvement, API evolution/deprecation, Better Dev Tools and Much More!

Today, the API Platform framework has reached 3k stars on GitHub, and it makes us very proud! To celebrate, I’ve just tagged the 2.3 version, that comes with a lot of amazing new features. Let’s discover them!

For newcomers, API Platform is a modern open source framework for API-driven projects. It allows, in just a few minutes, to expose hypermedia and GraphQL APIs. It also provides client-side tools leveraging the capabilities of auto-discoverable APIs: the admin interface and the React and Vue.js Progressive Web App generator. Finally, API Platform has been designed from the ground up as a Cloud Native solution that can run locally with the built-in Docker setup and be deployed instantly on a Kubernetes cluster (Heroku is also supported).

40% faster than the previous version!

We’re committed to continuously improving the performance of API Platform. In version 2.1, we’ve added an amazing invalidation-based cache mechanism. When enabled, HTTP responses are generated only one time then stored and served by a reverse caching proxy. When a resource is modified, all responses including or referencing it are automatically removed from the cache.

In version 2.3, Ben Davies has done an excellent job at profiling and patching the core component. Moreover, we’ve worked closely with the Symfony team to dramatically improve the performance of the Symfony Serializer component (one of the most important pieces of software used by API Platform).

With all these optimizations put together, in the scenario of a cache miss, API Platform-based apps are now more than 40% faster in version 2.3 compared to version 2.2 (Blackfire comparison):

A big thanks to Ben and to Nicolas Grekas from Blackfire.io for making it happen!

Support for API evolution (aka deprecating fields and resources)

A growing best practice is to use the evolution strategy for web APIs. Creating new versions of the API, of or its endpoints requires modifying all clients to upgrade, even the ones not impacted by the changes. On the other hand, this strategy (also known as versionless APIs) consists of deprecating the fields, resources types or operations that will be removed at some point. Most modern API formats including GraphQL, OpenAPI and Hydra are able to support this strategy.

In API Platform 2.3, we’ve introduced a new attribute to mark deprecated resource classes, operations and properties. All documentation formats generated by API Platform and having support for this feature will then automatically take it into account.

Here is how to deprecate an entire resource:

<?php

namespace App\Entity;

use ApiPlatform\Core\Annotation\ApiResource;

/**
 * @ApiResource(deprecationReason="Create a Book instead")
 */
class Parchment
{
    // ...
}

As you can see, to deprecate a resource, we just have to explain what the client should do to upgrade in a dedicated attribute. You can also use this new deprecationReason attribute on any operation.

The deprecation will automatically be taken into account by clients supporting the previously mentioned format. Here is how it renders for OpenAPI in the built-in Swagger UI shipped with the framework:

And now in the built-in version of GraphiQL (for GraphQL APIs):

It’s also possible to deprecate a single field:

<?php

namespace App\Entity;

use ApiPlatform\Core\Annotation\ApiProperty;
use ApiPlatform\Core\Annotation\ApiResource;

/**
 * @ApiResource
 */
class Review
{
    // ...

    /**
     * @ApiProperty(deprecationReason="Use the rating property instead")
     */
    public $letter;
}

All our client-side tools have been updated to ignore deprecated fields, operations and resources by default. The api-doc-parser library (which supports Hydra, and OpenAPI in the latest version) also support this new feature.

Dedicated Profiler Panel and Web Debug Toolbar Integration

Symfony comes with a nice set of development tools, including the Profiler, and it is compatible with API Platform! To install it, execute composer req profiler. Thanks to the contributions of Julien Deniau and Anthony Grassiot, the Web Debug Toolbar now displays an icon featuring our nice spider Webby that is linked to a new profiler panel dedicated to API Platform:

Shorter Attributes Syntax

Defining attributes using annotations can become verbose pretty quickly. As alternatives to annotations, for complex configurations API Platform also supports the XML and YAML formats. However, in version 2.3, Baptiste Meyer added a nicer and shorter syntax to define attributes on the @ApiResource and @ApiProperty annotations:

// Before
/**
 * @ApiResource(
 *     attributes={
 *         "validation_groups"={"bar"},
 *         "normalization_context"={"groups": {"book:read"}}
 *     }
 * )
 */
class Book
{
    // ...
}

// Now
/**
 * @ApiResource(
 *     validationGroups={"bar"},
 *     normalizationContext={"groups": {"book:read"}}
 * )
 */
class Book
{
    // ...
}

Of course the “old” syntax is still valid, but the new shortcuts allow a better discoverability and enable autocompletion in IDEs (if you are a PHPStorm user, install the PHP Annotations plugin to benefit from this new feature).

Revamped Admin

In version 2.3, a lot of love has been given to the API Platform Admin Component. This JavaScript library, maintained by Morgan Auchedé, dynamically constructs a UI for any API supporting Hydra or OpenAPI.

It uses React, and was built on top of the Admin On Rest library. But as you may know, Admin On Rest has been deprecated and Marmelab, the company behind the tool, has released a replacement called React Admin. React Admin comes with an updated fancy interface and it also fixes some design issues.

So we’ve patched API Platform Admin to use the new kid on the block, and we’ve taken this opportunity to also fix a well known annoying issue on our side: the Admin component wasn’t able to deal with embedded relations. This isn’t the case anymore. What’s even better, the Admin is now smart enough to reuse the data already downloaded as embedded relation instead of triggering a new HTTP request. It allows to dramatically improve the performance of the admin!

We’ve also added support for new features introduced by React Admin, including bulk delete.

Other Features

Of course, this new version also includes a lot of new features that are less visible but also very useful. Here is the curated list of some interesting changes:

  • Make resource class’s constructor parameters writable
  • Add support for interface as a resource
  • Throw an exception if a required filter isn’t set
  • Allow to specify the message when access is denied using the access_control_message attribute
  • Add a new option to include null results when using the date filter
  • Allow data persisters to return a new instance instead of mutating the existing one
  • Add a new attribute to configure specific formats per resources or operations
  • Add an --output option to the api:swagger:export command
  • Drop support for PHP 7.0
  • Upgrade Swagger UI and GraphiQL
  • GraphQL: Add a totalCount field in GraphQL paginated collections
  • JSONAPI: Allow inclusion of related resources

It is also worth mentioning that Teoh Han Hui has modernized the Docker setup we provide to leverage the new capabilities of this containerization technology.

As usual, thank you very much to all the developers who’ve contributed these new features, bug fixes, and the related documentation entries. You’re the most important part of the project: the community.

API Platform: A Framework for API-driven Projects (DevTalks Bucharest slides)

Here the slide deck I presented during DevTalks Bucharest 2018.

It covers the main features of the API Platform framework: we will install the framework, design an API data model as a set of tiny plain old PHP classes and learn how to get:

  • A fully featured dev environment with Symfony Flex and React containers, HTTP/2 and HTTPS support and a cache proxy
  • Pagination, data validation, access control, relation embedding, filters and error handling
  • Support for modern REST API formats: JSON-LD/Hydra, OpenAPI/Swagger, JSONAPI, HAL, JSON…
  • GraphQL support
  • An API responding in a just few milliseconds thanks to the builtin invalidation based cache mechanism
  • A dynamically created Material Design admin interface (a la Sonata / EasyAdmin – but 100% client-side) built with React
  • Client apps skeletons: React/Redux, React Native, Vue.js, Angular…

Finally, we’ll see ho to deploy the project in 1 command on Google Container Engine or any cloud with a Kubernetes.

This is an updated version of the talk I did during the SymfonyLive London 2017 conference, demonstrating the latest features of API Platform, and adapted to target a broader, non-Symfony/PHP audience.

Mastering the Symfony Serializer (PHP Tour slides)

The Symfony Serializer Component exists since the very beginning of Symfony 2. Years after years, it gained a lot of new features useful to transform various data formats to PHP structures and the opposite. It is also a foundation block of API Platform and a first-class citizen in FOSRest.

Let’s dive into this component. We will cover the basic concepts behind the serialization process: normalization, denormalization, encoding and decoding.

Then we will discover all the little known features of the component: JSON, XML and CSV support, file handling with “data:” URIs, serialization groups, attributes names conversion, trees support, updating existing objects, dealing with circular references.

Finally we will learn how to create custom normalizers and encoders; and how to modify the behavior of the builtin ones using composition.