Symfony Live 2015 : Construire des applications API-centric avec Symfony

Symfony Live 2015

Je donnerai une conférence de 40 minutes au Symfony Live 2015 qui se déroulera le 9 avril à la Cité Internationale Universitaire de Paris.

J’y parlerai de la conception et du développement d’applications construites autour d’une API REST :

Au cours de ce talk, je présenterai une architecture moderne qui permet de construire des applications performantes, évolutives et interopérables :

Le talk sera didactique et accessible aux développeurs Symfony de tous niveaux. La construction d’une micro-application suivant cette architecture sera présentée pas à pas.

 17 autres conférences atuour de Symfony auront lieu lors du même évènement. Les billets sont en vente sur le site du Symfony Live.

 

Introducing PropertyInfo: a PHP component to find types and doc of properties

PHP Beer Mug

PHP doesn’t support explicit type definition. This is annoying, especially when doing meta programming. Various libraries including but not limited to Doctrine ORM and the Symfony Validator provide their own type managing system. Another common way to describe type(s) of a property is integrated documentation aka PHPDoc.

The PHP Property Info library extracts various information including the type and documentation from PHP class properties from metadata of popular sources including:

  • Setter method with type hint
  • PHPDoc’s DocBlock
  • Doctrine ORM mapping (annotation, XML, YML or custom format)

Thanks to the visitor pattern, adding support for other metadata source is easy, and Pull Requests are welcome!

As usual, this library is freely available (MIT license) on my GitHub page.

As you maybe know, I’m working on a new way to build hypermedia REST API with Symfony (an experimental release is already available: DunglasJsonLdApiBundle). This bundle will leverage PHP Property Info and the Symfony Serializer Component that will be shipped in Symfony 2.7 to allow powerful deserialization.

Slides à propos des Linked Data et des API REST Hypermedia avec Symfony

Hier j’ai présenté certains de mes travaux autour des Linked Data et de Symfony au sfPot de Lille. Voici la présentation que j’ai projetée :

Voir les slides en plein écran.

sfPot à Lille sur Docker et le web sémantique avec Symfony

Alexandre Salomé (SensioLabs) et moi (Les-Tilleuls.coop) animeront un nouvel apéro Symfony à Lille le 15 janvier.

Au programmes deux talks :

  • L’un sur le web de données avec Symfony ou seront évoqués les APIs REST, l’hypermedia, Schema.org, PHP Schema et HydraBundle.
  • L’autre sur Docker ou l’on parlera… de Docker !

Tout cela arrosé de bières belges.

Rendez-vous le 15 janvier à 19h au Bar Braz rue Doudain à Lille.

Sfpot 15 janvier

Generate a Symfony password hash from the command line

There is an easy way to generate a Symfony compliant password hash from the command line. Assuming you’re using the bcrypt algorithm (the preferred choice according to Symfony’s security best practices), the default cost (13) and you have PHP >= 5.5 installed, just run the following command:

 It will output something like:  $2y$13$7mBTrD0lgdgBxt1.YbdvOOeSOrPUYOBfeC1Ra2osPs9lpCHdplw1m

You can directly use this value in your app/config/security.yml  file:

Thanks to Sarah Khalil, a built-in Symfony command will be available in a next release (and that command will support all installed algorithms).

CoopTilleulsOVHBundle: OVH SDK integration in Symfony

I’ve just released a new tiny open source bundle under the Les-Tilleuls.coop umbrella: CoopTilleulsOvhBundle.

That bundle integrates the official PHP SDK provided by OVH and exposes it as a Symfony service. It also provides a convenient way to store API credentials in parameters.yml.

The OVH API provides a lot of features including dynamic server management (elasticity) and monitoring, SMS sending, telephony configuration, billing… Almost all services provided by the first european internet hosting company (from Roubaix, France – our neighbors!) are exposed through their API, and can now be used from any Symfony application.

As usual, the source code and tests are freely available on our GitHub account. The bundle can be installed using Composer.

“Persistence in PHP with Doctrine ORM” eBook for $5 until January 6th 2015

Persistence in PHP with the Doctrine ORM

 

Thanks to Packt’s eBook Bonanza, my book “Persistence in PHP with Doctrine ORM” is available for only $5 until January 6th. Enjoy it!

What you will learn from this book

  • Install Doctrine through the Composer dependency manager
  • Configure Doctrine Command Line Tools
  • Learn to manage relations between entities with different association types
  • Create data fixtures, a custom entity repository, and native SQL queries
  • Master the query builder to generate DQL queries
  • Get started with inheritance and lifecycle events

In Detail

Doctrine 2 has become the most popular modern persistence system for PHP. It can either be used as a standalone system or can be distributed with Symfony 2, and it also integrates very well with popular frameworks. It allows you to easily retrieve PHP object graphs, provides a powerful object-oriented query language called DQL, a database schema generator tool, and supports database migration. It is efficient, abstracts popular DBMS, and supports PHP 5.3 features.

Doctrine is a must-have for modern PHP applications.

Persistence in PHP with the Doctrine ORM is a practical, hands-on guide that describes the full creation process of a web application powered by Doctrine. Core features of the ORM are explained in depth and illustrated by useful, explicit, and reusable code samples.

Persistence in PHP with the Doctrine ORM explains everything you need to know to get started with Doctrine in a clear and detailed manner.

From installing the ORM through Composer to mastering advanced features such as native queries, this book is a full overview of the power of Doctrine. You will also learn a bunch of mapping annotations, create associations, and generate database schemas from PHP classes. You will also see how to write data fixtures, create custom entity repositories, and issue advanced DQL queries. Finally it will teach you to play with inheritance, write native queries, and use built-in lifecycle events. If you want to use a powerful persistence system for your PHP application, Persistence in PHP with the Doctrine ORM is the book you.

Approach

Persistence in PHP with the Doctrine ORM is a concise, fast, and focused guide to build a blog engine with advanced features such as native queries and lifecycle callbacks.

Who this book is for

This book is primarily intended for PHP developers and architects who want to increase their skills in the field of Persistence and ORM to map the data they are working on to objects they are using in programming. Basic knowledge of databases and PDO and working knowledge of PHP namespaces is a prerequisite.

 

Two free chapters are available on the Pack website.

PHP Schema: generate a fully functional PHP / Doctrine / Symfony data model from Schema.org vocabulary in minutes

PHP Schema is a code generator that instantly generates a PHP data model from the Schema.org vocabulary : browse Schema.org, choose the types and properties you need, run our code generator and you’re done! You get a fully featured PHP data model including:

  • A set of PHP entities with properties, constants (enum values), getters, setters, adders and removers. The class hierarchy provided by Schema.org will be translated to a PHP class hierarchy with parents as abstract classes. The generated code complies with PSR coding standards.
  • Full high-quality PHPDoc for classes, properties, constants and methods extracted from Schema.org.
  • Doctrine ORM annotation mapping including database columns with type guessing, relations with cardinality guessing, class inheritance (through the @AbstractSuperclass annotation).
  • Data validation through Symfony Validator annotations including data type validation, enum support (choices) and check for required properties.
  • Interfaces and Doctrine ResolveTargetEntityListener support.
    Custom PHP namespace support.
  • List of values provided by Schema.org with PHP Enum classes.

Bonus:

  • The code generator is fully configurable and extensible: all features can be deactivated (e.g.: the Doctrine mapping generator) and custom generator can be added (e.g.: a Doctrine ODM mapping generator).
  • The generated code can be used as is in a Symfony app (but it will work too in a raw PHP project or any other framework including Laravel and Zend Framework).

Why use Schema.org data to generate a PHP model?

Don’t Reinvent The Wheel

Data models provided by Schema.org are popular and have been proved efficient. They cover a broad spectrum of topics including creative work, e-commerce, event, medicine, social networking, people, postal address, organization, place or review. Schema.org has its root in a ton of preexisting well designed vocabularies and is successfully used by more and more website and applications.

Pick up schemas applicable to your application, generate your PHP model, then customize and specialize it to fit your needs.

Improve SEO and user experience

Adding Schema.org markup to websites and apps increase their ranking in search engines results and enable awesome features such as Google Rich Snippets and Gmail markup.

Mapping your app data model to Schema.org structures can be a tedious task. Using the generator, your data model will be a derived from Schema.org. Adding microdata markup to your templates or serializing your data as JSON-LD will not require specific mapping nor adaptation. It’s a matter of minutes.

Be ready for the future

Schema.org improves the interoperability of your applications. Used with hypermedia technologies such as Hydra it’s a big step towards the semantic and machine readable web. It opens the way to generic web API clients able to extract and process data from any website or app using such technologies.

PHP 7: Introducing a domain name validator and making the URL validator stricter

PHP Origami

DNS comes with a set of rules defining valid domain names. A domain name cannot exceed 255 octets (RFC 1034) and each label cannot exceed 63 octets (RFC 1035). It can contain any character (RFC 2181) but extra rules apply for hostnames (A and MX records, data of SOA and NS records): only alphanumeric ASCII characters and hyphens are allowed in labels (we’ll talk about IDNs at the end of this post), and they cannot start nor end with a hyphen.

Until now, there was no PHP’s filter validating that a given a string is a valid domain name (or hostname). Worst,  FILTER_VALIDATE_URL was not fully enforcing domain name validity (this is mandatory for schemes such as http and https) and was allowing invalid URLs. FILTER_VALIDATE_URL was also lacking IPv6 host support.

These limitations will be fixed in PHP 7. I’ve introduced a new FILTER_VALIDATE_DOMAIN filter checking domain name and hostname validity. This new filter is now used internally by the URL validator. I also added IPv6 host support in URL validation:

There is still a big lack in PHP’s domain names and URLs handling: internationalized domain names are not supported at all in the core. I’ve already blogged about an userland workaround, but as IDNs becomes more and more popularsa core support by PHP in streams and validation is necessary. For instance, almost all french registrars support them, and even TLDs – such as the Chinese one – are available in the wild in a non-ASCII form). I’ve started a patch enabling IDN support in PHP’s streams. It works on Unix but still lacks a Windows build system. As it requires making ICU a dependency of PHP, I’ll publish a PHP RFC on this topic soon!

Continuous Integration for Symfony apps, the modern stack: quality checks, private Composer, headless browser testing…

Jenkins / Symfony

Updated 2014/11/21 : Fixed a bug when deploying (thanks @meyer_baptiste). Added a command to update GitHub commit status (thanks @samuelroze).

At Les-Tilleuls.coop, we built great e-commerce platforms and high-performance web applications with quality and agility in mind.

Most of our applications share the same technical stack:

  • the PHP language with the Symfony framework and Doctrine for back applications
  • front applications developed with CoffeeScript or JavaScript, AngularJS and Twitter Bootstrap (Grunt as build system, Yeoman for scaffolding, Karma and Jasmine for tests)
  • private Git repositories (hosted on GitHub)
  • private libraries and bundles shared between applications and exposed as Composer packages
  • phpspec unit tests
  • Doctrine Data Fixtures (enhanced with Alice and Faker)
  • Behat scenarios
  • exhaustive PHPDoc
  • capifony scripts for deployment

I managed to build a pretty cool CI system allowing us to deliver high quality Symfony applications. In this tutorial, you’ll see how it looks and how to install it step by step. If you aren’t using a CI system, it will increase drastically the quality of your Symfony projects and over the time, it will increase your team your development skills.

Our CI system mostly relies on Jenkins. At every commit (on every branch) it:

  • updates our private Composer package repository index
  • runs all tests and scenarios, including web acceptance test trough headless and real browsers
  • runs a bunch of quality checks on the code: security issues, dead code and code duplication detection, cyclomatic complexity analysis, good structuration and following of Symfony’s best practices
  • checks Coding Standards conformance
  • generates a browsable API documentation
  • instantly deploys the new version of the application on testing and sometimes even production servers (continuous deployment)

Finished feature branches are reviewed and merged in the master branch through GitHub Pull Requests by a lead dev only when all tests and checks are green.

The following tutorial can be followed on Debian (wheezy) and Ubuntu.

Installing Jenkins

The Jenkins team maintains a Debian repository providing up to date binary packages. Add it to your CI box and install the package (as root):

If nothing goes wrong, the Jenkins UI is now available on the port 8080 of your CI box. Open the UI in your favorite web browser. So easy.

Securing the installation

Jenkins Security

Currently, anyone knowing the URL of your Jenkins install can take the control over it. It’s a good idea to set credentials required to access the CI:

  1. go to “Manage Jenkins” then “Setup Security”
  2. check “Enable security”
  3. in the “Security Realm” section, select your user provider (“Jenkins’ own user database” is a good default if you don’t have a LDAP directory or any other centralized user provider for your organization)
  4. don’t forget checking “Allow users to sign up”, it’s mandatory to be able to log in after enabling the security
  5. in the “Authorization” section, choose “Project-based Matrix Authorization Strategy”, leave all checkboxes for “anonymous” empty and create a super admin user having all rights
  6. enable CSRF protection by checking “Prevent Cross Site Request Forgery exploits”
  7. save and go back to the homepage
  8. you’re prompted to authenticate, click “Sign up” and create an account with the same username you given administration rights earlier

Jenkins is now secured! Go back to security settings to disable the signup form.

Connecting Jenkins with GitHub repositories

Jenkins must be able to pull data from your privates GitHub repository. The best way I found is to create a GitHub machine user with a read-only access to private repositories. You can add users with write access to your repositories from GitHub by clicking “Settings” then “Collaborators” (the write permission is necessary to update the build status on GitHub. If you don’t care about this feature, just set a read only access to the repo).

Now, we’ll create private and public SSH keys for the UNIX account running Jenkins (on Debian it is called jenkins):

The last step is adding the public key to the GitHub account of the machine user. From GitHub’s homepage, logged in as your machine user account, go to “Settings” then “SSH Keys”. Add a new key with the content of  /var/lib/jenkins/.ssh/id_rsa.pub.

Jenkins is now able to connect to our GitHub repositories.

To update the GitHub commit status, our machine user will also need an access token to the GitHub API. Go to the settings page of the user, click on “Applications” then click on “Generate new token” and be sure to check the “repo:status” permission. Save the token, we’ll use it our build script.

Installing PHP

The PHP interpreter is needed to run our projects tests and other tools such as Composer and Satis. Install it:

apt-get install php5-cli php5-apc php5-sqlite

You noticed that I also installed APC to speed up script execution. I’ve also installed SQLite. I use it as DBMS for the  test  environment of Symfony applications.

Installing Satis

Satis

Satis is the most popular (and the only open source) private Composer package repository. This tool generates a static Composer index. Later, we will add a task in Jenkins to rebuild the repository at each commits. It will allow us to have an always up to date Composer Package Repository. To do that, we must install Satis and make the  jenkins user must able to run it.

Satis must be installed through Composer. Switch to the jenkins user and start by installing Composer, then Satis:

Then, create a config file for Satis called packages.json:

This config file will enable 2 privates Composer packages ( repo1  and repo2 ). Thanks to the SSH keys we configured later, the jenkins  user is able to connect to our privates GitHub repositories. Of course those repositories must have a composer.json file at their root. Full configuration options of Satis is documented on the Composer website.

Generate the package repository manually for the first time:

satis/bin/satis --no-interaction build /var/www/packages.json packages

The next and step is exposing our packages over HTTP. We will use nginx to do it. Switch back to the root user ( Ctrl + d) and type:

# apt-get install nginx

Then change the content of /etc/nginx/sites-enabled/default by the following:

Don’t forget to replace <my-ip>  by the list of IPs allowed to access the package repository. It will prevent exposing your private repository on the whole internet.

Restart nginx:

We should be able to browse the repository through our favorite web browser.

Enabling Git and GitHub support

GitHub Jenkins Webhook

Git and GitHub support are available through plugins. Go to “Manage Jenkins” then “Manage Plugins” and install plugins called “Git Plugin” and “GitHub Plugin”. Their names should be explicit.

Then, we will setup a GitHub Webhook to trigger a build every time a commit is pushed, in any branch.

Go to your GitHub repository, click on “Settings” then “Webhooks & Services”. Click on the “Add service” select box and choose “Jenkins (GitHub plugin)”. Enter your Jenkins’ webhook endpoint in the box, it should be something like http://ci.les-tilleuls.coop:8080/github-webhook/  (just replace the domain part of the URL) and click “Add service”.

The webook is set! Jenkins will be notified every time we push something our GitHub repo. Repeat this step for every GitHub repository you want to trigger builds.

Installing PHP CS Fixer

PHP CS Fixer is an awesome tool provided by SensioLabs. It’s 1.0 version has just been released. It automatically fixes PSR-0PSR-1PSR-2 and Symfony coding standard violations.

With its --dry-run option, it can be used in our CI to check if the produced code is clean.

PHP CS Fixer is packaged as a PHAR archive. Lets install it:

Installing phpDocumentator

phpDocumentator

phpDocumentator generates a beautiful HTML documentation based on DocBlocks documenting your classes and methods.

As for PHP CS Fixer, we will install its PHAR version, but before downloading it, we need to install some extra Debian packages:

Web Acceptance Testing with Behat and PhantomJS

We love Behat and Mink. We use them to manage our user stories and to run automated web testing scenarios.

Standards Symfony controllers are tested with the Symfony2 extension. But nowadays, we create more and more Single Page Applications usually composed of a Symfony 2 backed JSON / REST API and an AngularJS client.

It sounds like a good idea to also test interactions between AngularJS clients and Symfony REST API in the CI system. Behat and Mink will help.

I usually use Mink Selenium2 to drive PhantomJS, an awesome headless browser based on Webkit. Unlike the Symfony2 extension, the PhantomJS needs to access the application trough a public URL. We need a front controller exposing the test environment. Lets write it. We will also configure the built-in web server provided by Symfony to avoid configuring a heavier solution like nginx + PHP FPM.

The first step is to create a new front controller for the test  environment (Symfony comes with front controllers for prod  and dev  environments but not for test ).

Create a new front controller called app_test.php  in the web/  directory your application. It should contain something like the following:

We also need a custom router for the built-in web server. It will allow to use the test env. Create a file in the app/config/  directory of your application called router_test.php . Here is the sample code:

The next step is to install PhantomJS. The PhantomJS package is not available in Debian stable. We will fallback to the binary version provided on the official website:

The behat.yml  file  of your project must be changed to specify the base URL and the Selenium2’s WebDriver host to use. Here is a sample file:

You are ready to write scenarios testing the whole app. They will be executed at each commit! To test consistency across browsers and platforms (mobile devices, exotic systems…), you can take a look to SauceLabs. This SaaS platform is well integrated with Behat and can be a good (but more expensive) replacement to PhantomJS!

Checking code quality with SensioLabs Insight

SensioLabs Insight

SensioLabs published earlier this year a SaaS platform running quality analysis for PHP and especially Symfony projects. That service is called Insight. It’s user friendly, cool to play with (it provides some gamification with a system of “medals”) and finds a lot of bad practices in bundles and Symfony applications.

The first step is to create an account and register your project on the Insight website. Configure your credentials, set the Git repository to analyse and run manually a first analysis. You will probably find some problems in your app that must be fixed!

SensioLabs Insight provides a tool allowing to run new analysis in a CI system. Like other tools we already installed, the insight command is available as a PHAR. Download it as the jenkins user:

Find the uuid of the project you want to analyse with the following command:

The first time you’ll use insight.phar, you’ll be prompted for your user id and API token. You can find them in your SensioLabs Insight account page (“API/SDK” tab). Your credentials will be saved by insight.phar.

We are now able to integrate SensioLabs Insight in the Jenkins script. Write the uuid of your project somewhere, we will need it later.

Insight can output violations in the PMD format. You guessed it, Jenkins has a PMD plugin able to display nicely that sort of output.

Go to “Manage Jenkins”, “Plugin Manager”, click on the “Available” tabs and search for “PMD plugin”. Install it.

SensioLabs Insight is free for public open source projects but a subscription is needed to analyse private projects. I think it’s a good investment but I can’t blame you if you prefer using open source softwares. Sebastian Bergmann (the author of phpunit) maintains a great documentation explaining how to set up open source quality checking tools including PHPMD, phploc, PHP_Depend and phpcpd. Fell free to complete your installation with jenkins-php.org.

Note: at the time I write these lines there is some issues with the PMD output and the fail-condition option of the insight command. I’ve submitted fixes to them and I hope they will be integrated soon in the PHAR file.

Continuous Deployment with Capifony

It’s always good to have a testing server running the last version of the project. Sometimes, we go one step beyond and we let Jenkins pushing stable code in production thanks to Git’s tags (not detailed here).

Anyway, I use a tool you probably already know: capifony. It’s a deployment software specialized for Symfony projects built on top of Capistrano. It’s the easy way to deploy your application. It handle copying the source code to the server using Git, setting correct directory permissions, installing project’s dependencies, running database migration scripts, restarting services such as PHP FPM and many more. If you don’t already use it, give it a try, you will love it.

Like most Ruby applications, capifony is available as a gem. There is nothing easier than installing it on Debian:

Creating the Jenkins project

Jenkins Project

Go back to the Jenkins Dashboard and click on the “create new jobs” button. On the next screen give a name to your project and choose “Freestyle project”.

On the next screen:

  1. fill the “GitHub project” input with the base URL of your GitHub repository

In the “Source Code Management” section:

  1. select “Git” as “Source Code Management”
  2. in “Repository URL”, enter the SSH clone URL of your Git repository (it must be like git@github.com:coopTilleuls/myrepo.git )
  3. click the “Add” button below “Credentials”
  4. choose “SSH Username with private key” as “Kind”
  5. choose “From a file on Jenkins master” for “Private Key” and click “Add” (It allows to use SSH keys we created and added to GitHub in the previous step)
  6. set “Branches to build” to empty (to build all branches of the repository)
  7. select “githubweb” as “Repository browser” and enter the URL of your repo’s homepage again

In “Build triggers”:

  1. check “Build when a change is pushed to GitHub”

In “Build”:

Add a new “Execute shell” build step and use the following script as a template to fit your own needs:

Basically, the script run the various tools we installed earlier and don’t stop when an error occurs. Instead, it collect the return status of commands, and returns at the end of the script if the build is successful or not. It allows to always run all checks, even if one fail. The deployment (if enabled) only occurs if the build is successful. The built-in web server and PhantomJS run in background to allow them working simultaneously with Behat. They are killed after Behat exited.

Don’t fortget to customize value of variables in the top of the script.

Why not a XML file? Because I sometimes use other build servers such as Travis CI and Bamboo. Using a simple shell script allows to easily replace Jenkins with another server. Shell scripts can also be versioned directly in the Git repository.

In “Post-build Actions”:

  1. Add the “Set build status on GitHub commit” step
  2. Add “Publish JUnit test result report” and specify build/unit.xml for “Test report XMLs”
  3. Add “Publish PMD analysis results” and specify build/pmd.xml for “PMD results”
  4. Add “Publish Javadoc” and set build/doc as “Javadoc directory”
  5. Add other actions you’re interested in (like sending mails)

Troubleshooting GitHub API rate errors

If your start getting errors like “Could not fetch https://api.github.com/[…], enter your GitHub credentials to go over the API rate limit” in the console output, it’s because you exceed GitHub API rates for anonymous users. Just open a shell, switch to the Jenkins user, download something with Composer and enter your machine user credentials when prompted:

Because Composer stocks an OAuth in .composer/auth.json , all next calls to GitHub will succeed, even when launched by Jenkins.

Troubleshooting mails

To make Jenkins able to send mails (when a build fails for instance), you need to configure a SMTP server. The easier way is to install a local server on your CI server:

Default options should be OK.

To customize mail parameters, from the Jenkins homepage go to “Manage Jenkins” then “Configure System”. SMTP settings are under the “E-mail notification” section and the address used by Jenkins to send mails in “Jenkins Location”.

 Updating automatically the localy installed tools

As any software, all tools used by our CI server get updates fixing bugs and adding new features. Debian and Jenkins can be updated using apt. Jenkins plugins updates are managed directly trough the Jenkins user interface. But software we installed locally must be updated “by hand”. We will periodically run a small shell script I’ve written to update those tools.

Create the following update-tools.sh script as the jenkins user in its home directory ( ~jenkins):

Don’t forget to make it executable by executing  chmod +x update-tools.sh. To get the updates every night, run  crontab -e and add the following line:

00 00 * * * ~/update-tools.sh

 

You’re CI system is now ready. Your full test suit and quality analysis are running against new produced code in realtime! We also have quality checking and tests specifics for our frontend apps. Maybe If people are interested about that, maybe I’ll write another post detailing our infrastructure.

I’m sure you have your own preferred quality tools and CI best practices for Symfony projects. Share them with us in the comments :)