This is not a tutorial or guide that aims to teach you anything practical. It’s a backgrounder where we can elaborate a bit on how we approach this specific topic. It’s a way for us to tell you a little bit of how we work and by doing that, also tell a bit about who we are.
What does code quality mean to us
Testable code
Before anything really. We consider testable code being the most important corner stone of a high quality PHP application. This means that we keep classes and functions small and single purpose, go out of our way to avoid unwanted dependencies and that our code is extendable and yet maintainable. If this reminds you a bit about the SOLID principles, we’re doing a good job explaining this. The expression testable code should probably be clarified a bit, what we really mean is automatically testable code.
We’re mostly writing code for WordPress so we have to live with a few compromises. WordPress is still huge on global functions, global constants and a few other things that can make testing harder because of dependencies. But we believe we’re making the most out of it and to be fair we see that WordPress core and some of the bigger plugins (WooCommerce) is getting better over time.
Our experience is that the more you aim to write testable code, the more your code will adhere to the SOLID principles almost automatically. Turns out that all those hours of computer science training are useful also in the WordPress world. Who could have guessed?
Unit tests
So besides the side effect of testable code often meaning cleaner code, we do write a lot of unit tests. We aim for a test coverage of 70% – 90% in our unit test suits. At the time of writing this, this translates to 423 test cases doing about 1300 individual assertions (image below might be older). The production code to test code ratio is about 1:1 right now which might be on the lower end.
We like to keep an eye on these metrics because they say something about how we’re doing over time. But it’s not the KPIs that dictates our overall product development.
The thing to keep in mind with unit tests is that they are small and aim to tests small pieces of code without any dependencies. Our unit test suite runs in about 10-15 seconds which is very reasonable. This means that a developer can run this tests several times per hour to verify that things doesn’t break in unexpected ways.
To mock the WordPress dependencies we’ve worked with various mock libraries over time. But we’ve ended up writing our own mock objects by hand. For WunderAutomation we have about 10 WordPress and WooCommerce (very light weight) mock object implementations in place so it’s not too bad and absolutely worth the effort. We’ve also had to implement a way to mock WordPress global functions:
Example of mocking global functions. We either redirect the function call to a public function on the test class, or give it a hard coded return value. Our mocks also keep track of the call count, so after testing a function, we can check that our code called a WordPress global function the expected number of times:
We’re happy to elaborate over how we mock WordPress in a more detailed post, let us know in the comments if this is something you’d like to see.
Functional tests
Let’s be transparent. Unit tests are great for what they do. We manage to cover at least 70% of WunderAuomation with Unit tests. The funny thing is that when executing those tests, not one single line of WordPress code is loaded into memory. So they really doesn’t prove that our code works with WordPress at all. They only prove that our code works with our assumptions of how WordPress works.
To make sure our plugins actually works with the actual WordPress code we have a separate set of tests that loads the WordPress stack and runs our tests with or inside WordPress. This is called functional tests. Functional tests requires a working WordPress installation to work.
The flow of our functional tests is typically is something like this:
- Load the WordPress stack by including wp-load.php. This will load up everything needed to run a WordPress request, including database connection, loading plugins etc.
- Run an internal WordPress function, like wp_insert_post
- Assert that our own code triggered correctly by WordPress and returned the desired output.
- Assert that our code resulted in the expected updates in the WordPress database etc.
Since the functional tests require a working WordPress installation, they take a lot more time to execute. Each time the tests are run, a fresh installation is created, dependant plugins are installed and sometimes configured etc. And each test case will load the WordPress stack with database connection and what not so even if both preparing for and running the functional tests are fully automatic, they take a lot more time compared to the unit tests.
The functional tests are also based on PHPUnit but as opposed to the unit tests we don’t calculate any code coverage from these tests. Partly because it’s quite difficult but perhaps more because it’s not really the intended purpose.
Acceptance testing
Acceptance testing is sometimes also called user acceptance testing or black box testing. It’s tests that are designed to test the product from the view of the user. We’re using Codeception for this because it has great support for using chrome driver.
Our acceptance tests are essentially scripts that emulates how a user navigates a WordPress site with our product installed. Then the script asserts that the “user” sees certain things. A simple example of what we want to verify when an administrator logs in to the WordPress admin panel can look like this:
$I->amOnAdminPanel();
$I->see('Howdy, admin');
$I->see('Automation');
$I->click('Automation', '#adminmenu');
$I->see('Workflows');
$I->see('Settings');
$I->see('Log');
$I->see('Parameter test');
The great thing with Codeception / Chrome driver is that the web pages we test are actually rendered in a real chrome browser where css and javascripts also affect what’s happening. So we’re really testing from the user perspective and get at least indirect testing of or javascript code as well.
Code style
For us, code quality is also about code style. We’re going to be very transparent here. We care a lot about PHP code style our build system will scream at us if we fail to follow it. We care a little less about code style in our javascript code and almost nothing when it comes to CSS.
The whole point of adhering to a code style is that it makes it easier for humans to read and understand the code. Code is difficult enough as it is, we don’t want to put extra cognitive load on our brains by having to look hard just to figure out what the code actually says. A code style guide line helps us make sure that all code have the same visual markers, makes it easy to identify the start and end of control structures, identify variables, functions, etc.
$ phpcs /path/to/code/myfile.php
FILE: /path/to/code/myfile.php
--------------------------------------------------------------------------------
FOUND 5 ERROR(S) AFFECTING 2 LINE(S)
--------------------------------------------------------------------------------
2 | ERROR | Missing file doc comment
20 | ERROR | PHP keywords must be lowercase; expected "false" but found "FALSE"
47 | ERROR | Line not indented correctly; expected 4 spaces but found 1
51 | ERROR | Missing function doc comment
88 | ERROR | Line not indented correctly; expected 9 spaces but found 6
--------------------------------------------------------------------------------We follow the PSR2 / PSR12 style guidelines for all of our code that are part of our class structure. Code that directly interfaces with WordPress should adhere to the WordPress code style guide but we’ll be honest and say that we find it tricky to draw the line.
Our tools
PHPUnit
We’ve already mentioned PHPUnit, the de-facto standard unit test runner for PHP. It’s used for all our unit and functional tests.
PHP Code sniffers
We use PHP_CodeSniffer to enforce all code style rules:
- PSR and WordPress code style rules depending on the source file
- PHP version syntax checking, we check code for syntax issues from PHP 5.6 and later
- A set of our own pick and chose PHPCS rules put together using Edorians PHPCS config generator. We use this to enforce some rules that are not caught via other rule sets, including leaving TODO for FIXME comments in code, multiple statements per line, inline control structures etc.
Codeception with Chrome driver
We’ve already mentioned Codeception and Chrome driver above. We’re considering moving our unit and functional tests into the Codeception structure, right now we only use it for acceptance tests. But that’s an exercise for a seriously rainy day.
Robo
Finally, we use Robo to run a lot of actions from the command line. We have a simple built / publishing system implemented in robo as well as the more ambitious Carson that you will read about next. Robo is ideal for running small tasks like “bump the version number in the source”, “build an installable zip”, “send the resulting zip to the web server etc”. All of our WordPress plugins are using the exact same base Robo configuration and scripts so bumping a version number, building and publishing always works the same.
Vagrant and Ansible
Yes, we know everything is supposed to be about Docker these days, but we’re quite happy to continue working with Vagrant and Ansible.
We use Vagrant as the main development environment and we use Ansible to make sure that all developers have the exact same environment. Ansible sets up our test box so that is has:
- Latest Ubuntu LTS version (we’re switching to 20.04 as we write this)
- All PHP versions from 5.6 to 7.4, the latest version as the default.
- nginx (main) and Apache (for testing) web servers
- ngix mapped so that ports 8056 to 8074 are mapped to PHP versions 5.6 to 7.4.
- MySQL set up with separate databases for each PHP version (so that installations can co-exist)
- Chromium browser and the chrome test driver (for the acceptance tests)
- Composer, wp-cli, Robo and other PHP command line tools.
Continuous Integration and Mr Carson
Why the name
There’s been a few other build and continuous integration projects with names relating to fictional english butlers. We named our Carson after the butler Mr Carson from TV serie Downton Abbey simply because the Downton Abbey movie had just premiered the same week we were searching for a suitable name.
In the series, Mr Carson has a very strict approach to rules and traditions and is never afraid to speak out if something doesn’t sit right according to the rules. Perfect personality for automated QA.
What is Carson
Carson is a custom task build on top of Robo. It constantly monitors a few git repos and whenever a new commit is found, it executes configured tasks defined in a carson config file for that repo. The tasks are divided into two major parts:
Fast: tests code sniffers and unit tests
The first part of the tests are designed to be fast. Time needed to run the fasts tests on the WunderAutomation repository is about 45 seconds. The idea is that the fast tests should be run on every single commit. The fast part contains the following steps
- PHP Compatibility check – our currently supported range of PHP versions is 5.6 to 7.4. We use the PHPCompatibility plugin for PHP Code sniffer that checks for PHP cross-version compatibility.
- Code style checks – we check all pure PHP source files (classes) for PSR2 / PSR12 compatibility.
- Unit tests – since all our unit tests are quick (as they should be), we run the unit tests in the fast part.
- Unit test coverage – we measure the total code coverage from unit test and unit test to production code ratio.
Full tests
In the second part, we run the tests that take some more time, the functional and acceptance tests. Two things make these types of tests time consuming (a) they require full clean WordPress installation(s) and (b) we run the tests over a matrix of every possible combination of PHP version and the plugin dependencies.
We currently support six versions of PHP and test the all code on 2-3 versions of WordPress and 2 versions of WooCommerce, so thats currently 6 x 3 x 2 = 36 individual test rounds. Each round have to start out from a clean well known state that takes a few seconds to setup. This part of our test suite takes up to 20 minutes to run.
The full tests includes
- Functional tests – We still use PHPUnit as the driver for these, but each tests loads the full WordPress stack.
- Acceptance tests This is what we use Codeception for. The acceptance tests runs a virtual browser, logs in and makes sure things in the UI looks and works correct.
What Carson doesn’t do
Carson isn’t actually involved in building and publishing products to our web page or the WordPress plugin repository. The build and publish scripts are also implemented using Robo, so it would be a fairly easy thing to add but for now we keep them separate for various reasons.
Conclusion
If you’ve made it all the way down here you’re either very interested in this subject or have a specific reason figure out if our products adhere to your quality standards. We hope they do.
We take code quality seriously because it’s important. Furthermore we’re a small team and we need to protect our own time. We want to spend the most of our time developing exciting features and the least of our time debugging our code after it has been installed on someone else’s WordPress. A fairly ambitious testing procedure is the best way we came up with to make it so.
If you want us to elaborate more about any part of this test setup, please let us know in the comments below. We’re also open for suggestions, corrections or an occasional “thanks for sharing”.
Thanks for sharing this. I’m curious about how your mock objects for WordPress work. All my tests with PHPUnit uses WordPress (functional?). Would be interesting
/Lewis