This year I’ve started to use Codeception as an alternative to vanilla PHPUnit for testing WordPress plugins. I’m using the fantastic WP-Browser library written by Luca Tumedei. It makes writing WordPress acceptance tests sane by providing a suite of Codeception Modules to load WordPress and simulate WordPress APIs.

At iThemes we host all of our repositories on Bitbucket which made Bitbucket Pipelines a natural choice for automatically running our Codeception tests on commit. Getting everything up and running took a bit of work and wasn’t always straightforward. I intend for this post to be a guide to testing WordPress plugins with Codeception on Bitbucket Pipelines that’ll hopefully save you some time if you’re trying to set this up for yourself.

Prerequisites

Before you begin, you’ll need to already have setup Codeception and WP-Browser for your plugin. In other words, you should be able to run ./vendor/bin/codecept run <suite> successfully. Additionally, you should have a basic understanding of Docker and Docker Compose. Lastly, I’d recommend reading through the Bitbucket Pipelines basics, particularly the section on Configuration.

Server Setup

A bit of review: WP-Browser provides two main mechanisms to initialize the testing environment, WPLoader and WPDb. WPLoader is used for writing “traditional” WordPress unit tests. In other words, when you are calling your plugin’s functions/APIs that use WordPress; it only requires a database connection. WPDb is used for acceptance and functional tests. You’re testing against an actual WordPress installation, as such WPDb needs a running web server.

Local by Flywheel is my day-to-day development environment. When I started using WP-Browser I utilized Local’s built in MySQL server and ran my unit tests “in-place”. Getting this to work for acceptance tests is a bit more complicated. You need to create some mechanism to switch the database when in “testing mode” so as not destroy your development database.

To fully run our test suite we need a web server and a database server. The predominant way to setup these resources in Bitbucket Pipelines is to use their service definition API. This is what I’ve used in the past when running PHPUnit tests that only require a database. I ran into difficulties extending this setup to include a web server running WordPress.

It is certainly possible, but it wasn’t very straightforward. So instead of trying to create two complex solutions, one for local testing and another for Pipelines, I decided to explore using Docker Compose.

Docker Compose

Docker Compose allows us to have one server setup that any developer on our team can use, and can be easily reused in Bitbucket Pipelines. The first step to setting up our Docker Compose environment is choosing an image. I explored a couple of possibilities.

Choosing an Image

Vanilla PHP

One possibility was using the Vanilla PHP image provided by Docker. This would give us the greatest flexibility, but would require us to essentially develop a completely custom docker image that used this as a base.

Devilbox

The Devilbox image would give us the flexibility of a straight PHP image, but would give us a great starting point. It even comes with documentation for setting up a WordPress server. The image also includes a number of development tools like Composer, WP-CLI and even Codeception itself. The only annoyance here was that we’d still need to setup WordPress manually. Ideally we’d just request a given WordPress version and have an installation up and running immediately.

Bitnami WordPress

That led us to exploring dedicated WordPress images. The first one I tried was the Bitnami WordPress image. We can test against an arbitrary WordPress version just by pulling the correct tag, i.e. bitnami/wordpress:5.24. However, I ran into one significant issue with this image. Bitnami would chown all of the files in the server directory at startup. This would break all of our executables like webpack, vendor/bin/codecept, linting, etc…

Docker Official WordPress

When I first looked at the official Docker image I dismissed it because it didn’t include WP-CLI in the main image. We wanted to write acceptance tests for our CLI commands, and many of our other tests used our CLI commands to do things like changing settings. However, I forgot that WP-Browser includes WP-CLI as a dev dependency. When you use the WPCLI module, commands are forwarded to the dev dependency directly, a globally avaialble wp command is not required.

The only issue I ran into was that the docker image does not include the MySQL PDO extension. Codeception uses PDO in its Db module for making SQL queries. Fixing this required creating a custom Dockerfile that installed the extension. I don’t love this, it requires either building a new image when starting Pipelines, or publishing a built image and keeping it in sync with all the upstream images. For now we’re building a new image during our pipeline.

ARG WP_TAG
FROM ${WP_TAG}

RUN docker-php-ext-install pdo_mysql

One thing of note is how we are using ARG and FROM here. This allows us to dynamically set our base Docker image. We can specify the value for this ARG in our docker-compose.yml file and change it when we want to run our test suite on older WordPress or PHP versions.

Docker Compose File

Our docker-compose.yml file is fairly simple. We define two services, wordpress which will run our web server and db which will run MySQL.

version: '3.7'

services:
  wordpress:
    // ...

  db:
    // ...

Our wordpress service definition pretty much follows the official documentation. The only exception is that we are using build instead of image. Our custom Dockerfile is located in tests so we specify the context as the tests directory. Additionally, we specify the value for our WP_TAG argument as a concatenation of wordpress and an environment variable called WP_TAG.

wordpress:
  build:
    context: ./tests
    args:
      WP_TAG: wordpress:${WP_TAG}
  restart: always
  ports:
    - ${WP_PORT}:80
  environment:
    WORDPRESS_DB_HOST: db
    WORDPRESS_DB_NAME: wp
    WORDPRESS_DB_USER: wp
    WORDPRESS_DB_PASSWORD: pass
    WORDPRESS_SKIP_INSTALL: 'yes'
  volumes:
    - ./:/var/www/html/wp-content/plugins/your-plugin-dir
  depends_on:
    - db

Our db service definition is also pretty straightforward.

db:
  image: mysql:5.7
  restart: always
  ports:
    - ${DB_PORT}:3306
  environment:
    MYSQL_DATABASE: wp
    MYSQL_USER: wp
    MYSQL_PASSWORD: pass
    MYSQL_RANDOM_ROOT_PASSWORD: '1'

Lastly, we create a .env file that contains values for all our environment variables. DB_PORT and WP_PORT can be set to whatever values you like. They make it possible to connect to the test database and website from your host machine.

WP_TAG=latest
DB_PORT=9119
WP_PORT=7253

So at this point you should have the following files:

docker-compose.yml
.env
tests/Dockerfile

Codeception Setup

Now that we have an environment setup to run our tests, we need to tell Codeception about it. Typically this is done by creating a codeception.env file and specifying it as a source for params in the codeception.dist.yml file. We did nearly this except I created a docker-compose.codeception.env file. Then, when we want to run our tests, we copy that file to the main codeception.env. That main file is gitignored so if a developer doesn’t want to use our docker environment to run tests, they don’t have to.

WP_ROOT_FOLDER="/var/www/html"

TEST_SITE_WP_ADMIN_PATH="/wp-admin"
TEST_SITE_DB_NAME="wp"
TEST_SITE_DB_HOST="db"
TEST_SITE_DB_PORT="3306"
TEST_SITE_DB_USER="wp"
TEST_SITE_DB_PASSWORD="pass"
TEST_SITE_TABLE_PREFIX="wp_"
TEST_SITE_WP_URL="http://localhost"
TEST_SITE_WP_DOMAIN="localhost"
TEST_SITE_ADMIN_EMAIL="admin@localhost"
TEST_SITE_ADMIN_USERNAME="admin"
TEST_SITE_ADMIN_PASSWORD="password"

TEST_DB_NAME="wp"
TEST_DB_HOST="db"
TEST_DB_USER="wp"
TEST_DB_PASSWORD="pass"
TEST_TABLE_PREFIX="wp_"

Our env file just follows our Docker configuration. It should work with the standard WP-Browser setup.

At this point you should have the following files:

docker-compose.codeception.env
docker-compose.yml
.env
tests/Dockerfile

Now we can run our tests by logging into the docker container and running codeception.

docker-compose up -d
docker-compose exec wordpress /bin/bash
cd wp-content/plugins/your-plugin-dir
./vendor/bin/codecept run <suite>

Script Aliases

We wanted to provide a simple way to run tests without needing to login to the container and run a set of commands so we defined a set of scripts to use as shortcuts.

{
  "scripts": {
    "test-wpunit": "docker-compose exec -T -w /var/www/html/wp-content/plugins/your-plugin-dir wordpress ./vendor/bin/codecept run wpunit",
    "test-acceptance": "docker-compose exec -T -w /var/www/html/wp-content/plugins/your-plugin-dir wordpress ./vendor/bin/codecept run acceptance",
    "test-build": "docker-compose exec -T -w /var/www/html/wp-content/plugins/your-plugin-dir wordpress ./vendor/bin/codecept build"
  }
}

npm is our task runner of choice, so we defined these in our package.json file, but the same concept could apply to Gulp, Grunt, Composer, etc…

Bitbucket Pipelines Setup

Now it’s time to put this all together into a setup that Bitbucket Pipelines can use to test our code on every commit. We do this by creating a bitbucket-pipelines.yml file in the root of our repository.

First we need to define a step to build our plugin. This setup is based on Bitbucket’s documentation. Of note is the artifacts section. This specifies our entire plugin directory as an artifact that we carry to our other pipeline steps.

pipelines:
  default:
    - step:
        name: Build
        image: php:7.3
        caches:
          - composer
        script:
          - apt-get update && apt-get install -y unzip
          - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
          - composer install --ignore-platform-reqs
          - cp docker-compose.codeception.env codeception.env
        artifacts:
          - '**'

Next we specify the steps for actually running our tests. We use the parallel keyword to run all of these steps at once on different machines. Following the documentation for using Docker on Bitbucket Pipelines we specify docker as one of our services and caches. Then in our script, we call a custom script called setup-pipeline.sh and then run our tests.

pipelines:
  default:
    - step:
      name: Build
    - parallel:
        - step:
            name: WP Unit Tests ( Latest )
            services:
              - docker
            caches:
              - docker
            script:
              - ./bin/setup-pipeline.sh
              - npm run-script test-wpunit -- --xml test-reports/wpunit.xml

When we run our tests we’re also passing the --xml flag. This tells Codeception to generate JUnit files that contain machine readable information about our test results. Bitbucket uses this information to provide an indication of how many tests passed or failed.

If we want to run our tests in another environment, for instance WordPress 5.2, we can change the WP_TAG environment variable using sed.

- step:
    name: WP Unit Tests ( Latest )
- step:
    name: WP Unit Tests ( 5.2 )
    services:
      - docker
    caches:
      - docker
    script:
      - sed -i -e 's/WP_TAG=latest/WP_TAG=5.2/g' .env
      - ./bin/setup-pipeline.sh
      - npm run-script test-wpunit -- --xml test-reports/wpunit.xml

Our setup-pipeline.sh script is fairly simple. We start by downloading and installing docker-compose. Then, we start up docker and wait for our test site to become available.

#!/usr/bin/env bash
curl -L https://github.com/docker/compose/releases/download/1.25.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose -f docker-compose.yml up -d
./bin/wait-for-it.sh http://localhost:7253

Docker takes a few seconds to get all of our services to get up and running. So we use this simple wait-for-it.sh script to wait until the site returns a valid response.

#!/usr/bin/env bash
# Based on https://stackoverflow.com/a/50583452

attempt_counter=0
max_attempts=30

until curl --output /dev/null --silent --fail "$1"; do
    if [ ${attempt_counter} -eq ${max_attempts} ];then
      echo "Max attempts reached"
      exit 1
    fi

    printf '.'
    attempt_counter=$(($attempt_counter+1))
    sleep 1
done

echo "Connected to host"

Conclusion

I hope this helps setting up Bitbucket Pipelines to run Codeception tests for WordPress plugins. Feel free to leave a comment with any questions or issues you run into. I’ve also created a Gist that includes all of the code examples from this post in one place.