One of the primary tasks around maintaining digital.gov.bc.ca WordPress instance involve keeping relevant parts of the system up to date. There are 5 parts of the system that can be upgraded:

1. WordPress version upgrade

2. Nginx version upgrade

3. Stock plugins

4. BCGov Block Theme

5. Digimod plugins

All upgrades follow the same pathway to ensure the stability of the system:

1. Replicate production to the test instance to get a clone of production environment

2. Capture screenshots on the test instance before the upgrade is made

3. Perform upgrades on test

4. Verify screenshots to capture any regressions resulting from upgrades

5. Verifying any other functionality that may be affected by the upgrade. Important: due to deficiency in automatic testing for certain features (such as custom plugins, i.e. QA Plugin), these features need to be tested manually to ensure the stability of the system

6. Perform adjustments until screenshot tests pass and any manual testing issues are resolved

7. Perform the same set of upgrades on production

It is recommended that the system updates are carried out on a regular basis (weekly or biweekly).

Setup

The steps below are common to all upgrade tasks and should be done as a pre-requisite before an actual upgrade is attempted.

Create a fresh test instance

The first step for any upgrade process is to create a fresh instance of test instance of WordPress. To do so, run the Create WordPress Instance workflow with default parameters (“test”/”test”). This will create a blank WordPress deployment at https://digital-test.apps.silver.devops.gov.bc.ca/. For more information about this workflow, see the relevant section in the DevOps Guide.

Replicate production to test instance

Once the fresh instance of WordPress has been setup, run the Site Export from Production workflow with default parameters (“test”/”test”). For more information about this workflow, see the relevant section in the DevOps Guide. Once complete, verify that https://digital-test.apps.silver.devops.gov.bc.ca/ contains production replica.

Note: You will not be able to login into this instance as the SSO plugin has not been configured (user will receive “Invalid parameter: redirect_uri” error when attempting to login). To enable Keycloak logins into the test instance, see the next step

Enable Keycloak login on test instance

To enable Keycloak logins on the test instance (after Site Export from Production has been ran), run the Reconfigure SOO workflow with default parameters (“test”/”test”). For more information about this workflow, see the relevant section in the DevOps Guide. Once complete, verify that you can login using Keycloak into the test instance (Digital Government – Province of British Columbia).

Verify WordPress version

Ensure that the WordPress version installed on production is the same as that running on the test instance.

Capture Screenshots

Now that the test instance contains the exact replica of production, run the Take screenshots workflow to capture the state of the site before any upgrades. For more information about this workflow, see the relevant section in the DevOps Guide.

Upgrades

The following sections lists the instructions for doing various kinds of upgrades on the test instance once the setup section above has been completed.

WordPress version upgrade

Note that automatic WordPress upgrades have been disabled through “DIGIMOD - Disable Automatic Updates” plugin and system upgrades must be done manually on a regular basis. This is done to prevent divergence between the WordPress version that is running inside the pod and the supporting image. In case of a node restart, the pod will reboot and a rollback to an older version will happen because the image has not been updated. This rollback can have adverse effects on the system as such downgrades are not expected. Therefore the system in place ensures that there is a parity between the image and the running version of WordPress.

Check for new WordPress version

To see if a new WordPress version is available, check the status page of the production site logged in as administrator (https://digital.gov.bc.ca/wp-admin/update-core.php). Note: do not update WordPress through this interface, this is only for checking if a new version is available.

Determine tag name for the new WordPress version

The next step in the upgrade process is to find the tag name for the newly released version of WordPress. To do this, navigate to wordpress Tags | Docker Hub page and locate the fpm version of tag with the latest version, for example 6.2.2-php8.0-fpm-alpine. Important: ensure that the format of the tag is precise, e.g. {WORDPRESS_VERSION}-php{PHP_VERSION}-fpm-alpine.

Note: it is advisable to run the same version of WordPress as the GDX DES team to ensure compatibility between WordPress, DES plugins and the BCGov Block Theme. To check which version GDX is running, refer to the docker file in the upstream repository.

Build new WordPress version to latest tag in OpenShift

To create a new build for WordPress, run the Check for new WordPress Docker image workflow from the wordpress-deploy-digimod repository. This repository is a fork of wordpress deployments project. This forked repository should not be kept up to date with the upstream repository unless infrastructure upgrades are required, in which case the update would have to be tested thoroughly.

The “Check for new WordPress Docker image” workflow will update the dockerfile containing the version of WordPress and automatically trigger the Build WordPress image workflow that will build the image into the “latest” tag in the “wordpress-wordpress-run” image stream in the “tools” namespace on OpenShift.

Rename latest tag to test tag

Now that the latest image has been built to “latest” tag, rename it to “test” tag in order for the new image to be available to the test instance of the site. To do this, run the Rename latest WordPress tag workflow with default parameter (“test”).

Restart test instance

In order for the WordPress instance to pick the new image, the deployment must be rolled out. To do so, run the Restart Instance workflow with default parameter (“test”/”test”). Login to WordPress and verify that the new version of WordPress is now running.

Verify screenshots

Run the Verify Screenshots workflow to determine if any visual regressions took place as a result of an upgrade. Perform any necessary fixes and repeat the workflow until results are satisfactory.

Rename latest tag to production tag

Finally push the image to the production by running Rename latest WordPress tag workflow, this time with “prod” as an argument.

Restart production

Restart the production instance by running the Restart Instance workflow with “prod” as the first argument (the second argument is irrelevant). Verify that the new WordPress version is now running and apply any fixes that were done on the test instance to bring the site into alignment.

Rename latest tag to dev tag

“dev” tag is used for the development instance of WordPress. Once production has been upgraded, rename “latest” tag to “dev” to ensure that dev site is also running on latest WordPress version (restart dev using restart instance workflow).

Nginx version upgrade

Nginx version upgrade is similar to the WordPress upgrade.

1. Run Check for new nginx Docker image workflow with a desired nginx tag as input. This will update the appropriate docker file and trigger the build workflow. Refer to the upstream repository dockerfile for the current version of nginx ran by the GDX DES team.

2. Rename the latest tag in the image stream to “test” using Rename latest nginx tag workflow with default parameter (“test”)

3. Restart the test WordPress instance using Restart instance workflow and verify that it functions correctly.

4. Rename the latest tag in the image stream to “prod” using Rename latest nginx tag workflow with “prod” as the argument

5. Rest production server using the Restart instance workflow with “prod” as the argument

6. Rename “latest” tag to “dev” to ensure that dev instance of WordPress is also running the latest version (restart dev using restart instance workflow)

Updating Stock Plugins

There is currently no automatic way to perform upgrades of stock plugins. It is recommended to implement such approach using composer.json in combination with WordPress Packagist: Manage your plugins and themes with Composer service. This work has been started with the introduction of composer.json file, available on the plugins-composer branch.

To update stock plugins, perform the setup procedure as outlined below first. Next, manually upgrade all plugins on the test instance, verify the screenshots and then perform the same set of upgrades on production.

Updating Theme

The theme in use by digital.gov.bc.ca is a fork of the original BC Gov block theme and is available here: https://github.com/bcgov/bcgov-wordpress-block-theme-digimod. The fork does not contain any modifications beyond github workflows and it’s purpose is to control the release cycle of new versions of the theme.

Update Fork

First step in upgrading the theme is to bring the forked theme up to speed with the original repository. This can be done by creating a pull request.

Setup testing environment and publish new theme

Perform the setup process as described above first. Next, publish the new theme to test instance by running the Deploy theme workflow with default parameters (“test”/”test”). Once published, run the Verify Screenshots workflow and finally use the same workflow to publish the theme to production (this time with “prod”/”test” parameters - second one is irrelevant in this case).

Undo theme publish

When theme gets published, previous theme is kept. To undo the publish, simply activate the duplicated theme via the Themes interface in WordPress. Note: only one backup is kept, so if more than one publish was carried out in a row, the original theme will be lost.

Deploying custom plugins

Custom plugins are located in the https://github.com/bcgov/wordpress-digimod repository. To deploy a single plugin, use the Deploy Plugin workflow and use Deploy Plugins workflow to deploy all plugins in the repository. It is recommended that new functionality is tested on test or dev instance of WordPress before deploying it to production. Screenshot testing may be employed to ensure there are no visual regressions on the site (these tests won’t cover backend plugin functionality).

Manually adding a user to a WordPress instance

Sometimes it may be necessary to add a user to WordPress instance in order to login to the site (for example if a site was restored to an instance for which there is no KeyCloak config). To add a user in such cases, login to OpenShift, locate the the relevant deployment, navigate to the pod and open the terminal, in terminal, enter:

Replace USERNAME and SECURE_PASSWORD with your username and password.

Important: Ensure that the password used is secure (of appropriate length and complexity) and delete the user once it is no longer needed. Avoid this approach unless necessary (Keycloak login is preferred).

Analyzing activity log

If it is necessary to determine the source of a change on the site, use the Simple History plugin available via Dashboard → Simple History to view the activity log of the site. The page can be used to assess for suspicious activity and track changes on the site as well as who is the author of those changes.

Summary of periodic checks and additional information

The following maintenance tasks should be carried out on periodic basis to ensure stability of the platform and keeping it up to date:

• Check and install new WordPress versions

• Check and install new nginx versions

• Upgrade stock plugins (for example Advanced Custom Fields and All in One SEO)

• Periodically update BCGov Block theme

• Verify that backup is functioning correctly (verify log output of the Backup GitHub Worklfow and ensure Digital Government – Province of British Columbia is running the latest version). To login to the WordPress site, add a user to the WordPress instance manually and check activity log (available through Dashboard) to verify there is recent activity.

• Verify that file system volume and the database volume have sufficient space (digital-wordpress and digital-wordpress-mariadb PVCs, respectively)

• Ensure that OpenShift namespaces (prod, test and dev) do not contain loose applications that are becoming out of date.