One of the primary tasks around maintaining digital.gov.bc.ca WordPress instance involve keeping relevant parts of the system up to date. There are 4 5 parts of the system that can be upgraded:
WordPress version upgrade
Nginx version upgrade
Stock plugins
BCGov Block Theme
Digimod plugins
...
Replicate production to the test instance to get a clone of production environment
Capture screenshots on the test instance before the upgrade is made
Perform upgrades on test
Verify screenshots to capture any regressions resulting from upgrades
Verifying any other functionality that may be affected by the upgrade. Important: due to deficiency in automatic testing for certain features (such as custom plugins, i.e. QA Plugin), these features need to be tested manually to ensure the stability of the system
Perform adjustments until screenshot tests pass and any manual testing issues are resolved
Perform the same set of upgrades on production
It is recommended that the system updates are carried out on a regular basis (weekly or biweekly).
| Table of Contents |
|---|
Setup
The steps below are common to all upgrade tasks and should be done as a pre-requisite before an actual upgrade is attempted.
...
WordPress version upgrade
Note that automatic WordPress upgrades have been disabled through “DIGIMOD - Disable Automatic Updates” plugin and system upgrades must be done manually on a regular basis. This is done to prevent divergence between the WordPress version that is running inside the pod and the supporting image. In case of a node restart, the pod will reboot and a rollback to an older version will happen because the image has not been updated. This rollback can have adverse effects on the system as such downgrades are not expected. Therefore the system in place ensures that there is a parity between the image and the running version of WordPress.
Check for new WordPress version
...
The next step in the upgrade process is to find the tag name for the newly released version of WordPress. To do this, navigate to https://hub.docker.com/_/wordpress/tags page and locate the fpm version of tag with the latest version, for example 6.2.02-php8.20-fpm-alpine. Important: ensure that the format of the tag is precise, e.g. {WORDPRESS_VERSION}-php{PHP_VERSION}-fpm-alpine.
Note: it is advisable to run the same version of WordPress as the GDX DES team to ensure compatibility between WordPress, DES plugins and the BCGov Block Theme. To check which version GDX is running, refer to the docker file in the upstream repository.
Build new WordPress version to latest tag in OpenShift
To create a new build for WordPress, run the Check for new WordPress Docker image workflow from the wordpress-deploy-digimod repository. This repository is a fork of wordpress deployments project. This forked repository should not be kept up to date with the upstream repository unless infrastructure upgrades are required, in which case the update would have to be tested thoroughly.
The “Check for new WordPress Docker image” workflow will update the dockerfile containing the version of WordPress and automatically trigger the Build WordPress image workflow that will build the image into the “latest” tag in the “wordpress-wordpress-run” image stream in the “tools” namespace on OpenShift.
Rename latest tag to test tag
Now that the latest image has been built to “latest” tag, rename it to “test” tag in order for the new image to be available to the test instance of the site. To do this, run the Rename latest WordPress tag workflow with default parameter (“test”).
Restart test instance
In order for the WordPress instance to pick the new image, the deployment must be rolled out. To do so, run the Restart Instance workflow with default parameter (“test”/”test”). Login to WordPress and verify that the new version of WordPress is now running.
Verify screenshots
Run the Verify Screenshots workflow to determine if any visual regressions took place as a result of an upgrade. Perform any necessary fixes and repeat the workflow until results are satisfactory.
Rename latest tag to production tag
Finally push the image to the production by running Rename latest WordPress tag workflow, this time with “prod” as an argument.
Restart production
Restart the production instance by running the Restart Instance workflow with “prod” as the first argument (the second argument is irrelevant). Verify that the new WordPress version is now running and apply any fixes that were done on the test instance to bring the site into alignment.
Rename latest tag to dev tag
“dev” tag is used for the development instance of WordPress. Once production has been upgraded, rename “latest” tag to “dev” to ensure that dev site is also running on latest WordPress version (restart dev using restart instance workflow).
Nginx version upgrade
Nginx version upgrade is similar to the WordPress upgrade.
Run Check for new nginx Docker image workflow with a desired nginx tag as input. This will update the appropriate docker file and trigger the build workflow. Refer to the upstream repository dockerfile for the current version of nginx ran by the GDX DES team.
Rename the latest tag in the image stream to “test” using Rename latest nginx tag workflow with default parameter (“test”)
Restart the test WordPress instance using Restart instance workflow and verify that it functions correctly.
Rename the latest tag in the image stream to “prod” using Rename latest nginx tag workflow with “prod” as the argument
Rest production server using the Restart instance workflow with “prod” as the argument
Rename “latest” tag to “dev” to ensure that dev instance of WordPress is also running the latest version (restart dev using restart instance workflow)
Updating Stock Plugins
There is currently no automatic way to perform upgrades of stock plugins. It is recommended to implement such approach using composer.json in combination with https://wpackagist.org/ service. This work has been started with the introduction of composer.json file, available on the plugins-composer branch.
To update stock plugins, perform the setup procedure as outlined below first. Next, manually upgrade all plugins on the test instance, verify the screenshots and then perform the same set of upgrades on production.
Updating Theme
The theme in use by digital.gov.bc.ca is a fork of the original BC Gov block theme and is available here: https://github.com/bcgov/bcgov-wordpress-block-theme-digimod. The fork does not contain any modifications beyond github workflows and it’s purpose is to control the release cycle of new versions of the theme.
Update Fork
First step in upgrading the theme is to bring the forked theme up to speed with the original repository. This can be done by creating a pull request.
Setup testing environment and publish new theme
Perform the setup process as described above first. Next, publish the new theme to test instance by running the Deploy theme workflow with default parameters (“test”/”test”). Once published, run the Verify Screenshots workflow and finally use the same workflow to publish the theme to production (this time with “prod”/”test” parameters - second one is irrelevant in this case).
Undo theme publish
When theme gets published, previous theme is kept. To undo the publish, simply activate the duplicated theme via the Themes interface in WordPress. Note: only one backup is kept, so if more than one publish was carried out in a row, the original theme will be lost.
Deploying custom plugins
Custom plugins are located in the https://github.com/bcgov/wordpress-digimod repository. To deploy a single plugin, use the Deploy Plugin workflow and use Deploy Plugins workflow to deploy all plugins in the repository. It is recommended that new functionality is tested on test or dev instance of WordPress before deploying it to production. Screenshot testing may be employed to ensure there are no visual regressions on the site (these tests won’t cover backend plugin functionality).
Manually adding a user to a WordPress instance
Sometimes it may be necessary to add a user to WordPress instance in order to login to the site (for example if a site was restored to an instance for which there is no KeyCloak config). To add a user in such cases, login to OpenShift, locate the the relevant deployment, navigate to the pod and open the terminal, in terminal, enter:
| Code Block |
|---|
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar |
| Code Block |
|---|
php wp-cli.phar user create USERNAME username@example.com --role=administrator --user_pass=SECURE_PASSWORD |
Replace USERNAME and SECURE_PASSWORD with your username and password.
Important: Ensure that the password used is secure (of appropriate length and complexity) and delete the user once it is no longer needed. Avoid this approach unless necessary (Keycloak login is preferred).
Analyzing activity log
If it is necessary to determine the source of a change on the site, use the Simple History plugin available via Dashboard → Simple History to view the activity log of the site. The page can be used to assess for suspicious activity and track changes on the site as well as who is the author of those changes.
Summary of periodic checks and additional information
The following maintenance tasks should be carried out on periodic basis to ensure stability of the platform and keeping it up to date:
Check and install new WordPress versions
Check and install new nginx versions
Upgrade stock plugins (for example Advanced Custom Fields and All in One SEO)
Periodically update BCGov Block theme
Verify that backup is functioning correctly (verify log output of the Backup GitHub Worklfow and ensure https://digital-backup.apps.silver.devops.gov.bc.ca/ is running the latest version). To login to the WordPress site, add a user to the WordPress instance manually and check activity log (available through Dashboard) to verify there is recent activity.
Verify that file system volume and the database volume have sufficient space (digital-wordpress and digital-wordpress-mariadb PVCs, respectively)
Ensure that OpenShift namespaces (prod, test and dev) do not contain loose applications that are becoming out of date.