Code updates and Infrastructure upgrades have been mostly automated so that these operations can happen rapidly. This happens via multiple github repositories and actions.

Playbooks

Playbooks are composed of an action or series of actions that constitute a a common process.

Test a new version of WordPress and/or nginx

A scheduled process is responsible for periodically checking for a new version of WordPress and nginx. This process will deploy the image to the wordpress-wordpress-run:latest image tag, from which a test suite will be run comparing the dev and latest versions, at the end of which digital-latest.apps.silver.devops.gov.bc.ca will be a copy of production with the latest version of wordpress running on it.

Check the results of the screenshot tests, and do some user testing of the site. Once satisfied with the upgrade, you can change the test and production image tags in openshift and bring down the pods in those deployments. Once the deployments are restarted with new pods, the new version of wordpress will be in place.

Workflows

...

Workflows

Create a new WordPress instance

A new WordPress site can be created by executing the Create Wordpress Instance workflow. This will create all the necessary openshift resources in the specified environment, who’s names will be prefixed by the site-name input and the URL will be digital-site-nameapps.silver.devops.gov.bc.ca. The controlling infrastructure-as-code is implemented with kustomizedkustomize, and can be found in the wordpress-deploy-digimod repository. The action will also copy the wordpress WordPress cli to the container and perform a site install.

...

1. Create Wordpress Instance

2. Replicate Production

3. Sitemap Screenshots

4. Deploy theme

5. Deploy plugins

6. Verify Screenshots

Perform a test of a Wordpress upgrade

We check for wordpress and nginx upgrades with a github actions cron job. This is a more complex process that involves several workflows over multiple repositories.

1. Every Monday at midnight, a workflow from the wordpress-deploy-digimod repository compares the version found in the dockerfile of the build configuration for wordpress, and compares it to the latest version found in dockerhub. If these are not the same, it updates the dockerfile in the repository.

2. A second workflow is then triggered when the dockerfile is updated, which builds the image and updates the image in openshift, under the ‘latest’ tag.

3. A third workflow in the wordpress-digimod repository operates on a cron job scheduled at 1AM on Monday, one hour after the first workflow is executed. It checks the status of the second workflow; if it is successful, it deploys the new version of wordpress to digital-latest. It first deploys from the dev tag, then takes screenshots, deploys from the latest tag, and then compares.

Configure the SSO for a site

...

You can perform a migration to production and import it from the specified environment and site digital-site-nameapps.silver.devops.gov.bc.ca. This can be done by running the Site Export from Production workflow.

Note: this process was not verified against production.

Backup

The backup system utilizes all in one All in One WordPress Migration tool to create snapshots on a daily basis and stores them on the digital-backup site. Digital-backup uses netapp-file-backup PVC to store it’s data so it’s backed up offsite using OCIO backup infrastructure.

...

php wp-cli.phar user create backup-admin backup-admin@info.com --role=administrator --user_pass=********

Alternatively run the Restore backup action to test site and then run Reconfigure SSO plugin to allow Keycloak logins to test site that will now contain the backup.

Note: Currently there are no notifications if the backup process failed for some reason, for example due to failures of remote oc commands. It is advisable to create a separate process that verifies that backup in fact took place and that restore was successful.

...

Note: in case of OpenShift node failure that causes the failure of both the production PVC as well as the digital-backup PVC resulting in a situation where no snapshots are available on the OpenShift cluster to restore from, it is necessary to request a restoration from the offsite location back to the OpenShift cluster according to standard procedures. Note that this process have not been verified and may take some time (possibly days) during which the site will not be available. Once the restore is complete, the simplest way to restore would be to download the snapshot and perform a manual restore using the all in one plugin. Alternatively restore digital-backup site functionality and run the Restore Backup workflow as describe abovedescribed above.

Note: restore backup process was not tested against production, however works properly with “test” as the target. It is therefore recommended to verify that restore backup process is working correctly.

Take Screenshots

The first step in screenshot testing is to take an initial set of screenshots before making changes to a site instance. This way this set of screenshots can be compared to a new set of screenshots after changes have been made (for example theme or WordPress version upgrade).

...