Code updates and Infrastructure upgrades have been mostly automated so that these operations can happen rapidly. This happens via multiple github repositories and actions.

...

Workflows

Create a new

...

WordPress instance

A new WordPress site can be created by executing the Create Wordpress Instance workflow. This will create all the necessary openshift resources in the specified environment, who’s names will be prefixed by the site-name input and the URL will be digital-site-nameapps.silver.appsdevops.gov.bc.ca. The controlling infrastructure-as-code is implemented with kustomizedkustomize, and can be found in the wordpress-deploy-digimod repository. The action will also copy the wordpress WordPress cli to the container and perform a site install.

...

You can deploy the latest theme from the bcgov-wordpress-block-theme-digimod repository with the Deploy Plugins workflow. The will deploy to the specified environment for digital-site-nameapps.appssilver.devops.gov.bc.ca.

Update digimod plugins

You can deploy the latest plugins from the wordpress-digimod repository with the Deploy Plugins workflow. The will deploy to the specified environment for digital-site-nameapps.appssilver.devops.gov.bc.ca.

Update a single plugin

You can deploy the a specific plugin from the wordpress-digimod repository with the Deploy Plugin workflow. The will deploy the plugin to the specified environment for digital-site-nameapps.appssilver.devops.gov.bc.ca.

Run the test suite

Perform a full site export and test

...

1. Create Wordpress Instance

2. Replicate Production

3. Sitemap Screenshots

4. Deploy theme

5. Deploy plugins

6. Verify Screenshots

Perform a test of a Wordpress upgrade

We check for wordpress and nginx upgrades with a github actions cron job. This is a more complex process that involves several workflows over multiple repositories.

1. Every Monday at midnight, a workflow from the wordpress-deploy-digimod repository compares the version found in the dockerfile of the build configuration for wordpress, and compares it to the latest version found in dockerhub. If these are not the same, it updates the dockerfile in the repository.

2. A second workflow is then triggered when the dockerfile is updated, which builds the image and updates the image in openshift, under the ‘latest’ tag.

3. A third workflow in the wordpress-digimod repository operates on a cron job scheduled at 1AM on Monday, one hour after the first workflow is executed. It checks the status of the second workflow; if it is successful, it deploys the new version of wordpress to digital-latest. It first deploys from the dev tag, then takes screenshots, deploys from the latest tag, and then compares.

Configure the SSO for a site

WordPress deployments can be configured for SSO integration with Keycloak. The integrations are managed through the CSS app. SSO is managed on the client side with the MiniOrange OAuth plugin. Sites in Dev, Test, and Prod all must use different client credentials, so they must be set up differently. This can be set up by running the Reconfigure SSO workflow. Note that both the MiniOrange OAuth plugin and the digimod-misc plugin (which contains the wordpress script for changing the MiniOrange configuration) must already be installed on the site for this to work. This can be done either from an Export from production, deploy plugins, or deploy plugin workflow.

Export from production

You can perform a site backup from production and import it to the specified environment for digital-site-nameapps.silver.appsdevops.gov.bc.ca. This can be done by running the Site Export from Production workflow.

...

You can perform a migration to production and import it from the specified environment and site digital-site-nameapps.appssilver.devops.gov.bc.ca. This can be done by running the Site Export from Production workflow.

Note: this process was not verified against production.

Backup

The backup system utilizes all in one All in One WordPress Migration tool to create snapshots on a daily basis and stores them on the digital-backup site. Digital-backup uses netapp-file-backup PVC to store it’s data so it’s backed up offsite using OCIO backup infrastructure.

The backup is accomplished through a cron job ran by GitHub Actions via the Backup workflow. The workflow creates a backup on production using ai1wm CLI, copies that backup to digital-backup site and restores it so it’s available for viewing on a public URL.

Note: it is currently not possible to login into the backup site as the SSO configuration is invalid due to the restoration process. To login to the site, it is necessary to create a user manually by logging on to the pod terminal and running wp-cli command to create an admin user:

curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar

php wp-cli.phar user create backup-admin backup-admin@info.com --role=administrator --user_pass=********

Alternatively run the Restore backup action to test site and then run Reconfigure SSO plugin to allow Keycloak logins to test site that will now contain the backup.

Note: Currently there are no notifications if the backup process failed for some reason, for example due to failures of remote oc commands. It is advisable to create a separate process that verifies that backup in fact took place and that restore was successful.

Restore Backup

The restore process works through the restoration of the ai1wm image stored on the digital-backup site to a target instance (for example production). This is the same process that is used for site migration from a non-production environment to production (for example when there are major changes being done to the site).

To restore the site to a previous state, use the Restore Backup workflow. The workflow allows the restoration to various targets, including production (follow prompt instructions) as well as selecting from up to 5 latest backups.

Note: in case of OpenShift node failure that causes the failure of both the production PVC as well as the digital-backup PVC resulting in a situation where no snapshots are available on the OpenShift cluster to restore from, it is necessary to request a restoration from the offsite location back to the OpenShift cluster according to standard procedures. Note that this process have not been verified and may take some time (possibly days) during which the site will not be available. Once the restore is complete, the simplest way to restore would be to download the snapshot and perform a manual restore using the all in one plugin. Alternatively restore digital-backup site functionality and run the Restore Backup workflow as described above.

Note: restore backup process was not tested against production, however works properly with “test” as the target. It is therefore recommended to verify that restore backup process is working correctly.

Take Screenshots

The first step in screenshot testing is to take an initial set of screenshots before making changes to a site instance. This way this set of screenshots can be compared to a new set of screenshots after changes have been made (for example theme or WordPress version upgrade).

To take a set of screenshots, run the Take Screenshots workflow. Options include selecting which site instance to run the tests on, for production select “prod”. Any other value will be inserted into https://digital-{SITENAME}.apps.silver.devops.gov.bc.ca/. Note: currently the script is only configured to run with test due to configuration issues in cypress.config.js file that is not picking up passed in environmental variables in certain cases.

Once the action has ran, screenshots will be available for download in action details as an archive. Note that if the action is ran again, the previous screenshots will not be saved and will be overwritten with a new set.

Note: there have been some issues with images in the screenshot tests due to them not loading consistently before the screenshot is taken resulting in false negatives being produced during the verification stage. To mitigate this, images have been disabled. There are instances, however, where images are being set using background-image css rule (for example on communities of practice pages), so those may still produce failed tests that need to be reviewed manually. This can be fixed by targeting those elements specifically in the

Verify Screenshots

Once the initial set of screenshots have been taken and a site change has been implemented (for example a WordPress upgrade has been performed on test) run the Verify Screenshots workflow to compare the screenshots of the site before and after the upgrade. Once finished, an archive of screenshots will be available for download. The archive will contain before/after screenshots for failed tests as well as image diff files highlighting areas of difference in red color.

Verify Screenshots workflow can be run repeatedly to until outstanding issues have been eliminated. Once done so, perform the same series of upgrade steps on production with the result of the production site not experiencing any visual regressions.