Page Comparison

Project operations

Product name: Discrepancy Report On Payments (DROP)

Oct 2023

Scrum team - informed

N/A

N/A

Documentation storage

Scrum team - decision makers

SharePoint

Confluence

GitHub Wiki

README

Version management is done via Jira fix versions, GitHub tags, and package version

Scrum team - decision makers

DROP and its API have separate versions

Scrum team - decision makers

Infrastructure and tooling

AWS hosting

Spring 2022

Scrum team - decision makers

Given our product will spend most of its time inactive, periodically scaling up in response to large data submissions, a serverless implementation is the best fit to design for minimal compute cost.

OpenShift OCP

AWS LZ2

AWS Lambda

Spring 2022

Scrum team - decision makers

AWS best practice serverless implementation (see “AWS hosting”), fitted to our data needs (Kinesis deals with larger volumes).

AWS Lambda

AWS Kinesis

SNS

Spring 2022

Scrum team - decision makers

In order to decouple the lambdas and avoid creating a dependency between them, we chose SNS to set up execution triggers.

This service is on the allowed list of AWS services.

Step functions

SQS

Custom solution in code

Scripts

AWS Transfer Family

Summer 2022

Scrum team - decision makers

We need a secure and reliable way to get data from BCM to the AWS environment. Previously we were pulling from their server.

AWS Transfer family is a fully managed service that enables the transfer of files over SFTP, FTPS, and FTP directly into and out of Amazon S3. This was the best option because it is a fully managed service and it is on the allowed list of AWS services, and did not require us to create a custom solution.

Build and maintain our own server to pull the data from BCM’s server.

Use a 3rd party service to pull data from BCM.

Build and maintain our own server which BCM would push the data to, and then implement a custom solution to pull the data from our server into the AWS environment.

RDS

Spring 2022

Scrum team - decision makers

Given our product’s goal to identify relationships between datasets, a relational database is the best fit and RDS (PostgreSQL) is the AWS best practice implementation for our expected database usage.

RDS (PostgreSQL)

DynamoDB (NoSQL)

S3 Glacier

Oct 2023

Scrum team - decision makers

Glacier is a low cost solution for long-term storage. It is used on this project to archive files (submitted data and reports) that are older than 3 months.

See

Client/user experience

API Services Portal

Roles

Data submission

Because files are being pushed to DROP, our product includes custom parsers to import the contents of supported file types.

Location and payment method tables.

SFTP (push or pull)

JSON API

Parsing

Reporting

Generated daily, sent manually.

Alerts

Sent to payment inbox.

Features

Storing data

“we are saving the entire the ten file as a jsonb column, we are storing data that we don't use that is specific to the general ledger stuff”

Removing the JSON -> flat file parsing

“we're not currently planning to produce the gl generator flat files for the ministry-client”

How often should the report be generated, and how will we deliver it?

Scrum team - decision makers

Currently the report is generated daily and sent manually 1x/month. We would like to automate this process and generate/send the report daily, weekly or monthly.

Generate/Send daily/weekly/monthly.

Generate upon request via API.

Time-limited download link provided via email.

When are we able to send alerts directly to clients?

Scrum team - decision makers

Currently sent to payment inbox and manually forwarded to the appropriate person. We would like to automate this process and send the alerts to the appropriate person.

Tune alerts prior to switchover.

Add additional granularity to the recipient list for types of alerts.

Continue to cc payment inbox on all alerts.

Features

All of the submitted data is stored in the DB

Spring 2022

Scrum team - decision makers

We are currently storing more data than required to complete reconciliation/matching. This decision to store all of the data was made in order to continue building the feature which creates flat files from the transaction json and then submits these files to the General Ledger. We also chose to store the entire txn object as a jsonb column in the database in the case of errors or discrepencies in the data.

Deprecate the GL Generator

Spring 2022

Scrum team - decision makers

The General Ledger (GL) Generator transforms JSON from transaction API requests into flat files to submit to the GL. This was intended as one of the earliest and primary features of the application, but this feature is dependent on the implementation of a Garms replacement application by Service BC which is not currently in progress. The GL Generator was removed from our application to avoid having unused code/features.

API upload for all supported file types

Oct 2023

Scrum team - decision makers

The AWS SFTP transfer family is a significant cost so we would like to provide users with options to replace it. Since BCM submits file that we already have a parser for, a file upload API represents minimal impact to existing behaviour.

We were unable to identify an existing SFTP server we could pull from (unable to eliminate the transfer family). We considered whether our users could adopt a JSON-based API, but concluded the timeline for BCM to be able to do this would be too far out.

Can we retire the Transfer Family?

Heuristics

Using heuristics instead of a more general matching approach.

Strictest heuristics are applied first, in descending order.

Report length

How often to generate the report

How to distribute the report

How far back to reconcile

How should alerts be tuned in order to deliver them to clients directly?

Reporting dashboard