Page Comparison

Project operations

Product name: Discrepancy Report On Payments (DROP)

Oct 2023

Scrum team - informed

N/A

N/A

Documentation storage

Scrum team - decision makers

SharePoint

Confluence

GitHub Wiki

README

Version management is done via Jira fix versions, GitHub tags, and package version

Aug 2023

Scrum team - decision makers

DROP and its API have separate versions

Oct 2023

Scrum team - decision makers

The API is versioned separately using semantic versioning, with its own release notes, so that we can clearly communicate changes to its consumers (whereas if the report format changes, we want to update the application version but do not want to appear to have made changes to the API spec).

Infrastructure and tooling

AWS hosting

Spring 2022

Scrum team - decision makers

Given our product will spend most of its time inactive, periodically scaling up in response to large data submissions, AWS Lambda a serverless implementation is the best fit to design for minimal compute cost.

OpenShift OCP

AWS LZ2

AWS Lambda

Spring 2022

Scrum team - decision makers

AWS best practice serverless implementation (see “AWS hosting”), AWS LZ2fitted to our data needs (Kinesis deals with larger volumes).

AWS Lambda

AWS Kinesis

SNS

Spring 2022

Scrum team - decision makers

In order to decouple the lambdas and avoid creating a dependency between them, we chose SNS to set up execution triggers.

This service is on the allowed list of AWS services.

Step functions

SQS

Custom solution in code

Scripts

AWS Transfer Family

Summer 2022

Scrum team - decision makers

We need a secure and reliable way to get data from BCM to the AWS environment. Previously we were pulling from their server.

AWS Transfer family is a fully managed service that enables the transfer of files over SFTP, FTPS, and FTP directly into and out of Amazon S3. This was the best option because it is a fully managed service and it is on the allowed list of AWS services, and did not require us to create a custom solution.

Build and maintain our own server to pull the data from BCM’s server.

Use a 3rd party service to pull data from BCM.

Build and maintain our own server which BCM would push the data to, and then implement a custom solution to pull the data from our server into the AWS environment.

RDS

Spring 2022

Scrum team - decision makers

Given our product’s goal to identify relationships between datasets, a relational database is the best fit and RDS (PostgreSQL) is the AWS best practice implementation for our expected database usage.

RDS (PostgreSQL)

DynamoDB (NoSQL)

S3 Glacier

Oct 2023

Scrum team - decision makers

Glacier is a low cost solution for long-term storage. It is used on this project to archive files (submitted data and reports) that are older than 3 months.

See

Client/user experience

Our app is listed on the API Services Portal

2022

This allows provisioning of access and user management via IDIR, without needing our own keycloak implementation.

Separate roles for user type and client

Oct 2023

See

Data submission

2022

Because files are being pushed to DROP, our product includes custom parsers to import the contents of supported file types.

Our clients do not yet have access to development resources to submit files to an API, and/or have data governance concerns, so we set up the AWS transfer family in order to receive file submissions via SFTP.

Location and payment method tables are stored in the DB and normalized. There are pending stories to build an API to allow for updates that are not manual.

SFTP (push or pull)

JSON API

Parsing

How often should the report be generated, and how will we deliver it?

Scrum team - decision makers

Currently the report is generated daily and sent manually 1x/month. We would like to automate this process and generate/send the report daily, weekly or monthly.

Generate/Send daily/weekly/monthly.

Generate upon request via API.

Time-limited download link provided via email.

When are we able to send alerts directly to clients?

Scrum team - decision makers

Currently sent to payment inbox and manually forwarded to the appropriate person. We would like to automate this process and send the alerts to the appropriate person.

Tune alerts prior to switchover.

Add additional granularity to the recipient list for types of alerts.

Continue to cc payment inbox on all alerts.

Features

All of the submitted data is stored in the DB

Spring 2022

Scrum team - decision makers

We are currently storing more data than required to complete reconciliation/matching. This decision to store all of the data was made in order to continue building the feature which creates flat files from the transaction json and then submits these files to the General Ledger. We also chose to store the entire txn object as a jsonb column in the database in the case of errors or discrepencies in the data.

Deprecate the GL Generator

Spring 2022

Scrum team - decision makers

The General Ledger (GL) Generator transforms JSON from transaction API requests into flat files to submit to the GL. This was intended as one of the earliest and primary features of the application, but this feature is dependent on the implementation of a Garms replacement application by Service BC which is not currently in progress. The GL Generator was removed from our application to avoid having unused code/features.

API upload for all supported file types

Oct 2023

Scrum team - decision makers

The AWS SFTP transfer family is a significant cost so we would like to provide users with options to replace it. Since BCM submits file that we already have a parser for, a file upload API represents minimal impact to existing behaviour.

We were unable to identify an existing SFTP server we could pull from (unable to eliminate the transfer family). We considered whether our users could adopt a JSON-based API, but concluded the timeline for BCM to be able to do this would be too far out.

Product name: Discrepancy Report On Payments (DROP)

Oct 2023

Scrum team - informed

N/A

N/A

S3 Glacier for longterm storage

Oct 2023

Scrum team - decision makers

Can we retire the Transfer Family?

This represents a significant cost. We can implement a file submission API to replace it, but this will require client adoption.

Heuristics

2022

We are using heuristics instead of a more general matching approach.

The strictest heuristics are applied first, in descending order.

Report length

How often to generate the report

Daily.

Monthly.

Quarterly.

The report is distributed at the start of the month via an email with a download link (expires after 7 days).

Email attachment.

Email with download link.

How far back to reconcile?

How should alerts be tuned in order to deliver them to clients directly?

Reporting dashboard

Excel export.

Metabase-type dashboard.

Allow user to specify their own heuristics and file inputs on the fly.