DROP Decision Log
- Kendall.Olsen-Maier (Unlicensed)
This page captures key decisions made in the course of development.
Decision | Date (est) | Parties | Rationale | Options considered |
|---|---|---|---|---|
Project operations | ||||
Product name: Discrepancy Report On Payments (DROP) | Oct 2023 | Scrum team - informed | N/A | N/A |
Documentation storage | PENDING | Scrum team - decision makers |
| SharePoint Confluence GitHub Wiki README |
Version management is done via Jira fix versions, GitHub tags, and package version | Aug 2023 | Scrum team - decision makers |
|
|
DROP and its API have separate versions | Oct 2023 | Scrum team - decision makers | The API is versioned separately using semantic versioning, with its own release notes, so that we can clearly communicate changes to its consumers (whereas if the report format changes, we want to update the application version but do not want to appear to have made changes to the API spec). |
|
Infrastructure and tooling | ||||
AWS hosting | Spring 2022 | Scrum team - decision makers | Given our product will spend most of its time inactive, periodically scaling up in response to large data submissions, a serverless implementation is the best fit to design for minimal compute cost. | OpenShift OCP AWS LZ2 |
AWS Lambda | Spring 2022 | Scrum team - decision makers | AWS best practice serverless implementation (see “AWS hosting”), fitted to our data needs (Kinesis deals with larger volumes). | AWS Lambda AWS Kinesis |
SNS | Spring 2022 | Scrum team - decision makers | In order to decouple the lambdas and avoid creating a dependency between them, we chose SNS to set up execution triggers. This service is on the allowed list of AWS services. | Step functions SQS Custom solution in code Scripts |
AWS Transfer Family | Summer 2022 | Scrum team - decision makers | We need a secure and reliable way to get data from BCM to the AWS environment. Previously we were pulling from their server. AWS Transfer family is a fully managed service that enables the transfer of files over SFTP, FTPS, and FTP directly into and out of Amazon S3. This was the best option because it is a fully managed service and it is on the allowed list of AWS services, and did not require us to create a custom solution. | Build and maintain our own server to pull the data from BCM’s server. Use a 3rd party service to pull data from BCM. Build and maintain our own server which BCM would push the data to, and then implement a custom solution to pull the data from our server into the AWS environment. |
RDS | Spring 2022 | Scrum team - decision makers | Given our product’s goal to identify relationships between datasets, a relational database is the best fit and RDS (PostgreSQL) is the AWS best practice implementation for our expected database usage. | RDS (PostgreSQL) DynamoDB (NoSQL) |
S3 Glacier | Oct 2023 | Scrum team - decision makers | Glacier is a low cost solution for long-term storage. It is used on this project to archive files (submitted data and reports) that are older than 3 months. | |
Client/user experience | ||||
Our app is listed on the API Services Portal | 2022 |
| This allows provisioning of access and user management via IDIR, without needing our own keycloak implementation. |
|
Separate roles for user type and client | Oct 2023 |
|
| |
Data submission | 2022 |
| Because files are being pushed to DROP, our product includes custom parsers to import the contents of supported file types. Our clients do not yet have access to development resources to submit files to an API, and/or have data governance concerns, so we set up the AWS transfer family in order to receive file submissions via SFTP. Location and payment method tables are stored in the DB and normalized. There are pending stories to build an API to allow for updates that are not manual. | SFTP (push or pull) JSON API Parsing |
How often should the report be generated, and how will we deliver it? | PENDING | Scrum team - decision makers | Currently the report is generated daily and sent manually 1x/month. We would like to automate this process and generate/send the report daily, weekly or monthly. | Generate/Send daily/weekly/monthly. Generate upon request via API. Time-limited download link provided via email. |
When are we able to send alerts directly to clients? | PENDING | Scrum team - decision makers | Currently sent to payment inbox and manually forwarded to the appropriate person. We would like to automate this process and send the alerts to the appropriate person. | Tune alerts prior to switchover. Add additional granularity to the recipient list for types of alerts. Continue to cc payment inbox on all alerts. |
Features | ||||
All of the submitted data is stored in the DB | Spring 2022 | Scrum team - decision makers | We are currently storing more data than required to complete reconciliation/matching. This decision to store all of the data was made in order to continue building the feature which creates flat files from the transaction json and then submits these files to the General Ledger. We also chose to store the entire txn object as a jsonb column in the database in the case of errors or discrepencies in the data. |
|
Deprecate the GL Generator | Spring 2022 | Scrum team - decision makers | The General Ledger (GL) Generator transforms JSON from transaction API requests into flat files to submit to the GL. This was intended as one of the earliest and primary features of the application, but this feature is dependent on the implementation of a Garms replacement application by Service BC which is not currently in progress. The GL Generator was removed from our application to avoid having unused code/features. |
|
API upload for all supported file types | Oct 2023 | Scrum team - decision makers | The AWS SFTP transfer family is a significant cost so we would like to provide users with options to replace it. Since BCM submits file that we already have a parser for, a file upload API represents minimal impact to existing behaviour. | We were unable to identify an existing SFTP server we could pull from (unable to eliminate the transfer family). We considered whether our users could adopt a JSON-based API, but concluded the timeline for BCM to be able to do this would be too far out. |
Can we retire the Transfer Family? | PENDING |
| This represents a significant cost. We can implement a file submission API to replace it, but this will require client adoption. |
|
Heuristics | 2022 |
| We are using heuristics instead of a more general matching approach. The strictest heuristics are applied first, in descending order. |
|
Report length | PENDING |
|
|
|
How often to generate the report | PENDING |
|
| Daily. Monthly. Quarterly. |
The report is distributed at the start of the month via an email with a download link (expires after 7 days). | PENDING |
|
| Email attachment. Email with download link. |
How far back to reconcile? | PENDING |
|
|
|
How should alerts be tuned in order to deliver them to clients directly? | PENDING |
|
|
|
Reporting dashboard | PENDING |
|
| Excel export. Metabase-type dashboard. Allow user to specify their own heuristics and file inputs on the fly. |