ACH Processing Issue

Incident Report for RebelGive

Postmortem

In addition to mitigating the ramifications of the issue encountered today due to an unexpected API change at the ACH processor (that they then subsequently rolled back), we have added additional safeguards to protect our system to rely less on initial success/approved messages from their system and instead focus on error messages which will result in more of a failsafe.

We are also addressing this internally with our processing partner to ensure their communication and changes in the future are handled in a better way to ensure this kind of an issue does not occur again.

NOTE:
Any givers who tried to give between 6/29/2021 at 10pm - 6/30/2021 2:51pm CST and got a failure message on the GivingFlow should retry their gifts.

Any givers who received a failure message on a scheduled or recurring ACH gift will automatically be retried on our regular retry schedule (tomorrow (7/1/2021) at the same time).

Again, thank you for your patience as we resolved this unexpected (and unprecedented) issue.

Posted Jun 30, 2021 - 20:17 EDT

Resolved

The issue at the ACH processor (and all ramifications of the issue in our system) have now been fully resolved. 🥳

GivingFlows are all active (and have been for hours, but to avoidance of doubt we wanted to restate)

All recurring/scheduled giving is now processing as normal.

Any scheduled/recurring ACH gifts that failed today will be automatically retried according to their schedule and the retry parameters we already had in place prior to today's issue.

Thank you for being patient as we worked to take care of all the details to make sure things got fully back to normal as fast as possible.

Posted Jun 30, 2021 - 20:07 EDT

Update

We're continuing to resolve the last of the effects of this on a block of transactions during the issue timeframe. Will post one or two more final updates after this to tie things up cleanly.

Posted Jun 30, 2021 - 18:16 EDT

Monitoring

Another remedy has been put in place to handle the unexpected rollback of a prior change from the processor (the original change that caused the initial issue which we subsequently fixed rapidly), but now had to rollback our fix for their original change that caused the issue since they rolled it back.

We have tested the latest change and have now safely reactivated GivingFlows.

We are continuing to monitor the situation closely and continue to patch up transactions and records.

Most one-time transactions will need to be retried. We will have more updates and details on this as soon as the issues are all completely resolved.

NOTE: Recurring/Scheduled Giving processing remains paused for the time being.

Posted Jun 30, 2021 - 15:51 EDT

Update

We're continuing to work hard on a full and complete resolution

Posted Jun 30, 2021 - 15:31 EDT

Update

We are continuing to work on a fix for this issue.

Posted Jun 30, 2021 - 14:59 EDT

Identified

Due to another unnoticed change with the ACH processor, we have temporarily paused GivingFlows from loading until we update our system again to accomodate their sudden additional change.

We are so sorry for any inconvenience and are working hard to get everything back to normal as quickly as we can due to these unexpected processor issues and changes that were initiated without any forewarning to us and our platform.

Again, all recurring giving is still paused in the queue as we continue to resolve so we don't add to the scope of transactions that need to be patched up +/or voided due to this issue.

Posted Jun 30, 2021 - 14:57 EDT

Update

We're continuing to monitor while we resolve and restore all transactions to their correct state and status.

GivingFlows are processing gifts normally, and recurring giftschedules are still paused until we fully correct all transactions that need to be addressed. As noted previously, recurring gifts simply got the wrong failure email when they should have received a success email. We will be resending corrected receipt emails as part of the final resolution

Posted Jun 30, 2021 - 14:02 EDT

Monitoring

An initial resolution has been shipped to production and we have re-enabled the GivingFlow.

Processing of Recurring Giving remains paused for a bit longer while we work behind the scenes to remedy and link up transaction statuses and continue monitoring to ensure everything remains working as expected with this fix that has been implemented

Posted Jun 30, 2021 - 12:00 EDT

Update

We are continuing to work on a fix for this issue.

Posted Jun 30, 2021 - 11:26 EDT

Identified

We've identified what we think is the root cause of the issue (an unexpected change in validation messages at the processor for ACH gifts causing error messages to be shown and failure emails to be sent unnecessarily.

Will continue to post updates as we learn more and continue to resolve the issue

Posted Jun 30, 2021 - 10:51 EDT

Update

We've temporarily disabled the GivingFlow while we continue to investigate the root cause of the issue.

Posted Jun 30, 2021 - 10:32 EDT

Update

We have temporarily disabled recurring giving out of an abundance of caution. Gifts will resume normally and catch up / retry once we have identified and resolved the current issue

Posted Jun 30, 2021 - 10:27 EDT

Investigating

We have received a few reports of ACH processing of gifts failing. We have confirmed there are some failures, and are investigating further. We will keep this issue updated as we learn more.

Posted Jun 30, 2021 - 10:09 EDT

This incident affected: GivingFlows and ACH Payment Processing.