A Fiber line was accidentally cut through construction at our ACH Processor's primary data center, causing traffic/processing attempts to automatically route to their secondary data center.
However, the secondary data center then became overwhelmed with requests, which slowed / stopped transactions from being processed. The processor added additional server resources to mitigate the issue, but the system still had to play catch-up for a while.
As of early this morning, they posted an update saying the connection had been restored to the primary data center, and things had returned to normal.
We tested things on our end, and bank gifts appear to be processing normally again through RebelGive.
All ACH scheduled / recurring gifts that had been scheduled for yesterday/today while there was an outage will be automatically retried. We are waiting a bit longer before turning back on our systems for processing recurring gifts to ensure things stay stable at our processor.
Again, there is no need for us or Admins or givers to manually intervene to "make up" for failed gifts. All will be re-attempted today/tomorrow.
We will continue to monitor and post an update when processing of recurring and scheduled gifts have been turned back on.
Thanks for your patience as we waited for this to be resolved 🙏
Posted Sep 15, 2020 - 11:14 EDT
Update
We are continuing to monitor for any further issues.
Posted Sep 15, 2020 - 10:53 EDT
Update
We are running some final tests on our end before confirming everything is back to normal and starting up some of our recurring giving processing which we had paused to ensure donors didn't get unnecessary failure emails
Posted Sep 15, 2020 - 10:53 EDT
Update
We are continuing to monitor for any further issues. As of 6:40AM PST, we received a message from our processor, letting us know that their system was starting to work normally again.
Posted Sep 15, 2020 - 06:30 EDT
Monitoring
ACH (Bank) gifts from one-time and scheduled/recurring gifts are failing via our payment processing partner. Credit Card gifts are currently unaffected. We are waiting for a further update from our processing partners that this issue has been resolved on their end. Currently, all attempted ACH gifts from about 1PM EST are affected.
Scheduled/Recurring ACH gifts will be automatically retried by RebelGive over the next 3 days. There is no need to schedule or initiate additional gifts to make up for the failed attempts.
We will post more technical details and updates as we get more information.