Delay in processing payments, syncing data to external services (QuickBooks, Salesforce, etc.)

Root cause

• A hardware failure in ChargeOver’s storage system caused severely degraded performance.
• Automatic failover to redundant systems did not occur, which resulted in ChargeOver being completely unavailable for approximately 2 hours.

Timeline

• 06:10 CT on Jan 1 2022 - ChargeOver staff are alerted to abnormally slow invoice generation / payment processing times, and start investigating immediately.
• 07:27 CT on Jan 1 2022 - Data storage for company logos is identified as causing performance impacts.
• 08:16 CT on Jan 1 2022 - A hardware component of ChargeOver’s data storage system fails. Despite hardware redundancy being in place, the failure impacted other components and stops invoice generation / payment processing. Other components of ChargeOver remain operational.
• 11:30 CT on Jan 1 2022 - Further hardware failures when trying to recover from the initial failure. Fail-over to the in-place redundant systems does not occur automatically, which causes ChargeOver to become entirely unavailable.
• 13:15 CT on Jan 1 2022 - Access to most ChargeOver systems is restored.
• 13:30 CT on Jan 1 2022 - ChargeOver staff immediately begin work on ensuring that invoice generation delays / payment processing delays are remediated.
• 00:15 CT on Jan 2 2022 - ChargeOver staff notify affected customers of the need to check / update logos on their invoices.
• Continued work - ChargeOver staff continue to check to ensure any data needing sync to third-party systems does get synced.

Impact

The following systems were known to be impacted by this failure:

• Some company logos may be unavailable - all ChargeOver customers should check to ensure their company logo is showing up correctly on their invoices / payment receipts.
• ChargeOver, and all related components (website, app, REST API, sync to external systems, customer portal, etc.) was unavailable from approximately 11:30 CT to 13:30 CT on Jan 1 2022.
• Approximately 40% of ChargeOver customers experienced a delay in generating invoices.
• Approximately 40% of ChargeOver customers experience a delay in processing payments.
• Syncing of data to third-party integrations (QuickBooks, Xero, Salesforce, etc.) was delayed (and in some limited cases, the sync to these systems may still be delayed as of Jan 3).
• Some emails (invoice due notices, payment receipts) may have failed to send.
• Some webhooks may have failed to send.
• Dunning / retry processes may not have run on Jan 1 and Jan 2 for a small subset of ChargeOver customers.

Future planning

ChargeOver recognizes the impact to our customers that this incident has caused.

At this time we are assessing what needs to change to make the storage systems that failed more tolerant of failures, so that this sort of incident cannot be repeated.