Delay in processing payments, syncing data to external services (QuickBooks, Salesforce, etc.)
Incident Report for ChargeOver

Root cause

  • A hardware failure in ChargeOver’s storage system caused severely degraded performance.
  • Automatic failover to redundant systems did not occur, which resulted in ChargeOver being completely unavailable for approximately 2 hours.


  • 06:10 CT on Jan 1 2022 - ChargeOver staff are alerted to abnormally slow invoice generation / payment processing times, and start investigating immediately.
  • 07:27 CT on Jan 1 2022 - Data storage for company logos is identified as causing performance impacts.
  • 08:16 CT on Jan 1 2022 - A hardware component of ChargeOver’s data storage system fails. Despite hardware redundancy being in place, the failure impacted other components and stops invoice generation / payment processing. Other components of ChargeOver remain operational.
  • 11:30 CT on Jan 1 2022 - Further hardware failures when trying to recover from the initial failure. Fail-over to the in-place redundant systems does not occur automatically, which causes ChargeOver to become entirely unavailable.
  • 13:15 CT on Jan 1 2022 - Access to most ChargeOver systems is restored.
  • 13:30 CT on Jan 1 2022 - ChargeOver staff immediately begin work on ensuring that invoice generation delays / payment processing delays are remediated.
  • 00:15 CT on Jan 2 2022 - ChargeOver staff notify affected customers of the need to check / update logos on their invoices.
  • Continued work - ChargeOver staff continue to check to ensure any data needing sync to third-party systems does get synced.


The following systems were known to be impacted by this failure:

  • Some company logos may be unavailable - all ChargeOver customers should check to ensure their company logo is showing up correctly on their invoices / payment receipts.
  • ChargeOver, and all related components (website, app, REST API, sync to external systems, customer portal, etc.) was unavailable from approximately 11:30 CT to 13:30 CT on Jan 1 2022.
  • Approximately 40% of ChargeOver customers experienced a delay in generating invoices.
  • Approximately 40% of ChargeOver customers experience a delay in processing payments.
  • Syncing of data to third-party integrations (QuickBooks, Xero, Salesforce, etc.) was delayed (and in some limited cases, the sync to these systems may still be delayed as of Jan 3).
  • Some emails (invoice due notices, payment receipts) may have failed to send.
  • Some webhooks may have failed to send.
  • Dunning / retry processes may not have run on Jan 1 and Jan 2 for a small subset of ChargeOver customers.

Future planning

ChargeOver recognizes the impact to our customers that this incident has caused.

At this time we are assessing what needs to change to make the storage systems that failed more tolerant of failures, so that this sort of incident cannot be repeated.

Posted Jan 03, 2022 - 08:48 CST

We continue to monitor and will provide a postmortem. All systems are operational.
Posted Jan 02, 2022 - 00:15 CST
All ChargeOver systems are operational at this time.

Invoices for Jan 1, 2022 have been generated, and payments for Jan 1, 2022 have been processed.

A small percentage of payments for a small subset of ChargeOver customers have been deferred for processing until tomorrow (Jan 2, 2022). These deferred payments will process automatically.

Some data may not have synced to third-party integrations (e.g. QuickBooks, Salesforce, Xero) yet. We are working on ensuring that everything syncs as quickly as possible.

ChargeOver customers should watch for an email regarding company logos, as you may be required to verify the company logo you uploaded into ChargeOver is up-to-date.

We are continuing to monitor all systems and will continue to post updates, and a postmortem.
Posted Jan 02, 2022 - 00:12 CST
Any invoices due to generate have generated. Delayed payments continue to processing and catch up.

Thanks for your patience.
Posted Jan 01, 2022 - 21:04 CST
Invoices that needed to be generated today and were delayed are generating now, and should be caught up shortly.

Payments that were delayed should begin running shortly.

Further updates will follow.
Posted Jan 01, 2022 - 16:04 CST
We are continuing to work on a fix for this issue.
Posted Jan 01, 2022 - 12:40 CST
Some services are restored. We are still working on restoring all functionality/services.

We will provide further updates soon.
Posted Jan 01, 2022 - 12:23 CST
We are continuing to work on a fix for this issue.
Posted Jan 01, 2022 - 10:51 CST
We are continuing to work on resolving this issue. ChargeOver may be unavailable for some customers.
Posted Jan 01, 2022 - 10:44 CST
ChargeOver is experiencing a delay in processing payments, and syncing data to some external services, for some ChargeOver accounts.

Our team is investigating the delays and will provide further updates soon.
Posted Jan 01, 2022 - 09:23 CST
This incident affected: Main Application, Email Sending, Payment Processing, and Integrations.