Invoice generation, sending, and payment processing is delayed for some accounts
Incident Report for ChargeOver

What Happened / Customer Impact

Our certificate authority issued ChargeOver a TLS/SSL certificate with a CA chain certificate which expired before our actual TLS/SSL certificate did. This caused automatic invoice generation and payment processing to be delayed for some customers.

Technical Details

ChargeOver secures internal and external connections with TLS/SSL certificates. The company ChargeOver purchases certificates from issued us an SSL/TLS certificate with a chain file which expired before our actual certificate. So even though our SSL/TLS certificate was valid and not expired, a certificate in the chain needed to validate the certificate expired on May 30th.

Most web browsers were unaffected, so access to the website and app were unaffected. However, common libraries and tools like cURL and wget began rejecting connections due to the expired chain certificate. ChargeOver's scheduled invoice generation and scheduled payment processes depend on the cURL library, and so the system was unable to trigger invoice generation and payment processing for some accounts scheduled to generate invoices after the certificate had expired.

Our monitoring picked up the issue, and alerted our engineering team. ChargeOver then removed the expired certificate from the chain, and invoices and payments began processing normally again.

This caused a delay of invoice/payment processing of between 30 minutes and 4 hours for some ChargeOver accounts.

Ongoing Efforts

We are working on some additional validation to ensure that our CA cannot issue us certificates with a certificate chain that expires prior to the certificate we are being issued.

Posted Jun 03, 2020 - 10:58 CDT

This issue has been resolved. Postmortem to follow.
Posted May 30, 2020 - 19:14 CDT
We continue to monitor things, as things catch up -- invoices are generating, emails are sending, and payments are processing.
Posted May 30, 2020 - 16:29 CDT
We have resolved the issue, and services are being restored. Invoices are generating, emails are sending, and payments are processing as normal now.

Some ChargeOver accounts may see delayed generation of invoices while things catch back up. A further update will follow.
Posted May 30, 2020 - 14:23 CDT
Some ChargeOver accounts are experiencing delayed invoice generation, invoice sending, and automated payment processing.

We are investigating.
Posted May 30, 2020 - 13:09 CDT
This incident affected: Email Sending and Payment Processing.