What Happened / Customer Impact

Our certificate authority issued ChargeOver a TLS/SSL certificate with a CA chain certificate which expired before our actual TLS/SSL certificate did. This caused automatic invoice generation and payment processing to be delayed for some customers.

Technical Details

ChargeOver secures internal and external connections with TLS/SSL certificates. The company ChargeOver purchases certificates from issued us an SSL/TLS certificate with a chain file which expired before our actual certificate. So even though our SSL/TLS certificate was valid and not expired, a certificate in the chain needed to validate the certificate expired on May 30th.

Most web browsers were unaffected, so access to the ChargeOver.com website and app were unaffected. However, common libraries and tools like cURL and wget began rejecting connections due to the expired chain certificate. ChargeOver's scheduled invoice generation and scheduled payment processes depend on the cURL library, and so the system was unable to trigger invoice generation and payment processing for some accounts scheduled to generate invoices after the certificate had expired.

Our monitoring picked up the issue, and alerted our engineering team. ChargeOver then removed the expired certificate from the chain, and invoices and payments began processing normally again.

This caused a delay of invoice/payment processing of between 30 minutes and 4 hours for some ChargeOver accounts.

Ongoing Efforts

We are working on some additional validation to ensure that our CA cannot issue us certificates with a certificate chain that expires prior to the certificate we are being issued.