Delayed delivery of some emails

Incident details

From July 25th 14:00 PST to July 29 20:41 PST, some emails sent via ChargeOver were delayed.

Only emails sent via ChargeOver’s email provider were affected (emails sent via ChargeOver’s custom SMTP, custom SendGrid, Mailgun, and Mandrill integrations were not impacted.)

Root cause

On two separate days in July, malicious users logged into two separate ChargeOver accounts, and used them to send a large number of spam/scam emails.

The ChargeOver accounts were customers of ChargeOver - no

Both ChargeOver accounts used extremely easy to guess passwords, used those passwords across many other applications beyond ChargeOver, and did not have 2FA/MFA enabled within ChargeOver. Malicious actors were able to guess the ChargeOver users' passwords to log in to ChargeOver. ChargeOver itself was not hacked and did not suffer any sort of security breach.

This led to ChargeOver’s primary email provider (SendGrid) temporarily placing a hold on some outgoing email from ChargeOver.

ChargeOver worked closely with SendGrid to restore normal email delivery.

Incident timeline

• Early July - two malicious actors guess ChargeOver user passwords (or re-use passwords from other applications), and send many malicious emails via ChargeOver
• July 12 - 06:22 CT - ChargeOver’s monitoring automatically alerts our team of unexpected activity on two ChargeOver accounts, and we notify the affected customers and force password changes on the affected accounts
• July 13 - 08:25 CT - ChargeOver begins work towards future mitigation strategies
• (July 13 - July 29) - ChargeOver pushed out 19 separate updates through this 2-week period to mitigate impact, protect against future attacks, and migrate some critical pieces of infrastructure away from the affected SendGrid account)
• July 25 - 16:00 CT - SendGrid places a suspension/hold on one of ChargeOver’s email accounts, but does not notify us
• July 26 - 11:55 CT - ChargeOver staff reach out to SendGrid because we are seeing some email being delayed
• July 29 - 22:41 CT - SendGrid removes the hold/suspension on the account, and normal email delivery returns

Remediation plan

We’ve identified a number of items to be addressed to protect against future attacks, and mitigate impact.

The most important item is to encourage all ChargeOver customers to enable 2FA/MFA on their ChargeOver account. Enabling 2FA/MFA is the quick, easy, free, and the single most important thing anyone can to do protect against any sort of unauthorized access to any application you use online. ChargeOver customers will see a push for 2FA/MFA adoption across all ChargeOver accounts.

Items already accomplished during the July 13 - July 29th period:

• Improved / more proactive monitoring of spam reports
• Added rate limiting for sending email through the ChargeOver admin panel
• Move critical email infrastructure to separate, independent SendGrid accounts

Future plans:

• Enforce rate limiting for email sending through the REST API
• Enforce rate limiting for email sending through the ChargeOver.js API
• Add various rate limits into a number of in-app endpoints