This report was written by Coil's founder and CEO, Stefan Thomas, with input from the Coil team.
A few days ago, we sent out a routine account update email which unintentionally disclosed email addresses from other users in the To field. We immediately took steps to diagnose the issue and determine what we could do to mitigate the impact. However, once an email is sent, there isn't much that can be done to take it back. Once we identified and confirmed the root cause, we sent a follow-up email apologizing to our users. We then set out to further analyze what happened and develop a plan to prevent anything like this from ever happening again.
I'm writing this post to document the exact mistakes we made leading up to the incident. Obviously, this doesn't change the outcome, but we believe we owe the community an explanation, and perhaps seeing our analysis and response to an incident like this will help those of you who run an online service avoid making the same mistakes.
On November 16, 2020 at 6:02 pm PT using an internal admin tool we sent out an email notifying all Coil Users with verified emails about the updates which were made to our Terms of Service & Privacy Policy. This email exposed other users' email addresses in the To field, meaning the email recipients were able to see each other's email addresses within the same batch.
We made a seemingly innocuous change to our custom mailing list service used for account emails. This change triggered a different code path in our email provider's API which unexpectedly changed the handling of the To field. Because we only expected content changes—not a change in the email metadata—our testing process did not require a review of the To field.
In order to notify users via email about important account updates such as changes to our Terms of Service, we use a dedicated service which interfaces with the API provided by our email provider (Mailgun). We had used this service before without issue.
These previous Terms of Service & Privacy Policy email updates included a personalized greeting. This personalization was accomplished by using Recipient Variables, which is a Mailgun feature, to enable dynamic content in the body of the email such as: Hi %recipient.firstName%, where %recipient.firstName% would be replaced with the user's given name.
However, we recently changed our signup flow to no longer ask users for their name. This change was done both to simplify the signup process and to reduce the amount of personal information we collect. Because we no longer ask for a first name, we removed the personalized greeting and replaced it with a simpler generic greeting: Hi,
The following code changes were done to accomplish this:
1. Update to the greeting
2. Removal of the recipient-variables
The change was then put through our code review process. In addition to the original author, two other Coil engineers and one other team member reviewed the change before it was approved. It was then tested against a local database containing fictional test accounts. During the testing of the emails we were focused on testing the formatting and content of the emails and did not notice that there were multiple email addresses in the To field.
Since the formatting was correct and links were working, we moved forward with sending the emails to users notifying them of the Terms of Service and Privacy Policy changes. When this notification went out the extremely unfortunate impact was that users were able to see the email addresses of other users in the same batch.
The logic in the above code calls a function emailAllUsers and passes in the emails object. That function then calls the /messages endpoint from Mailgun's API. This endpoint is used for both normal emails and batch emails. We did not expect the absence of the recipient-variables parameter to affect anything other than disabling the templating feature. However, unbeknownst to us, Mailgun's API documentation states:
"Warning: It is important when using Batch Sending to also use Recipient Variables. This tells Mailgun to send each recipient an individual email with only their email in the to field. If they are not used, all recipients’ email addresses will show up in the to field for each recipient."
When we removed the Recipient Variables we implicitly and unexpectedly changed the semantics of the To field which caused all of the email addresses to be populated within each batch. We want to make it perfectly clear that we do not blame Mailgun for our incorrect usage of their API.
These are the immediate steps taken but we will continue to analyze and learn from this incident and implement further improvements to our processes.
Why did we use the To field instead of the Bcc field?
Was this not tested?
What is Coil doing to prevent this from happening in the future?
Did you notify all the users whose emails had been leaked?
Aside from the mistakes we made, there are some things that went right.
I'd like to once again offer my deepest apologies to our users. I let you down. Our entire team and I are committed to making sure we never let anything like this happen again.
Next, I'd like to thank you for your understanding. Many of you have reached out to us to express your empathy and support and that means a lot to us. Thank you!
Finally, I'd like to say that I hope that our mistake does not reflect poorly on the Web Monetization community overall. Web Monetization is a proposed open standard and there are many websites that support it, but that does not automatically mean those sites are endorsing Coil. This incident only underscores the need for more Web Monetization providers.
If you require any assistance or help, please contact us at [email protected].
