What happened on the nightmare Monday?

October 13, 2017

We made a mistake and lost some important data from the Automation Workflows. The lost data  tracked the specific workflow steps a subscriber was on.

It didn’t affect subscribers who already completed the workflow. However, those who were in the queue no longer had data connected with the step they were currently on.

We’re truly sorry. We know how important the automation is for your business. We understand that our mistake might have caused lost sales or other damage. We promise to do everything possible to prevent this situation from happening in the future.

Here are more details:

How did it happen?

  • On Oct 9 an update for better system performance was released.
  • It caused some Automation Workflows to stop.
  • Our developers reacted fast and started fixing the issue.
  • However, they made a mistake in a database query that deleted some data which tracked the workflow steps of queued subscribers.
  • We tried to restore the data from backups. We do backups every day and save them for a week. However, it turned out that backups exactly for that database silently failed. It meant we didn’t have the data to restore.

What did we do next?

  • It took us almost 24 hours to try out all possible solutions data restoration approaches. We managed to restore around 50% of affected automation workflows based on the last email sent to the subscriber. However, we could not restore longer and more difficult workflows.
  • We decided to stop all automations that we couldn’t restore. This prevented the risk of sending all workflow emails from the beginning to the subscribers who were somewhere in the middle of the workflow.
  • We informed all the accounts (that had their automations stopped) about the issue, and the steps they could take to make sure their automations continue to work.Here’s the email we sent.
  • Our recommendation was to make a copy of the automation workflow and turn it ON. This guarantees that your workflows work well for all new subscribers.
  • We continue to help everyone who needs our assistance to solve this issue. In which case please contact us by live chat or email (info@mailerlite.com). We will investigate your account and determine if it’s possible to restore some of the stuck subscribers. Each workflow is unique and can have not only emails but other actions and conditions, this is why it needs individual approach.

What are we doing to prevent such cases in the future?

  • First of all, we already fixed the backups issue.
  • Now we’re working on a solution for better backup monitoring so that a silent backup failure situation never happens again.
  • Starting today our development team is switching to hackathon mode and will work through the weekend to improve our testing environments and procedures.

Thank you for your trust!

In the last few days we received lots of supportive emails and messages from you. It really helps us to stay focused and overcome this crisis.

Some users have taken an extra step to try to help us and other users. I’d like to share a blog post by George Donnelly: Turn the MailerLite Data Loss Into a Marketing Win

Another user Alexis Anne created a video on how she solved the issue.

Again, I sincerely apologize. This won’t happen again.

Enjoyed the post?

Subscribe to our newsletter and get the latest updates.

Thank you!

You made the right choice.

  • Andrew

    Well, at least it wasn’t a data breach. If that ever happens, ML is sunk, and you’d have some VERY pissed off users. I trust you’ve got the best security going to make sure that doesn’t happen.

  • Fail , but shit happens . can somebody tell me if my old workflows are working, i just switched them ON.

    • Add an email, yours would work, to the respective group whose automation you want to test. Did you get the first email in the sequence? If yes, then it is working.

    • Ignas Rubezius

      It should work fine for all new subscribers. Those who were in the middle of the workflow before the data loss, will start the workflow from the beginning now.

      If you want us to check you account and confirm, just write us to info@mailerlite.com or in the live chat.

  • These things happen, Ignas and Team Mailerlite. I appreciated the help and confirmation I got for 2 of my affected workflows. It felt like we were working together in the same city and not in 2 different continents.

    At least there was no security breach or compromise of data.

    I remember a few years back, a web developer at the company that hosted my Mom’s art website deleted it from their server during a cleanup and didn’t even know it until I pointed out the website was missing! Luckily there was a backup but in this case, the owner didn’t know that his employee had made a mistake.

    Considering my Mom was one of his first clients when he set up his company, you can imagine how he felt and the steps he took to make up.

    At least Team Mailerlite discovered what had happened and took immediate action.

  • I must say that I really wish the option to simply turn on the affected workflows and have everyone in queue start from the beginning had been mentioned in the initial email. I would have much preferred that option to the duplicate option. I followed the instructions for the only option given and that made things even worse.

  • Meg Konovska

    You, guys, are amazing! <3

    I do feel for you, as I have experienced data backup failure myself. Your support, spirit, and professionalism are outstanding, and your transparency regarding the issues only solidifies the trust I have for you. Thank you! <3

  • Sooo transparent even with “faults”. YOU ARE GREAT !! Thank You !

  • Harris Roberts

    Just a month later automations are broken again. I want to love MailerLite but Shopify integration updates should be second to the main features. The service has become less stable recently. It is even more difficult when communication is delayed and unreliable.

    • Ignas Rubezius

      Automations are not broken. There were some delays in sending, but it was temporary. We’re doing our best to keep MailerLite stable during this busy season.