Post-Mortem of email outage at The Podcast Consultant on Sept. 23rd, 2024

On Sept. 23rd, 2024  at 9:49am Eastern Time our domain provider Squarespace caused a major outage in our email system, which got resolved by 11:18pm Eastern Time.

Ever since Squarespace bought Google Domains on June 15th, 2023, Squarespace is the official reseller of our Google Workspace and our domains at The Podcast Consultant.

On Sept. 23rd, we attempted to create an additional Google Domain on our subscription. This caused a bug with Squarespace, making it impossible for our team to log into our Google Accounts.

It seems like our issues on Sept. 23rd were not an isolated instance as Squarespace’s email support made us aware that they are experiencing a high volume of similar support requests.

In its function, Squarespace provides us with Google Workspace services, making it our main provider for emails, domains, cloud storage, and more.

While the email functionality has been restored by now, we are still experiencing issues with our cloud storage which we expect to get resolved soon. In order to have redundancy when handling client files, we purchased cloud storage with Dropbox years ago, meaning your files and our handling them will not be impacted by this issue. 

How we make sure this won’t happen again

In order to make sure clients will not feel the impact of external outages in the future, here are the steps we will take:

  • We will change our provider for above mentioned services. We are currently investigating which alternative will be best.
  • Should we ever experience such an outage again, we will first proactively reach out to any clients that have an episode airing due the day-off or the following day. We will reach out via our backup email address and keep communication open.
  • We will update the top banner on our website to indicate an ongoing outage and how to reach out to us.
  • We will publish a post on our LinkedIn page letting you know about an ongoing outage. 
  • If the outage persists longer than three hours, we will send an email to our main contact at all active clients, letting them know about the issues. We will send an email update once the issues are resolved.

Going forward, we will make sure that we don’t rely on a single provider of a service at any point in our production chain. We appreciate your understanding and hope that the transparent communication around this post-mortem can restore your trust in us.