Authentication database outage

Resolved
Major outage
Started 5 months ago Lasted about 1 hour

Affected

Planning
Music Stand
www.worshiptools.com
Presenter
Updates
  • Resolved
    Resolved

    Our apps and apis were not able to connect to our main authentication database, causing many requests to fail. This issue was triggered when our primary authentication database ran out of storage space, causing the server to become unresponsive. The read replicas did take over, but was only a solution for a few minutes, as required writes to the primary began to fail.

    This issue was resolved by increasing the storage quota for the primary authentication database.

    To prevent this from happening again, we have implemented warnings at multiple intervals to be notified when storage may be getting full. We've also enabled a feature our cloud provider offers to auto expand database storage automatically when needed.

  • Identified
    Identified