Our apps and apis were not able to connect to our main authentication database, causing many requests to fail. This issue was triggered when our primary authentication database ran out of storage space, causing the server to become unresponsive. The read replicas did take over, but was only a solution for a few minutes, as required writes to the primary began to fail.
This issue was resolved by increasing the storage quota for the primary authentication database.
To prevent this from happening again, we have implemented warnings at multiple intervals to be notified when storage may be getting full. We've also enabled a feature our cloud provider offers to auto expand database storage automatically when needed.
