There was a misconfiguration on our load balancer where it automatically updated to the next minor version at 7:00pm PDT. In doing so, new required configuration options were not set, leading the load balancer to not restart successfully. We received a pager alert that the application was down, identified that it was the load balancer, and set about determining the issue. We then downgraded to our previous version, and after doing so, it started successfully and the application came back fully online at 7:57pm PDT. Action items resulting from this outage are: