GitHub Availability Report: August 2024

In August, an incident occurred that caused performance degradation across all GitHub services.

August 14, 23:02 UTC (takes 36 minutes)

On August 14, between 23:02 and 23:38 UTC, all GitHub services on GitHub.com were inaccessible to all users.

At 22:59 UTC, an incorrect configuration change was made to GitHub.com’s databases that impacted the database’s ability to respond to health check pings from the routing service. This resulted in these database hosts being considered unhealthy and the endpoint of the read-only production database becoming inaccessible. As a result, the application was no longer able to connect to critical data for read operations. This resulted in widespread impact on GitHub.com as of 23:02 UTC. There was no data loss or corruption in this incident.

We mitigated the incident by reverting the configuration change and confirming restored connectivity to our databases. At 23:38 UTC traffic resumed and all services were fully operational again. Out of an abundance of caution, we continued monitoring before resolving the incident at 00:30 UTC on August 15.

To prevent a recurrence, we have implemented additional safeguards into our database change management process. We also prioritize several repair points, such as: B. faster rollback functionality and greater resilience to dependency failures. Given the severity of this incident, all repair items are being worked on as a top priority.

Please follow our Status page for real-time updates on status changes and post-incident summaries. To find out more about what we’re working on, check out GitHub Engineering Blog.