GitHub Availability Report: September 2024

In September, there were three incidents that caused performance degradation across all GitHub services.

September 16th 21:11 UTC (takes 57 minutes)

On September 16, 2024, between 21:11 UTC and 22:08 UTC, the GitHub Actions and GitHub Pages services were impacted. Customers deploying Pages from a source branch experienced delayed executions. We have determined that the root cause is a misconfiguration in the service that manages runner connections, which has resulted in CPU throttling and performance degradation in this service. Promotional orders experienced an average delay of 23 minutes, with some orders experiencing delays of up to 45 minutes. During the course of the incident, 17% of trips experienced a delay of more than five minutes. At peak times, up to 80% of journeys were delayed by more than five minutes.

We mitigated the incident by redirecting runner connections away from the misconfigured nodes starting at 21:16 UTC. In addition to resolving the configuration issue discovered as a result, we have enhanced our general monitoring to reduce the risk of a similar recurrence and to reduce our time to automatically detect and resolve such issues in the future.

September 24th 08:20 UTC (takes 44 minutes)

On September 24, 2024, from 08:20 UTC to 09:04 UTC, the GitHub Codespaces service experienced an interruption in network connectivity, resulting in an error rate of approximately 25% for the outage period. We traced the cause to an interruption in network connectivity caused by the Source Network Address Translation (SNAT) port exhaustion after a deployment, resulting in individual codespaces losing their connection to the service. To mitigate the impact, we increased port allocations to provide enough buffer for more outbound connections shortly after deployment. We will be expanding our outbound connectivity in the near future and adding enhanced network capacity monitoring to prevent future regressions.

September 30th 10:43 UTC (takes 43 minutes)

On September 30, 2024 from 10:43 UTC to 11:26 UTC, GitHub Codespaces customers in the Central India region were unable to create new codespaces. CVs were not affected and there was no impact to customers in other regions. We traced the cause to storage capacity constraints in the region and addressed this by temporarily redirecting create requests to other regions. We then added additional storage to the region and redirected traffic back. We also identified a bug that caused some available capacity to go unused, artificially limiting capacity and prematurely stopping creations in the region. We have now also fixed this bug so that available capacity scales as expected according to our capacity planning forecasts.

Please follow our Status page for real-time updates on status changes and post-incident summaries. To find out more about what we’re working on, check out GitHub Engineering Blog.