On December 19, 2024, a change was implemented in our application proxy that inadvertently caused all uncached requests to return a 404 error. This incident began at 18:12 UTC (11:12 MST) and was promptly detected through our alert systems. The issue arose during an intended rearchitecting of the proxy, aimed at removing a component known to be prone to failure. Unfortunately, the removal led to an unforeseen consequence where the absence of this component resulted in failures for uncached requests.

By 18:29 UTC (11:29 MST), our team successfully addressed the issues with the application proxy, restoring normal operations for uncached requests. However, to ensure that erroneous 404 responses did not persist, it was necessary to purge the CDN cache. This process involved carefully removing all cached 404 errors to prevent them from being served to end-users. The purging process was completed at 19:08 UTC (12:08 MST), ensuring that the content delivery network was fully updated with the corrected responses.