Outage ng Fluxer noong Agosto 18

Issues with platform connectivity

1. Naayos · 4:18 AM

   On 18 August, Fluxer was down for about 15 minutes, from 03:54 to 04:05 UTC. Traffic reaches us through Caddy. Caddy terminates TLS and forwards into Kubernetes via ingress-nginx. Page loads are served by app-proxy. Everything else goes to the API. On every page load, app-proxy refreshed a cached discovery document before responding. It fetched that document from its own public address, <https://api.fluxer.app/.well-known/fluxer>. The request therefore travelled back out through Caddy rather than staying inside the cluster. A rolling API deploy made that endpoint briefly unavailable. The fetch had a five second timeout, so every page load began taking five seconds. Those slow requests filled up ingress-nginx. They then filled Caddy's queue of pending connections, and Caddy stopped accepting new ones. At that point app-proxy could no longer reach the API at all. The route it needed ran through the front door it had just blocked. The outage held itself open and could not recover on its own. We restarted Caddy on all three web nodes to clear the backlog. That restored service. The root cause was in app-proxy. Its background refresh already backed off when the API was unhealthy. The page load path, however, called the refresh directly and skipped that backoff. Every request therefore paid the full timeout. Page loads now serve the cached document. They make no network calls at all while a cached copy exists. The first fetch after startup is rate limited, so an unavailable API cannot cause a retry storm.

2. Naayos · 4:10 AM

   This incident has been resolved.

3. Iniimbestigahan · 3:56 AM

   We are currently investigating this incident.