--- service: "Apwide" status: "up" reports_last_hour: 0 reports_last_12_hours: 0 baseline_reports_per_hour: 0 checked_at: "2026-10-31T02:31:21.182Z" url: "https://instatus.com/now/ko/apwide.com" official_status_page: "https://apwide.statuspage.io" website: "https://www.apwide.com" --- # Apwide 다운되었나요? 아니요, 정상이에요. 최근 1시간 동안 0건 보고됨. ## 공식 업데이트 - **Security Patch** (NOTSTARTEDYET, 2026-07-17T18:30:00.000Z): ## Summary On **2026-07-19**, our production services experienced an extended outage following a planned database security maintenance announced by OVH. While the maintenance was initially expected to last no more than **30 minutes**, an infrastructure failure at the OVH datacenter significantly prolonged the interruption. After assessing the situation and the available recovery options, we activated our Business Continuity Plan \(BCP\) by restoring the production database from the latest available backup and upgrading it to PostgreSQL 18. All services were fully restored on **2026-07-20 at 11:15 UTC**. ## Customer Impact During the incident, **Golive** and **Time Squad** were unavailable for approximately **20 hours**, preventing customers from accessing the applications and performing normal operations. As part of the recovery process, we restored the database from the latest available backup. The most recent backup had been completed **25 minutes before the outage began**, resulting in a **potential data loss window of up to 25 minutes**. A review of the database audit logs showed that, due to the weekend period, only a limited number of write operations occurred during this timeframe. These updates were primarily related to deployment and monitoring activities, significantly reducing the effective customer impact. ## Root Cause The root cause of the incident was a **hardware failure affecting a storage device in the OVH datacenter** during a planned security maintenance operation on the managed PostgreSQL infrastructure operated by Aiven on behalf of OVH. Initially, the incident was communicated as an extension of the planned maintenance, delaying visibility into the actual nature and severity of the problem. Several hours later, OVH published a dedicated incident confirming that the outage was caused by a hardware failure rather than the maintenance activity itself. ## Incident Timeline ### Context The incident occurred during a planned maintenance window announced by OVH on **2026-07-16**. OVH informed customers that they would apply a security patch to our production database over the weekend. The expected service interruption was **no longer than 30 minutes**. ### 2026-07-19 **14:39 UTC** Our monitoring systems and availability probes detected an outage affecting our production services. As this matched the scheduled maintenance window, our support engineers considered the interruption expected. **16:00 UTC** After allowing an additional one-hour buffer beyond the planned maintenance window, our monitoring continued to report that the database was unavailable. Our support engineers checked the OVH status page, which still showed the maintenance as ongoing with no additional information. The OVH control panel also reported the database status as **"Updating"**. At this stage, the extended downtime was still considered part of the maintenance. **17:15 UTC** The OVH status page remained unchanged. To gather additional information, our engineers consulted the public OVH Discord community, where other customers reported experiencing the same issue: databases remained in the **"Updating"** state while waiting for the security patch to complete. **21:20 UTC** With the announced maintenance window approaching its end, the database still unavailable, and business hours about to begin in the Australia/Oceania region, our support engineers opened a **Priority 1 \(P1\)** support ticket with OVH. **21:25 UTC** OVH acknowledged receipt of the incident and confirmed that the ticket had been assigned. **21:41 UTC** OVH confirmed an incident involving **Aiven**, the service responsible for managing the cloud database platform. They indicated that the issue was related to the planned maintenance operation and that the outage was taking significantly longer than the initially announced 30 minutes. They committed to providing further updates. [https://public-cloud.status-ovhcloud.com/incidents/f02sjn0qw27f](https://public-cloud.status-ovhcloud.com/incidents/f02sjn0qw27f) At this point, we updated the Golive and Time Squad status pages to report a service outage. ### 2026-07-20 **00:46 UTC** After more than ten hours of downtime without any meaningful update, our support engineers requested additional information from OVH to determine whether our Business Continuity Plan \(BCP\) should be activated. During this investigation, we also discovered that the OVH user interface did not allow us to create a database fork from an existing backup. **04:55 UTC** As no progress had been communicated publicly, our support engineers contacted OVH support again and also reached out to the on-call support team. **05:30 UTC** OVH support confirmed that the planned maintenance was still in progress but could not provide further details regarding the broader incident. We expressed our concerns about the lack of communication and requested additional information to support our operational decision-making. We also requested that OVH create a dedicated public incident rather than continuing to associate the outage solely with the maintenance operation. **07:20 UTC** Our engineers discovered that creating a database fork from a backup using the OVH REST API was possible, even though the option was unavailable in the web interface. The most recent backup had been created at **14:15 UTC**, while the outage started at **14:40 UTC**, resulting in a potential data loss window of approximately **25 minutes**. After reviewing the database audit logs, our engineers confirmed that only a limited number of write operations had occurred during this period, primarily related to deployment and status monitoring activities due to the weekend. Based on this assessment, management decided to activate the Business Continuity Plan. **07:48 UTC** OVH published a dedicated incident, confirming that the outage was caused by a hardware failure rather than simply an extended maintenance window. [https://public-cloud.status-ovhcloud.com/incidents/bn54j3232tlv](https://public-cloud.status-ovhcloud.com/incidents/bn54j3232tlv) **08:00 UTC** Because our PostgreSQL version did not support database forking through the OVH web interface, we took advantage of the BCP activation to upgrade the database to **PostgreSQL 18**, a version that had already been successfully validated in our development and staging environments. **08:50 UTC** The restored database became available on PostgreSQL 18. Database integrity and sanity checks completed successfully, and Time Squad was restarted. **08:55 UTC** Time Squad was fully operational. Our status page was updated, and Golive services were restarted. **09:05 UTC** Despite several restart attempts, some Golive components failed to start correctly. Our engineers began investigating the root cause. **09:30 UTC** The investigation identified the issue as a backlog of asynchronous Atlassian webhook events \(issue creation, updates, etc.\) that had accumulated during the outage. Once connectivity was restored, Atlassian attempted to replay all missed events simultaneously, creating a significant load on our infrastructure before caches had been rebuilt. **09:55 UTC** A temporary mitigation was implemented by blocking incoming webhook traffic. This allowed Golive to complete its startup sequence and resume serving normal transactional traffic. The status page was updated accordingly. **10:13 UTC** A performance improvement that had already been validated in our development and staging environments and scheduled for an upcoming release was deployed to production. Webhook traffic was then re-enabled, and the combination of rebuilt caches and the performance improvement proved effective. **11:15 UTC** After more than one hour of stable event processing and normal system behavior, the incident was declared resolved. The status page was updated to indicate that all systems were fully operational. ## Actions Taken During the incident, we initially could not activate our Business Continuity Plan because the OVH web interface did not allow us to create a database fork from a backup. This limitation was caused by the legacy PostgreSQL version running on our production environment. As part of the recovery process, we upgraded our production database to **PostgreSQL 18**, the latest version supported by OVH and already validated in our non-production environments. This upgrade removes this limitation and enables faster recovery should a similar incident occur in the future. ## Follow-up Actions * Work with OVH to improve communication during major incidents, particularly regarding impact assessment and incident classification. * Request earlier publication of dedicated incident reports instead of extending planned maintenance notifications when an unexpected failure occurs. * Continue reviewing and improving our Business Continuity Plan activation criteria to enable faster decision-making when infrastructure providers experience prolonged outages. * Follow up with OVH on the restoration of the original database service, which remains unavailable at the time of writing this post-mortem. ## Apwide 장애 FAQ ### 지금 Apwide 다운됐나요? 아니요, 정상이에요. 최근 1시간 동안 0건 보고됨. ### Apwide이(가) 모두에게 안 되나요, 저만 안 되나요? 당신만 겪는 문제일 수 있어요. Apwide 제보가 평소 수준이니, 페이지를 새로고침하거나 연결을 확인하거나 로그아웃 후 다시 로그인해 보세요. ### Apwide에서 사람들은 어떤 문제를 제보하나요? 지난 1시간 동안 Apwide에서 무엇이 안 되는지 말한 사람이 없습니다. 안 되는 게 있다면 위에서 선택해 다른 사람들도 볼 수 있게 해주세요. ### Apwide의 마지막 장애는 언제였나요? Apwide 제보는 최근 30일간 급증하지 않았어요. ### Apwide에 상태 페이지가 있나요? 네. Apwide은(는) 공식 상태 페이지에 업데이트를 올리며, 현재 상태는 정상입니다. [상태 페이지로 이동](https://apwide.statuspage.io/) ### Apwide이(가) 안 되면 어떻게 하나요? 이 페이지와 Apwide의 상태 페이지에서 업데이트를 확인하고, 몇 분 기다린 후 다시 시도하세요. 재설치는 하지 마세요. 장애라면 해결은 서비스 쪽에서 합니다. ### Apwide 문제는 어떻게 제보하나요? 이 페이지 상단의 “저도 같은 문제가 있습니다”를 누르고 무엇이 안 되는지 선택하세요. 제보는 1분 안에 실시간 차트에 표시되며, 댓글로 어떤 상황인지 다른 사람에게 알려줄 수도 있습니다. ### Instatus Now는 어떻게 알죠? 이 페이지의 사용자 제보와 매분 확인하는 Apwide 공식 상태 페이지를 함께 살펴봐요.