INVESTIGATING Time: 2026-03-04 09:29 +01:00 Components affected: Bahrain We are currently investigating an issue affecting fraud-check lookup and calculation services in the Bahrain region. Clients may be experiencing 5XX errors on the fraud-check endpoint. Our team is actively investigating the cause. We will provide an update shortly.
IDENTIFIED Time: 2026-03-04 ~10:00 +01:00 (adjust to actual time if known) Components affected: Bahrain We have identified the cause of the issue. During routine maintenance on the Bahrain Elasticsearch cluster, a healthy node was inadvertently terminated, reducing the cluster below the minimum required nodes. This has caused disruption to fraud-check lookup and calculation services. Our team is actively working to recover the cluster and restore service. We will provide further updates as recovery progresses.
MONITORING Time: 2026-03-04 13:00 +01:00 Components affected: Bahrain We have switched DNS to our backup Elasticsearch cluster in the Bahrain region, restoring fraud-check lookup and calculation services. We are actively monitoring the environment to confirm stability. Clients should no longer be experiencing 5XX errors on the fraud-check endpoint. We will confirm full resolution shortly.
RESOLVED Time: 2026-03-04 13:25 +01:00 Components affected: Bahrain This incident has been resolved. Fraud-check lookup and calculation services in the Bahrain region have been fully restored via failover to the backup Elasticsearch cluster, confirmed at 13:25 +01:00.
Summary: On 2026-03-04, the Bahrain Elasticsearch cluster became unavailable following a routine maintenance operation in response to an AWS health recommendation. During node replacement, an autoscaling adjustment inadvertently terminated a healthy node, reducing the cluster below quorum. Manual recovery attempts were extended by a bootstrap script incompatibility with Amazon Linux 2023 and EC2 provisioning delays in the Bahrain region. Service was restored by switching DNS to the backup cluster.
Impact: Clients experienced 5XX errors on the fraud-check endpoint between approximately 09:29 and 13:00 +01:00 (~3.5 hours). Next steps: We are rebuilding the primary Bahrain Elasticsearch cluster, restoring data from backup, and will switch DNS back to the primary cluster once health is validated. We will also be reviewing our node replacement and bootstrap processes to prevent recurrence.
Posted Mar 04, 2026 - 17:33 UTC
This incident affected: Bahrain (Portal, API, Analytics).