CockroachDB

Feb 9, 15:20 UTC Resolved - Standard and Basic clusters continue to function normally. Feb 9, 04:15 UTC Monitoring - We have resolved an internal misconfiguration, and see that Standard and Basic clusters are functioning normally again. We will continue to monitor to ensure that the error does not recur. Feb 9, 03:04 UTC Investigating - We are investigating connectivity issues to Standard and Basic clusters on AWS.

Feb 2, 19:46 UTC Resolved - We have deployed and verified a fix. Advanced clusters should now create within normal time frames. Feb 2, 19:31 UTC Update - We are continuing to work on the fix for this issue. Next update will be at 15:00 US/Eastern. Feb 2, 17:57 UTC Update - We are preparing a fix for deployment, which should restore AWS cluster creation times to normal levels. We will update the status page when the fix is deployed, or by 14:30 US/Eastern with more information. Feb 2, 16:56 U...

Oct 31, 22:09 UTC Resolved - All cluster operations are working normally. Oct 31, 20:18 UTC Monitoring - We have resolved the underlying issue that was preventing cluster creation and other cluster operations. We have re-enabled cluster creation on all clouds. We will monitor the situation to ensure that everything continues to operate normally. Oct 31, 19:58 UTC Update - We are continuing to work on a fix for this issue. Next update will be by 4:30 PM US/Eastern time. Oct 31, 18:59 UTC Update -...

Oct 30, 16:05 UTC Resolved - Cluster creation is operating normally, and all delayed clusters have finished creation. Oct 30, 15:25 UTC Monitoring - We have applied a mitigation and observe that cluster creations are proceeding again. We will monitor all impacted cluster creations to ensure they succeed. Oct 30, 14:50 UTC Investigating - We are investigating errors that cause creation for Advanced clusters to take longer than usual. Standard and Basic cluster creation is operating normally.

Oct 21, 14:56 UTC Resolved - The earlier AWS outage in the us-east-1 region has been fully resolved. All CockroachDB Cloud operations, including cluster creation, scaling, and backups, are functioning normally again. We’ve verified recovery across affected systems and customer clusters, and normal operations have resumed. Oct 20, 22:52 UTC Monitoring - AWS has resolved the underlying outage in the us-east-1 region, and we’re seeing recovery across CockroachDB Cloud operations. Cluster creation, ...

Sep 24, 16:00 UTC Resolved - Google has reported the issue has been resolved and we've verified cluster operations are functional. Sep 24, 12:31 UTC Identified - We are currently experiencing an incident impacting certain cluster operations on Google Cloud Platform due to an known ongoing Google Kubernetes-related issue. Customers may encounter errors when creating or editing clusters. Our engineering teams are actively monitoring the situation and communicating with Google regarding resolution.

Jul 29, 23:35 UTC Resolved - We have confirmed resolution of the incident impacting SQL availability for the affected cluster. SQL availability is restored and the associated cluster is now healthy. Jul 29, 23:17 UTC Monitoring - Correction from the original status update. This issue appears to have been isolated to cluster crl-prod-6rw. We have identified the cause, mitigated and confirmed that cluster availability has returned. We are currently completing additional validations. Jul 29, 23:08 ...

Jul 22, 04:22 UTC Resolved - LetsEncrypt has indicated the issue has been mitigated. We have validated cluster creation behaviour is now returning to normal. Jul 21, 21:39 UTC Identified - We are aware of an ongoing Incident with dependent service "Let's Encrypt": https://letsencrypt.status.io/, which may impact cluster creation and add region operations for CockroachDB Cloud Advanced clusters. We will update this status page with more information as it becomes available.

Jul 3, 17:39 UTC Resolved - We have corrected a misconfiguration which was preventing cluster operations from succeeding. We are monitoring the success of cluster operations already in progress and new operations. Jul 3, 16:58 UTC Investigating - We are investigating an issue that may prevent or delay cluster operations from succeeding.

Jul 2, 20:46 UTC Resolved - All cluster operations are succeeding normally at this time. Jul 2, 19:32 UTC Monitoring - We continue to see cluster operations succeeding, and will actively monitor to ensure the issue does not reoccur. Jul 2, 18:50 UTC Identified - We have identified the issue preventing cluster operations from proceeding, and we see that operations are moving forward. We will next update this incident by 16:00 EDT. Jul 2, 17:42 UTC Investigating - We are investigating cluster ...

Jun 27, 22:36 UTC Resolved - We have identified and rolled out a fix for the issue impacting cluster creation in AWS. This incident is now resolved. Jun 27, 22:29 UTC Identified - We have identified an issue impacting Basic and Standard tier cluster creation in AWS. We have conducted an initial investigation and actively working to resolve the issue.

Jun 12, 21:31 UTC Resolved - Google has reported that mitigations have been applied and services are now recovering. Jun 12, 19:54 UTC Monitoring - Google has updated their public status page to indicate the underlying issue is resolved, and individual Google Cloud services are in the process of restoring full service. We are continuing to monitor the issue and impact to CockroachDB Cloud. Jun 12, 19:08 UTC Update - In addition to cluster operations, backups to Google Cloud Storage, including ma...

Jun 4, 21:35 UTC Resolved - The issue that was preventing metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters has been fully resolved. Metrics are now loading as expected, and we’ve received confirmation from our service provider that the underlying issue has been addressed. We will continue to monitor for any signs of recurrence, but no further impact is expected at this time. Thank you for your patience. Jun 4, 20:19 UTC Monitoring - Metrics visuali...

Jun 1, 07:29 UTC Resolved - Incident Start AWS experienced a brief network blip in the eu-west-1, eu-west-2, and eu-central-1 regions. Clusters in these regions were unavailable between June 1, 19:29–19:35 UTC. Incident End The network issue affecting clusters in the impacted AWS regions has been fully resolved *automatically* as of 19:35 UTC. All affected clusters are now operating normally. Please contact support if you have any concerns.

Nov 19, 18:36 UTC Resolved - The Engineering team has isolated and resolved the elevated error rate for all affected clusters. Nov 19, 17:06 UTC Monitoring - The root cause of the issue was identified and resolved. Engineering is monitoring the situation. Nov 19, 16:48 UTC Investigating - We have received alerts indicating elevated error rates on isolated Standard and Basic clusters. The Engineering team is currently investigating the issue.

Nov 12, 01:00 UTC Resolved - On Tuesday Nov 12, at approximately 1am UTC, we received alerts of elevated errors due to an underlying host hardware failure. The workloads on this host were migrated to a healthy host and the errors resolved once the migration was complete.

Oct 24, 21:05 UTC Resolved - The upstream incident has been resolved, and we have confirmed that cluster creation is working normally. Oct 24, 20:06 UTC Update - We are continuing to monitor the upstream incident. Oct 24, 19:00 UTC Identified - We are aware of an AWS incident that is preventing successful completion of Advanced cluster creation and region addition. Once the cloud provider has resolved their incident, we will ensure clusters create successfully.

Sep 25, 14:30 UTC Resolved - We have deployed a mitigation and all clusters are available again. Sep 25, 14:15 UTC Investigating - We are aware of an issue affecting a small number of serverless clusters that is causing unavailability. Our SRE team is investigating.