Data Platforms
Jan 23, 15:30 UTC Resolved - We identified the cause of elevated latency impacting some databases in us-east-1 region between 15:30–15:35 UTC as a sudden surge of connection attempts that hit OS-level connection limits on our proxy layer. This resulted in slower new connection establishment and increased latency for some requests. Databases were not impacted. We are implementing additional proxy-level metrics and safeguards to detect and manage similar edge cases earlier.
Dec 10, 12:08 UTC Resolved - Issue has been identified and replicas were successfully reconnected. Dec 10, 11:50 UTC Investigating - We identified an issue in Regional Databases in US-East-1 where database replicase have connectivity issues with each other. Other regions are not impacted. Global databases are not impacted. We are working on the issue.
Dec 5, 09:18 UTC Resolved - This incident has been resolved. Dec 5, 09:03 UTC Investigating - Upstream provider confirmed an incident. We are investigating the impact and potential resolutions.
Dec 1, 07:00 UTC Resolved - We identified and fixed a bug that could cause messages with Flow Control enabled to be delayed longer than their configured delay, resulting in unexpectedly long pending times. The fix is in place and the issue should not recur. If you’re still seeing unusually long-delayed messages, please contact support@upstash.com and we can help with remediation.
Oct 20, 18:50 UTC Resolved - A fix has been deployed as a workaround so that our systems are not affected from the ongoing incident of the cloud provider Oct 20, 18:07 UTC Update - Developer API (api.upstash.com) is having availability issues alongside Upstash Console. While we are monitoring the underlying cloud provider's status updates, we are also working on a remediation. Oct 20, 15:05 UTC Monitoring - Console is back to normal again. We are currently monitoring. Oct 20, 14:52 UTC Investiga...
Oct 20, 07:00 UTC Resolved - As a side effect of an incident on the underlying cloud provider, Upstash Console has had availability issues between 07:00UTC and 09:23UTC. Only Upstash Console is impacted, Upstash products remained operational.
Oct 10, 15:46 UTC Resolved - Between 15:46–15:55 UTC, some client connection attempts to databases in us-east-1 timed out due to unexpected high load on a server. The node was recovered at 15:50 UTC, and the updated DNS record propagated by 15:55 UTC. Services are operating normally.
Sep 9, 12:00 UTC Resolved - Impact: A subset of clients connecting through the eu-central-1 region experienced increased error rates and timeouts when accessing certain databases. Clients in us-west-2 were also briefly affected. The issue was limited in scope and did not impact other regions. Root Cause: During an ongoing migration to improve database routing reliability, a configuration step was applied inconsistently across regions. Resolution: Our monitoring alerted us within minutes, and t...
Aug 18, 12:37 UTC Resolved - Between 12:37 UTC and 12:50 UTC, an overload in the connection proxying system resulted in instability for a subset of databases. The root cause was identified as a misconfiguration in the routing rules, which caused certain requests to experience timeouts during the initial phase of a gradual deployment. Upon detection, the deployment was immediately rolled back, restoring normal service. We are reviewing our deployment and configuration validation processes to prev...
Jun 26, 07:34 UTC Resolved - This incident has been resolved. Jun 26, 07:12 UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun 26, 06:53 UTC Identified - Some users may experience login issues due to a disruption in our authentication provider. We’re actively monitoring the situation.
Jun 12, 16:07 UTC Resolved - This incident has been resolved. Jun 12, 14:40 UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun 12, 14:28 UTC Investigating - We are currently investigating this issue.
Jun 11, 10:02 UTC Resolved - This incident has been resolved. Jun 11, 09:20 UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun 11, 08:24 UTC Update - We are continuing to investigate this issue. Jun 11, 06:51 UTC Investigating - We are currently investigating this issue.
Jun 10, 12:08 UTC Resolved - This incident has been resolved. Jun 10, 07:45 UTC Monitoring - A fix has been implemented and we are monitoring the results. Jun 10, 07:23 UTC Investigating - We are currently investigating this issue.
May 22, 17:41 UTC Resolved - This incident has been resolved. May 22, 16:44 UTC Monitoring - A fix has been implemented and we are monitoring the results.
Apr 18, 13:18 UTC Resolved - This incident has been resolved. Apr 18, 12:46 UTC Monitoring - A fix has been implemented and we are monitoring the results. Apr 18, 12:36 UTC Update - We are continuing to investigate this issue. Apr 18, 12:36 UTC Investigating - We are currently investigating this issue.
Apr 2, 21:12 UTC Resolved - QStash service and Event logs are fully functional without any remaning issues. Apr 2, 20:20 UTC Update - Monitoring: Main QStash service is back to normal. Event logs service is back online but events will be lagging a few mins. Apr 2, 17:33 UTC Monitoring - Main QStash service is back to normal. Event logs are still temporarily unavailable. Apr 2, 16:33 UTC Update - We are continuing to investigate this issue. Apr 2, 16:29 UTC Update - We are continuing to i...
Mar 12, 10:16 UTC Resolved - This incident has been resolved. Mar 12, 10:08 UTC Monitoring - A fix has been implemented and we are monitoring the results. Mar 12, 09:43 UTC Investigating - We are currently investigating this issue.
Feb 25, 21:23 UTC Resolved - This incident has been resolved. Feb 25, 20:54 UTC Monitoring - A fix has been implemented and we are monitoring the results. Feb 25, 14:03 UTC Identified - Regional AWS eu-west-1 cluster is experiencing performance degradation, and we are adding more resources to the cluster.
Feb 20, 08:00 UTC Resolved - We experienced a very short period of API downtime for the incoming requests to QStash due to urgent maintenance to ensure the stability and performance of our services. Our team acted quickly to address the issue, and everything is now fully operational.
Feb 6, 10:10 UTC Resolved - Latest QStash release caused some Workflow runs to fail due to a bug in Workflow URL detection mechanism. Affected workflows did not start at all and moved directly to DLQ. These workflow runs, which show "detected non-workflow destination" message in the response body, can be retried from DLQ. This was a partial failure, lasted from 10:10 to 11:00 UTC, not all users' workflows were affected.