Resolved
All workspace starts now occur in the *41 and *41xl clusters. We think the issue is resolved.
Monitoring
All workspace starts for XL clusters are entirely running on *41xl. 50% of workspace starts for other clusters land on *41, which also contains the fix for performance degradation.
Monitoring
We've received positive feedback from customers who were previously experiencing issues. Additionally, metrics look promising, too. We've increased the traffic shift for XL clusters to 50%.
Monitoring
We've deployed a fix for workspaces that run in XL clusters.
25% of new XL workspaces will start on *41xl, which include a fix for this issue: https://github.com/gitpod-io/gitpod/issues/9406
Identified
We've identified a likely suspect and are testing multiple fixes internally.
Investigating
Customers have reported that workspaces which were previously running well now have performance issues. We are inspecting recent changes to the system, and are running internal tests.