We found regression in api reporting request that was causing significant amount of memory consumption impacting the nodes and entire system. The fix was been deployed and verified.
Posted Aug 23, 2022 - 20:44 CDT
Monitoring
We have confirmed performance has stabilized and we continue monitoring affected applications. We are planning to introduce significant improvements to our progress cluster during the next database maintenance.
Posted Aug 23, 2022 - 17:26 CDT
Update
The rollback was completed and services have been restored. We are continuing to investigate this issue.
Posted Aug 23, 2022 - 13:41 CDT
Update
The restart did not alleviate the issues. We are restoring the services and rolling back the changes.
Posted Aug 23, 2022 - 13:35 CDT
Update
We are conducting a short restart of services which can cause non-learner applications to be unavailable for a few minutes.
Posted Aug 23, 2022 - 13:17 CDT
Investigating
We are experiencing an issue with high load times within our non-learner BenchPrep applications. Learner applications are not affected. We are currently provisioning more resources and will continue to investigate.