Resolution: Engineers have fully resolved the incident through a restart of the Scheduler Services. Engineers continue to monitor the system.
Steps to Mitigate: Enhanced alerting and monitoring has been added to the scheduler service. Engineers are implementing an auto restart process as part of the fail over procedures to mitigate future incidents.
Update: Engineers have identified the root cause and all jobs are now being queued to be processed. We have scaled out services to address t...