Subscribe to

The Unconventional Guide to Work

Life’s too short to be bored at work. We can help.

Unexpected WorkflowMax Outage

The application has been slow and unresponsive for the past 15 minutes, the team are currently investigating the issue and will look to put a resolution in place as quickly as possible.

We will provide an update shortly via the blog and twitter on when we expect the issue to be resolved.

We apologise for any inconvenience this has caused.

UPDATE - 11:53 - We are taking the database offline currently and will look to restart the database.

UPDATE - 12:30 - We're currently performing a number of checks on the database. We hope to have an update shortly

UPDATE - 12:45 - We have brought the database back up successfully, we have found the cause of the issue and are performing checks across several tables to confirm that we are all okay to bring the application back up online

UPDATE - 12:55 - WorkflowMax is now back up online. We apologise for the massive incovenience this outage has caused. We will follow up with a further blog post shortly detailing what the cause of the issue was and how this wil be avoided in future.

UPDATE - 2:30pm - One of our database servers failed to complete a routine maintenance task which resulted in a temporary performance degradation on the core invoicing table within the database. This resulted in the database queuing requests from the application which in turn resulted in the degrading performance of the application.

To resolve the issue the operations team took the database offline to ensure all loading was removed while we investigated the cause and subsequent resolution.

There was no loss of data as a result of this issue however the application was offline for approximately 1 and three quarter hours. We have a number of levels of redundancy within our hosting environment that typically means our system is not impacted by database issues. Unfortunately in this situation there was no way for the automated recovery to resolve the issue.

We are currently planning an upgrade to the database platform that supports more advanced resilience, which should further reduce the risk of a similar outage.