Over the past week, users will have noticed a few issues happening with WorkflowMax and the way it connects with Xero. We think it’s only fair to give you an update on what is causing some of the pain you have been experiencing, and what we’ve done to fix it and how we’re going to prevent this issue in future.
The issue in a nutshell: From 1 July, for just over a week, users might have seen an error message which looks like the below image.When users re-connected WorkflowMax to Xero, you will have likely got the same error shortly afterwards when one of the background processes ran.This meant that several parts of WorkflowMax stopped syncing with Xero.
- Sending invoices and purchase order receipts to Xero wasn’t impacted, however importing payments against those invoices was
- Importing invoices for WorkflowMax Premium customers also stopped importing
- Any changes to clients were also not synced across to Xero
There were several causes of this issue which is why it was complex for our team to uncover and to understand.Recently security standards have changed for communication between internet applications, and globally all internet applications were required to upgrade to the latest or previous version.
We took the opportunity to upgrade all components to the newest version (TLS version 1.2) to ensure the highest level of protection of our users data. Parts of this upgrade were unsuccessful. This subsequently caused a few things to not work as intended such as third party apps that connect with WorkflowMax.
We resolved the issue on Thursday 12 July.
It’s important to note that there wasn’t any data loss. All data was still queued and will be processed once the connections have been refreshed. We expect that the backlog of data to process will take around 24-48 hours once the majority of users have reconnected.
We have changed the way that the connection is refreshed in the background so that in future we won’t have similar failures. We are also investigating the points of failure to see what changes we can implement to our systems to prevent this in future.
Why did the issue take so long to fix?
This issue had a few moving parts and we had had several teams working around the clock working on getting systems humming. Once we identified the causes and what needed to be changed to resolve the issue, we also needed to ensure that in fixing it, there would be no data loss.
On 10 July we released a fix which unfortunately didn’t work as intended. Despite our best efforts to fix this, we took the decision to remove the release and revert to the previous version.
#Ownership is one of our values
Lastly, we can’t apologise enough for the inconvenience this has caused some businesses, we understand that WorkflowMax is integral to the operation of your business and that when technology fails it can be extremely frustrating. We completely #own the fact this has been far from ideal, and appreciate the support we have got from many of our customers.
Needless to say, we will also be conducting a full investigation to understand what changes we need to make to prevent this issue from happening in future.
If you would like the opportunity to give us more feedback please do so by emailing firstname.lastname@example.org