We have continued to monitor the platform overnight and are happy that no further issues exists.
Timeline:
Issue reported and observed: Senders (to protected customers) were receiving block messages when sending inbound emails, with various (but not all) being marked as blocked or quarantined.
We started to receive reports of inbound mail issues from various customers around 12:30 (07/01), due to similarity of the issues being reported we immediately commenced our investigation along with engaging the platform support team. At this point we also raised this incident status as well as pushing notifications to our desktop application (where in place on customer sites)
During our initial diagnostics and prior to confirmation from the platform support team, we identified a common issue across all of all of the customers reporting issues. Within 15 minutes of this finding we had implemented a work around to mitigate the impact of the issue (across the platform) and started to see no further issues for new inbound mail.
During this time, the platform support team confirmed that they were seeing the same issue and that it was a global issue impacting customers in all countries. They had highlighted an issue with DNS resolution between them and Microsoft and they were working with to get this resolved.
Whilst our workaround was in place customers continued to receive any new emails without issue. Customer were required to access the Barracuda Portal (https://ess.uk.barracudanetworks.com/user) and release any emails that had be incorrectly blocked or quarantined during the initial issue and before our work around.
We received an update from the platform at 15:48 to confirm that the underlying issue had been resolved and updated our incident status. We however decided not to rollback all customers until we were happy that everything had settled down so implemented the roll back on our test platform only.
This morning we have continued to review the logs and are happy that everything is now functioning correctly. We will now commence the roll back to normal operation.
We would recommend that all users do know and maintain securely their password to the Barracuda Portal so they can administer and monitor their quarantine/blocked emails, we have links to guides on how to reset/change your password and the user guide for the platform below
We are sorry for any inconvenience that this issue may have caused, if you have any further questions please do not hesitate to contact us.
Posted Jan 08, 2021 - 06:04 GMT
Update
We have been advised that the underlying issue has now been resolved.
We have removed the work around we implemented on our test platform, and will monitor before removing the general mitigation measures from the remainder of the platform.
Posted Jan 07, 2021 - 15:59 GMT
Monitoring
The implemented work around we have put in place has mitigated the current Microsoft issue and monitoring of the logs are showing more positive results with no further new inbound emails being incorrectly blocked/quarantined.
If you require any assistance please do not hesitate to contact us.
Posted Jan 07, 2021 - 13:46 GMT
Update
On further investigation it has been highlighted that the issue is within the Microsoft network.
We have implemented a work around during this issue however any emails that have already been blocked or quarantined incorrectly will need to be released from Quarantine via the Portal - https://ess.uk.barracudanetworks.com/user