I have been using Microsoft Exchange and Outlook since the late 1990s. I started out running my own Exchange server in my home. After six or seven years of doing that, I grew weary of maintaining the Exchange server. Day to day maintenance wasn’t bad, but the upgrades to the server needed manual fiddling to keep from losing data in the transition. And I wanted access to the server from off-site, but didn’t want to deal with the risks and hassle of doing that safely. So, in 2005 or so, I switched to Rackspace hosted Exchange. For more than 15 years, it was great. They handled the upgrades fairly seamlessly. The service was reliable. I could get to the Exchange server from off-site. Apply iOS devices synched well with it. I was a happy camper.
I am not happy with Rackspace any more.
It all started early last Friday morning. My iOS clients prompted me to enter a new password. Same thing with the desktops. I went to Rackspace’s web site, and the Exchange servers were down. I figured they’d fix things shortly, and didn’t worry about it. By Friday afternoon, it was apparent this was going to be a long outage. The Rackspace notices were vague, which is always a bad sign.
We are investigating an issue that is affecting our Hosted Exchange environments. More details will be posted as they become available.
We are investigating reports of connectivity issues to our Exchange environments. Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).
We will provide further updates as they become available.
All hands are on the deck & right resources have been engaged and are actively working on the issue. All new updates will be posted here as they become available.
We are aware of an issue impacting our Hosted Exchange environments. Our Engineering teams continue to work diligently to come to a resolution. At this time we are still in the investigation phase of this incident and will update our status page as more information becomes available.
To all of our valued customers, we understand the connectivity and login issues in our Cloud Office environments are greatly impacting you. We are working diligently to resolve the issue and it is currently our highest priority. Please continue to monitor our status page for the latest updates. Again, thank you for your patience, as we work to provide you a resolution soon.
And then things got really scary:
First and foremost, we appreciate your patience as we are working through the issue with your Hosted Exchange account, which we know impacted you greatly today. We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further issues while we continue work to restore service. As we continue to work through the root cause of the issue, we have an alternate solution that will re-activate your ability to send and receive emails.
At no cost to you, we will be providing you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.
To activate, please use the below link for instructions on how to set up your account and users.
Things are so bad that they are recommending their customers start over with a different hosted Exchange service! Ransomware? Sure sounds like it.
On Saturday, they made things a little clearer, while still providing little actionable information:
On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.
The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.
Security incident, huh? I think they’re saying they’ve been hacked. Twitter was saying the support queues were hours long, and the Rackspace people couldn’t provide much help. I decided not to bother.
There’s a parable in Matthew that applies to what happened next:
There was a man who had two sons. He went to the first and said, ‘Son, go and work today in the vineyard.’ ‘I will not’, he answered, but later he changed his mind and went. “Then the father went to the other son and said the same thing. He answered, ‘I will, sir’, but he did not go. “Which of the two did what his father wanted?” “The first”, they answered.
In my story, Microsoft is the second son, and Google the first.
I had a Microsoft Office 365 business account that supposedly supported custom email domains. I went through the process of setting it up with a domain that I owned but wasn’t using.
The way Microsoft and Google handle custom domains is a multi-step process. First, they make sure you own the domain. They do that by giving you a text string with a long pseudo-random group of characters, and tell you to add a TXT record in your domain’s DNS with that string. They check the DNS and make sure the string is there. Then they give you the MX records they want to see in the DNS. You put them in. They check them to make sure they’re right.
After I went through all that, Microsoft liked what it saw:
But there was no mailbox:
And the diagnostic message was distinctly unhelpful:
I checked, and the licenses looked good. I set up another user, and tried again. Same thing.
I opened up a trouble ticket, and didn’t hear back after several hours, so I moved on to Plan B, which was Google Workspace.
At first, things didn’t look good:
But — surprise, surprise — everything seemed to work.
I exported contacts and calendar events from Outlook and Imported them into Google Workspace. That went swimmingly. I synched my iOS devices to the Google Workspace account, and that went fine, too. I added another user to Google Workspace and ported over her contacts and events. The only issue — and it’s a tiny one — with iOS synching to Google Workspace is that no push synching is available. The iOS device has to poll the server to find out if there are new messages, events, or any other things that need updating. If it polls infrequently, there is latency. If it polls often, it uses power. There is an “automatic” option that attempts to find a happy medium.
I haven’t spent a lot of time with the administrative tools in Google Workspace, but they appear to be powerful and relatively easy to understand. My only niggling worry is that the Google apps require in Internet connection in order to work, where Outlook doesn’t.
I’ll wait for the dust to settle on the Rackspace outage — about 84 hours long now — before I decide whether to stay on Goggle Workspace or switch back to Exchange. I have serious reservations about going back to Rackspace because they’ve played things so close to their vest with this outage. They were great for more than 15 years, but I think it’s time to move on.
On Tuesday, another update from Rackspace:
We appreciate your patience as we continue to work through the security issues that have affected our Hosted Exchange environment. As you know, on Friday, December 2nd, 2022, we became aware of suspicious activity and immediately took proactive measures to isolate the Hosted Exchange environment to contain the incident. We have since determined this suspicious activity was the result of a ransomware incident.
Alongside our internal security team, we have engaged a leading cyber defense firm to investigate. Our investigation is still in its early stages, and it is too early to say what, if any, data was affected. If we determine sensitive information was affected, we will notify customers as appropriate.
Based on the investigation to date, we believe that this incident was isolated to our Hosted Exchange business. The Company’s other products and services are fully operational, and we have not experienced any impact to our Rackspace Email product line and platform. Out of an abundance of caution, we have put additional security measures in place and will continue to actively monitor for any suspicious activity.
I’ve set up a test account under Google Workspace, and will be experimenting with various ways to upload the Outlook emails to that account. I have already backup up the Outlook clients to .pst files.