1. Have there been issues with Zoho Mail over the last few days?
Yes, customers are facing issues with Mail. Many are unable to receive email. Email on mobile devices is very slow and unreliable. Updates are available at https://blogs.zoho.com/service-updates and we're also posting on our @Zoho Twitter handle.
2. What is the issue?
Along with our genuine apologies, we owe you a straight explanation. Here's what's going on.
The Mail issue started with an inadvertent bug that was introduced in the mail software while we were making infrastructure upgrades to the system. Stuff like this unfortunately happens in software and vendors have learned to manage this. We caught this and fixed it. This was an issue limited only to mobile mail and to users with non-Zoho email client software, like Outlook (these use one of the IMAP or POP "protocols" to pull down and send mail from your mobile device or a different mail client). Under normal circumstances this would have caused a glitch in the customer experience. But this time, things got exacerbated.
The bug (and its fix) resulted in all mail clients (the software on your phones) re-syncing with our servers, overwhelming them with simultaneous traffic. Imagine each user device reloading all their messages, keeping track of read and unread messages, folders, hierarchies, and everything you've come to expect from modern email - all at the same time. We added more servers and nodes, but were unable to match the demand.
Over the last 48 hours, we have observed a bottle-necking pattern. As major geographies come online early in their mornings, all devices start to sync at the same time, slowing down service for everyone, all over again. This started to impact not just mobile mail, but ALL mail, including web mail.
That our technical staff were hit by the Chennai flood disaster (#ChennaiFloods) - the worst in almost a hundred years - with limited or no connectivity, power, or transportation added to the whole mess.
3. How will you resolve this?
This is our top issue now. We have found some bottlenecks on the storage and server side, and relieved them. We have also made some infrastructure changes to manage the backlog itself. We continue to remain watchful to see if there are other issues that we might have missed.
4. How can I use email now?
You can still use your desktop browser and your mobile mail app. Some of you may know that you can also download the Zoho mail app from any of the app stores. These apps are “native”, more feature-rich, and do not use the inefficient POP or IMAP protocols. Unfortunately, we expect you will still see mail delays until the backlogged messages are freed.
6. Will I lose the emails my customers and contacts may have already sent me, that haven't arrived?
Your messages will not be lost. They are all just backlogged on our servers. Once the system gets to normal service, you will receive them. We do understand that many email messages are time-critical and we're really sorry we cannot fix this issue right now.
7. Do you have any idea when Mail services will be back up and normal?
Our best expectation is that this whole process will take a few days before mail response is immediate. Until then your mail will trickle in, possibly slower than usual. Please look out for our updates.
Looks like this issue has risen again! I'm able to send emails but clients are getting a failed notice email. Bad for business!!!
I am not going to criticize Zoho for the bug that caused this all mess. It can happen and I know that first hand. What I think has been handle very poorly is the response to the problem when it became evident and this is much more critical. First, communication with the customers was appallingly bad. Wrong information was sent out, when it was sent at all. The flood is not an excuse: if you sell a service globally you need a decentralized support system. Second, if, as you say, the problem was caused by the email clients trying to resync all at the same time, why you didn't shut down (or limit access to) POP and IMAP so to preserve SMTP and access to the mailbox via webmail and native clients that, according to your statement, were unaffected by the problem? At least everyone would have had access to their mail using alternative methods. This is the reason why I have chosen Zoho in first place: multiple way to access my email but you fail me. Third, this sound like a self-inflicted denial of service attach that killed the whole email system for days. It casts some serious doubt on how strong Zoho infrastructure is and how strong its disaster recovery plans are. Just my 2 cents for today.
I just want to follow up. We are still unable to send email using zoho support or zoho crm. When is this expected to be resolved? We still get the error "Sending Failed" when you attempt to reply to customer tickets.
I've forwarded your details to support. They're on it.
I am not using free email, I am paying you for email and this is not working, I need to know something or we are going to have to cancel all of our services. Please give me an idea of how soon this will be resolved.
John, I've forwarded your note to our team. They will connect with you.
To The Zoho Team: First, I sincerely hope you are all safe as are all of your families. I also offer my sympathy for any of you who are experiencing personal loss from these unprecedented weather events in your area. As far as the business issues are concerned ... (1) Mail appears to be back up at this point and round trip emails are being sent and received. That's great and I'm sure you've been working hard to get things fixed. (2) I certainly know that back to back issues with the DDoS and Flooding are hugely impactful on your organization and your explanation of the problem was refreshing, but also a bit frightening as well. As the CEO of a group of companies that has committed our backend operations and recently our email communication to your platform and infrastructure, I trust that you will gain great experience from these events and will very soon detail in a similarly transparent manner, what your plans are to ensure that events like this (and the DDOS and other possible impacting events) are being planned for with immediate failover processes, truly diverse data centers and similarly geographically diverse experienced personnel in place to manage any forseeable challenges. We live in a global economy that is getting more and more challenging each day and cyber issues are one of the biggest vulnerabilities we all must cope with. You are competing in what is very much a "public utility sector" market. We want you to be the best there is and wish you well in that uphill battle. Thanks for all that you do ~ Tim
After the backlogs are cleared, do you expect new backlogs to appear once new geographic regions come online in the morning (IE: Eastern US) and the process to be repeated? Or is there now a fix in place to eliminate new backlogs? Also, can you advise on a timeframe when you expect the current backlog to fully clear? Thanks
We have made a few changes. Only tomorrow will we know how they hold up, when the volume spikes again. The current backlog is clearing now and looks like it will take a few hours.
Hi there A very very frustrating situation for you, your staff and of course your customers. Can you advise if there is currently any way to access mail other that via the web interface or via your Apps? Thanks
Those are the two obvious ways, Martin. You will likely see delays no matter what you do until the backlog of messages is cleared. Thanks for your patience and obvious understanding.
Hi Vijay Thanks for your response. Could a possible option be to redirect all mail to an alternate email account (non Zoho) configured via the Web Mail 'Email forwarding' option ? I would guess the same delays would apply, but it would allow those people who are tied to Outlook to continue working? Thanks Martin
Martin, that will not work. The forwarding logic is applied by the mail server (even though the user sets this on the mail client). This means the server has to receive any mail on your behalf, before it can process it (in this case, forward it to a different address). This is where the current bottleneck is, so it won't work.
Contrary to what this post says, at least a few emails of mine got missing. I know this because, I sent myself 4 test emails today and only got two of them this evening. Also my contact sent me a reply that quoted a previous email that I never received. Zoho, would you please recover my lost emails! Some of them are extremely important!
Messages are coming in, sometimes out of turn. I experienced this myself. The backlog is still unclogging, so you will still not have received all the messages sent to you. We understand their importance to you and will be on it.
Vijay, Thanks for the reply. Yes, I'm still missing emails. The timing is really bad that I'm waiting for some important email replies. Now I don't know if my contacts replied my email and the messages are lost or my contact didn't see my email in the first place. What's worse is that I can't easily verify with them. Is there anyway you can look into this for me?
I'm sorry, but this will be near impossible, Javy. I will ask. Meanwhile, my humble advice is that you connect again with those important contacts (use the phone, or send email from another account) and check into things quickly, just to be sure. My sincere apologies.
I just wanted to thank you guys for offering such a "usually" reliable and FREE mail service. Floods happen (literally and figuratively) and we have decided to stick with you in spite of these temporary issues. Good luck recovering from this and we truly hope you are able to recover your lost customers and it doesn't impact you too greatly.
Webmail is no longer working for sending or receiving either. Getting very stressful for businesses that rely on email for incoming new business or other daily business communications. Kind of glad I was slow to move all ov mine over there, but unfortunately I've spent 2 days fiddling with my servers trying to figure out why mail wasn't working properly before finally seeing the message on the webmail interface. A notification email to the secondary email accounts on file would have been VERY helpful.
That's a good suggestion and we will take it up. Thank you.
Seriously, an update after online outcry for three days of business not receiving their email, all during which you were tweeting that it was functional when it is clearly not? And your servers are not actually where the floods occurred, but spread out "worldwide"? Seems like three days of incompetency with no contingency plan and a blatant plea for pity after a bunch of lies.
solve the urgent problem of emails because companies are losing money because of problems occasioned by these units, we want to know whether the problem will continue and speed up pc not solve this very day the inconvenience caused by more than two days for not having a contingency plan in its main base, you will have to take responsibility for the loss of money that each customer has not getting their emails, since today an email to a customer means money. We look forward to your answers and solutions. Best Regards and God Bless!
Back to back issues is discouraging at best. We are looking for a server that has a contingency to prevent these issues from paralyzing our business. This is very disappointing.
While I agree this is a good update, I also agree that more needs to be done. For people like me who use the free email for personal emails, this is mostly just an inconvenience. However, since there are many business customers who pay for the service, there needs to be redundancies built in. Clearly, there is a need for redundant systems in other geographical areas that would be far enough away to not be impacted by any type of disaster that might affect the main site, and the switch-over should be quick and transparent. I've worked in IT for 32 years for various world-class IT providers, and all of them designed their systems with automatic redundancies and both automated and manual monitoring around the clock. If Zoho wishes to keep the paying customers and be world class, this is what they must do as well. This is the sort of issue that can make or break a company. I hope for Zoho's sake that they do everything required to achieve redundancy and stability.
Has this issue caused delays in sent mail?
Yes it has. Please see this post for details.
I've been in IT for 30 years. So I know what I talking about when I say this is a top notch update, one of the best I've ever seen. You've provided enough detail to satisfy the tech savvy, but not so much as to confuse the average user. That's very hard to do. Most of all, I can tell you are being completely honest and transparent. Also, what you are doing is spot on. You are dealing with multiple major issues at the same time that you couldn't have prevented. Great job and keep your chin up. Nothing, good or bad, last forever. We'll get through this.
I agree. Given the extremely tough circumstances, it is amazing how well Zoho has responded to this crisis. Great job!
Thanks for the comprehensive update. I'm fine with this one-time-only problem. If it happens again, I will re-think. Tod
Agree, good update but unfortunately with back to back issues ranging from the DDoS last month and now this my customers are fleeing to MS.
I simply cannot understand why a global business offering IT applications does not have an emergency disaster recovery plan? You clearly do not have one. It is an absolute disgrace. You have sent me this notice after 3 days and I have left endless messages and calls over the last 3 days to find out what is happening and no one responded and now you say it is working and guess what.... IT IS NOT WORKING.
Good update- thank you for it.