Dear Zoho Customer,
Let me first acknowledge the obvious. The last several hours have been a nightmare for some of you since you have not been able to access the Zoho services you trust and rely on, to run your business. The zoho.com domain was inaccessible for many customers. Rapid corrective action has been possible for many customers to restore service availability but has not worked for some others.
Before I offer explanations, let me offer you my genuine apology. I run Zoho and as a business owner and CEO can fully understand what it means to not be able to access the software and services that keep your businesses on track and serve your customers. For this, I am truly sorry. I have been at the helm of this situation since it broke many hours ago and will continue to be here until everything is fully resolved.
What happened?
Here’s what happened. Our domain name registrar blacklisted (shut down) our domain. (Registrars are independent organizations that manage the reservation of internet domain names. The registrar does not host any Zoho site, they simply register the zoho.com domain name.) The blacklist lasted about an hour before it was restored. This means any incoming services request to Zoho.com cannot get resolved into the proper IP address that can deliver the services (although the service is still up at the specific IP address). The shutdown impacted some, but not all, customers who tried to use any Zoho service. Unfortunately, domain names still remain a single point of failure in the system.
The shutdown was done by an automatic algorithm in response to phishing complaints against Zoho. (Phishing is a fraudulent attempt by a malicious third party to impersonate a legitimate email address for nefarious activity, like fake invoicing). Phishing has successfully targeted all major email services providers around the globe. Phishing is rampant and mail services providers like Zoho have devised multiple methods to combat it like blacklisting, flagging suspicious emails, scanning, smart filters, and other methods. According to Symantec, 76% of all organizations have reported falling victim to phishing attacks in 2017.
In this case, the registrar received 3 phishing complaints over the last two months (from recipients of third parties phishing messages impersonating Zoho mail), 2 of which were addressed immediately and 1 was under investigation. To put these numbers into context, just one security service company blocked 51 million phishing attempts in 2017.
Somehow this automated algorithm decided to shut down the Zoho domain based on these 3 cases—without prior warning of the shutdown, or investigation into the traffic supported by this domain.
Let me also be clear that there was no cyber attack on Zoho.
What have we done so far?
The registrar restored our name service (DNS) within an hour, but new names (including more than 100 Zoho subdomains, like projects.zoho.com, that have been impacted) take anywhere from 24 to 48 hours to propagate to DNS servers around the globe and reach your business. This is an exceedingly frustrating wait for all of us. We have also migrated to a new registrar (Cloudflare) already.
Until then we have shared multiple workarounds on our @zoho handle on Twitter (and other Zoho social media sites). Many internet service providers are slow to update their domain name resolution servers (DNS servers) but Google and Cloudflare provide fast-updating DNS servers, and those already have the restored Zoho.com name servers cached in them. This is the essence of the workarounds. We have explained how to use them on various operating platforms like Windows, MacOS, Linux, Android, and iOS. These work for many impacted customers, but perhaps not for all. We will continue to explore and post others. In any event, DNS server updates will automatically happen across the globe, making services accessible.
What can you do?
Watch our posts on the @zoho Twitter handle.
If you still face issues, see if any of the workarounds posted under the Zoho handle work for your business
Write to us at support-escalation@zohocorp.com. We will instantly monitor and respond to all requests to this line.
What are we doing long term?
You have my assurance that nothing like this will ever happen again. We will not let our fate be determined by the automated algorithms of others. We will be a domain registrar ourselves.
I thank you for your support and I will be here until you do not need me anymore.
Sincerely,
Sridhar Vembu
CEO
Zoho Corporation
Shocked to read what happened and disappointed and sorry about the lack of management you received. Zoho deserves better. I was reading not too far back EasyDNS are the ones making progress in this territory of 100% unconditional uptime. Perhaps you should go take a look for the sake of such a huge customer base. Teach these googly-eyed flare pants a lesson perhaps. I'm only just acquainted with Zoho and am very excited the more I learn. All the best.
I didn't know I had to join Twitter in order to get notifications from Zoho about service failures/fixes.
Hi Chip, although outage situations like these are extremely rare, when they do happen, we also post updates on our Facebook page. So if you prefer, you can follow them on Facebook instead of Twitter.
Thank you for your thorough response and the corrective action taken.
Great response and a great company. Hope Cloudflare will treat you better :)
Things break down sometimes and we have to live with it. But I didnt find it out from Zoho. I never received any emails. I have updated my contact no and email in ZOho books. No information was sent via these channels. By sending an email or sms with a work around or an ETA you could have saved a lot of trouble for our company. I raised a ticket and I received a reply now. I keep finding bugs in zoho books which never get fixed. The only thing holding me back from moving away from books is the significant effort needed to integrate with API from another product.
Hi Avinash, I am extremely sorry for this. Can you please share the issues you are facing with Zoho Books to support at zohobooks dot com ? We will immediately look into it. I would be happy to talk to you on this. Regards Prashant
I've done a lot of work with Zoho stuff lately, and I could not be more impressed with their products or their support team. This is in the same vein, and Sridhar's response to this issue is absolutely spot-on. Keep it up, guys - you're doing fantastic work.
My pop3 was not working from day before yesterday afternoon till 4pm, I thought it was problem with my ms outlook, thanks for your prompt apologies
Great attitude man !
> We will be a domain registrar ourselves. When is that coming, I want to move my domains :) Also what about payment processing, Tired of getting ripped off by stripe and paypal.
I can feel how challenging this situation would be. You and Zoho are a great inspiration for many. It is great that you stand in-front and do the best possible. Thank you.
Getting in front of it and personally getting involved with an explanation / Apology and a fix you've set an example on how CEO's of large organisations should handle such crises. Best of luck.
Happy to see the choice of Cloudflare for new registrar. From fly-by-night to one of the most trustworthy names in the industry. Good move.
The world did not come to an end. I understand the pressure your company was under to fix this. I also note that you are taking steps to make sure it does not happen again. It was a problem. It was solved within 24 hours. ZOHO is a great product. I can live with an occasional problem. But I am going to get serious about backing up my data to my computers and not just the ZOHO servers. I suggest an automated way to do this daily would give us all peace of mind. When Zoho was down there was this stunning realization that so were my backups. THAT needs to change.
We use Skyvia to automate daily back-ups. Might be worth taking a look.https://skyvia.com/backup/zoho-crm-backup
The best one-- We will be a domain registrar ourselves. Y should depend on others? Also then, when for silly things they created this much big issue.
Well said, Sridhar. Thank you for a very detailed and thoughful explanation. Shit happens. It's about taking owebership, outlining a move forward plan, and being sympathetic to your customers needs. And you sir have done all of that. Thank you!
Classy! All the best.
Thanks Sridhar. Problems come to everyone big or small.. its about how you manage them.... You've come a long way and I learn a lot from you..... I am your customer and your fan too..... God bless you and your team for the hardwork
Thank you for the post. The first thing I did when I started getting reports of the services down from within our company was to look at the whois info and see your DNS was suspended. While this had a major impact on me and our companies daily operation, my only complaint is the lack of meaningful communication. I had to dig for info.
Thank you for the acknowledgement but no matter how you slice it, this is a clownish approach to your domain/DNS strategy. Were you unaware before yesterday that domain registrars can be the single point of failure? Were you unaware of the existence of industry leaders such as CloudFlare? Why is your status page hosted on status.zoho.com when it should be hosted on a completely independent service like status.io, that's how you could've kept your customers informed. I watched the debacle on Twitter, and there was NO acknowledgement about the outage for hours, until finally, someone emerged. So there was no domain/DNS strategy, a totally rookie mistakes, but what other rookie moves should we expect in the future? It's anyones guess.
Hi Alex, unfortunately, domain does remain a single point of failure in the system. That was true when we were with Tierra yesterday, and that still holds true today with Cloudflare. This applies not just to Zoho, but to any other domain on the internet. With respect to your query about our status page, I'd like to inform you that it is independent of other Zoho services. It is powered by Site24x7 (similar to status.io) which monitors services from multiple locations. Our status page didn't show the downtime because the domain was resolving from various locations.
Thanks for the thorough and timely response. I have a question and a suggestion. For those of us that made system changes, like changing the DNS settings, what should we do now? change them back, or keep them at 8.8.8.8? And, perhaps it might have been better to push some notification out via email and/or text, rather than have customers search to find information and updates?
Hi Mike, We'd recommend retaining the DNS settings for one more day, as that will complete the 48 hour period that is required for DNS changes to propagate. After that, you can revert back to your original DNS settings. Although outage situations like these are extremely rare, we usually post updates on our Twitter profile and other social media pages to keep customers posted. Sending out notifications over email or text could be intrusive since not all customers were affected by the outage.
Thanks, Meera. How do I revert back to my prior DNS settings in Windows?
Mike, you can restore your original settings by following the same steps that you followed earlier. Just select "Obtain DNS server address automatically" if you had that option selected previously.
Thanks Shridar for being involved personnaly and have the team working hard for solving.
Your leadership is certainly an inspiration for many of us. Thanks for rising to the top.
Glad that the services are back. Still, the algorithms need to be thought in a challenging way.
The damage caused by the registrar caused more harm than the phishing emails would have ever caused. It is good that you switched services already to move away from this company.
Good to hear. We'll be staying with Zoho. You may wish to consider hosting business-critical services (e.g., the web interfaces) under a different domain name than is used for user email addresses.
The domain name is the kingpin on which the software is floating.. I don't think this is not just a small non conformance. Its a system breakdown. Wonder how it's going out of your Team's scanner. Once I tried to use zoho inspired by its concepts ,But unable to move further as I forget my password and bounced back to the starting point again and again. Well, We hope fails are needed to build ourselves so stronger. Best of Success!
You would think with all the smart people there that you would not have let your company (which is critical to others) ever go with a company that had such an automated algorithm in the first place. Our company was down for 8 hours and only got working once we went through the Google DNS server 8.8.8.8. Very Poor results in my opinion and I am sure other will agree.
Amazing note that really gives a personalized touch and am sure that your affected business partners will support in this tough situation.. Wishing all the best ZOHO
Excellent explanation from Sridhar and fast acting from Zoho to resolve the issue and avert more damage. It’s refreshing to see a CEO’s personal involvement in solving a crisis and addressing the public. Zoho has been very reliable and worked out great for us since we subscribed to their services 6 years ago.
This is happened for the second time mr. Sridhar , your team doesn't seems to understand that
Hi Imran, could you clarify? This is the first time we've had a domain blocking issue. Please let us know in case you're still facing issues accessing services. We can reach out and help.
Meera, it happened a couple of years ago. There was a problem with the DNS back then as well.
Steve, I believe you're referring to a partial outage situation that was caused by a different issue: https://twitter.com/zoho/status/789521563414515712 Details of all major outage situations that have affected our customers in the past are available here: https://www.zoho.com/service-updates/blog
Very impressed by Zoho's response. Other CEO's could learn by this example. Bravo
Very honest and transparent which reflects the ethos of this great organization. I am sure that the corrective measure taken by the Zoho team will prevent any such surprises in the future
Good reply Sridhar. Pls incorporate GSTR6 filing as well. Regards.
Good morning, Sridhar... I appreciate your detailed explanation and empathize completely. However, I tried to use this email as you suggested (support-escalation@zohocorp.com) and my mail client gives me a message that it does not appear to be a valid email address and to verify that it is correct. I really do need to access out client database desperately today. Please advise.
Thank you for the information and solutions you have put forward.
Thank you for your letter, Sridhar. Well written!
This is the reason why Zoho is one of the greatest companies and why I spend my money at Zoho - great honest communication, genuine customer service, and quick reaction. It's obvious that all this comes from the top. Thank you Sridhar for being a role model for all businesses.
Zoho is a great company staffed by great people. There were businesses affected by this debacle but I sincerely hope that the hard work of Zoho family will not be permanently scarred by this single blot which was very unexpected.
the mistake is human and I trust you. Congratulations for your business. I was impacted but nothing serious in my case but I understand that others may have suffered. The mistake is part of life and your Indian values teach us that well ... Good luck with all people dissatisfied to see their habits disturbed ...
Thanks Sridhar Vembu. Great!
Thanks Sridhar Vembu.
Apt and satisfying response to your customers! This is THE BEST response I've seen by a CEO of a company to his customers addressing the disruptions in Service.
Absolutely brilliant! Well done.
Thank you Sridhar, for your dedication, honesty, and transparency. You don’t see this level of ownership in all companies - it is both refreshing and reassuring.
Thank you Sridar for the candid update. I am very confident you will do everything in your power to address this and make sure it does not happen again! Keep up the amazing work at Zoho!
Thank you for your candid explanation Sridar - I am confident you will fix this so it does not happen again. Keep up the amazing work!
Thanks Sridhar ! We are good ?
Still not able to access our accounting
Hi Alex, sorry to hear that you're still facing issues. We'll have someone from our support team reach out to you asap.
Sridhar a total nightmare by any measure but well handled by Zoho. As mature as the internet is gaps are still found. Ironic that Zoho can pass the tightest security audits by major banking institutions interested in the Zoho platform, and yet a silly automated bot can cause this havoc. For those with time to read - when Maersk Shipping was brought to its knees along with the global logistics network it was slow DNS replication and a lucky powercut that saved it. https://www.linkedin.com/feed/update/urn:li:activity:6441559782229843968
Thank you very much Sridhar. Thanks for your personal involvement into this crisis (watched you post and answered clients on Twitter for hours) As a loyal customer since many years you have my full support.