UPDATE: Dec 4, 0800 Pacific Time
Mail has been normal over the last several hours. Messages are getting delivered immediately, with normal response times. The backlogged messages appear to have cleared completely. We waited until the US east coast customers started their day to make sure no new bottlenecks sprung up. So far we have none. We will continue to watch carefully. Once again, thanks for the immense patience from all of you.
A CLARIFICATION: Dec 3 1800 Pacific Time
We have received many questions and enquiries on whether the Mail issues we face are a result of not having geographically dispersed, redundant systems within the Zoho infrastructure. Here's one customer opinion:
“Clearly, there is a need for redundant systems in other geographical areas that would be far enough away to not be impacted by any type of disaster that might affect the main site, and the switch-over should be quick and transparent. I’ve worked in IT for 32 years for various world-class IT providers, and all of them designed their systems with automatic redundancies and both automated and manual monitoring around the clock.”
The recent issue with Zoho Mail is not a result of our systems being impacted by the Chennai flood disaster. That would be a misunderstanding of the situation. In fact, the mail servers and storage systems are not located in Chennai. Zoho has multiple, redundant data centers across multiple continents. We have six copies of all customer data, replicated across data centers.
The Mail problem coincided with the natural disaster and was not caused by it. As explained in Q2 on this blogpost, the Mail issue was a result of a software bug that had unusually severe service consequences, even after it was fixed. It was our complete bad luck that this bug showed up at the same time as the disaster. So, no, we are not using the natural disaster to mask another underlying cause.
What we did say was that our ability to respond to the bug – examine it, test its scope and impact, fix it, test the fix, and update the software accordingly - was certainly impacted by the scarce availability of the technical staff of the Mail development team. This team is based in Chennai and reeling from the floods. In addition, our ability to respond to this bug's unique aftereffects on the customer experience required the technical expertise of the same team. No additional system redundancy could have fixed this issue. The people with the specific skills were simply up against a bigger struggle.
Think of it this way. If an automaker uncovers a new issue with the braking system on one of their car models, the issue must be examined and resolved by their engineering team at their research headquarters. The extensive service infrastructure available at their hundreds of dealers cannot solve this problem. The technical expertise - and all the equipment for its investigation and resolution - are only available with the R&D team, a core centralized corporate asset. Now if that R&D team in unavailable for any reason, the issue remains unresolved until they return, and the customer unfortunately suffers.
That's exactly what happened here and we're really sorry about it.
UPDATE: Dec 3 1200 Pacific Time
We have identified the issues with our Mail - both incoming and outgoing messages - and are working on it. We have posted some answers to your questions here. All other services are up and running normally.
UPDATE: Dec 3
Unfortunately, our Chennai office has completely shut down. This is a sad first for us in over twenty years of operations. We had four redundant Internet connections and the last one shut down early this morning. The city is a disaster zone. This has been the worst rainstorm and floods in almost a hundred years. The airport is closed and most rail connections are suspended or diverted. Thankfully, our employees and their families are safe. We had a few cases that required rescues from flood waters and abandoned apartments. Other Zoho employees were among the first to respond. The city continues to suffer. Incessant rain is part of the forecast for the next few days.
We expect that Chennai will take some time, perhaps weeks, to recover completely. We are planning accordingly. We have opened up customer support centers in Austin (USA), in Tenkasi (India), and have moved 50 available technical staff to Bangalore to set up another emergency customer service and support center. More people will follow. We will be coordinating across all these offices to try and provide the best support we can over this period. Service will be slow. More importantly, problems will be difficult to track and resolve because many of the skilled and technical staff, along with their families, are victims of the floods themselves. Some have had to evacuate their homes. Many are unreachable today.
All Zoho services continue to be up and running as they are served from our distributed data centers worldwide. It's the people-related issues that have been impacted, primarily customer support and issue investigation and resolution.
As of now, Zoho Mail is not working reliably, at least for many. Customers are experiencing problems receiving mail. We had been making progress on this issue over the last 24 hours and were testing out a fix, but the office shutdown has set us back. We are scrambling now to assemble the staff and location for a dedicated emergency Mail team right away, to pick up where we left off.
We will continue to post updates here and on Twitter. We completely understand that your businesses are also hurting and that many of your customers are impacted. We are truly sorry. We are focused on fixing this now. The vast majority of you have extended your goodwill and patience to us and we sincerely thank you. For the second time in one unfortunate month, we request you to stand by us.
Dec 1, 2015
Dear Customer:
Some of you have noticed and complained that our customer support has become less responsive over the last few days. Unfortunately, this is indeed true. Our support is completely backlogged because of the flood disaster in Chennai.
Many of you might know that Chennai is suffering its heaviest rains and floods of the last decade. They have battered the city for more than a month now and the forecast today predicts a worse downpour over the next four days. More than 10,000 residents have evacuated, including some of our own employees, and many are unable to get to work due to severed rail and road connections.
Zoho has increased support from our offices in Austin, USA to serve global customers during this emergency. People are also working from their homes, where possible. All this will certainly help but we still expect delays in customer service. Please be patient with us.
The service is working fine. No delays.
Sending prayers for all of you in Chennai. May families be safe and find their way through this crisis as they navigate the crazy waters! Thanks to Zoho for stepping in to help everywhere they can and for doing your best to support your clients' email during a very difficult time, Your integrity is the highest!!
Thank you very much ZOHO team! We came out of this and hope everyone in your team is fine! God bless Chennai!
It seems okay as of this current morning. The past few days has been a struggle. It seems as if it might be a bit slow still in US (for me at least) but it's no longer than 4 or so minutes.
Dear Vijay Sundaram, You have my utmost empathy and heartfelt sympathy for the problems you're experiencing. While I do not want to minimize the horrific personal losses of your staff and families, I feel more for you because of the fact that you unfortunately have to deal with stupid customers too. Sometimes I am struck absolutely speechless by the utter stupidity and ignorance of some people. And then I think, you know, these idiots don't even know how bloody stupid they are, that's why they advertise their ignorance and idiocy in a blog for all the world to see. Ignorant comments like: "The fact that they are not working has nothing to do with flooding in Chennai." and "solve the urgent problem of emails because companies are losing money because of problems occasioned by these units", also "More excuses and reasons why they can not simply provide reliable service . The people running this company are nothing more than a bunch of clowns on a bus going nowhere." must drive you up the wall. Then again, your get wonderful supportive comments like: "ZOHO and all the people who make it happen rock.", "You guys are doing such a terrific job and for you guys to have the time to update us and show your support even under those conditions just goes to show how great of a company you are." and "However, ZOHO has been upfront and honest about the issues at hand and have never tried to hide anything, for this I applaud their entire team.", which hopefully makes up for the negative ones and restores some of your faith in the human race, don't they? I am an applications development specialist with more than 34 years experience in the IT industry and I must admit, people who know next to nothing about how computers, applications and networks function, irritate the bloody hell out of me. I take my hat off to you for the patience you show in dealing with that kind of ignorant stupidity!
Thanks for your reply on mail incoming and outgoing issue.we prey for Chennai flood affected persons.god will take care of him. we are new in zoho mail.
We use many ZOHO products, email, support, etc...and as an MSP our business has also been impacted by these problems. However, ZOHO has been upfront and honest about the issues at hand and have never tried to hide anything, for this I applaud their entire team. Regardless of business impact, the most important thing here is that ZOHO's employees and families are safe and remain healthy! We simply used plan B - we reverted to calling all of our customers on the "telephone" - informed them that "we" were having email issues so should they require immediate support, to call us on the "telephone" rather than email tickets in. OK - is this "old school" - sure, but it keeps the business running and allows ZOHO to get on with their business of providing us all with great service to the best of their ability under some very extreme and stressful situations! Thank you ZOHO for all you have done, and our prayers and best wishes for all those impacted by the flooding.
I love you guys. Stay safe. My family and I are thinking about you, and praying for you and your families.
I am really sorry about what is going on, how long will it take for this site to be normal again? We at Gloo are thinking of the Zoho employees and their families in Chennai and hoping everyone stays safe.
My thoughts are with the engineering team affected by the natural disaster - I hope their families are safe. We also really appreciate the crystal clear clarity and honestly on your updates - you have no idea how refreshing it is. Having said that, my business is impacted due to this and we do think a few things in place 1) Geo redundant infrastructure (you latest update indicated you have this) 2) Gradual rollouts of updates (e.g. v1.1 deployed over v1.0 first hits only 5% of machines, then 25% over 6 hours, 50% over 18 hours and 100% over 24 hours. You should have also collect some basic metrics (e.g. # of connections or average queue or average response rates or whatever makes sense to your business. If any of those metrics spike during the rollout, that rollout stops progressing (or even rolls back). I'm sure this incident will cause your engineering team to revisit their approach. We'd love a blog post on this sometime. Also, some on our team suggested moving to another mail provider (AWS or Google Apps) but the honestly in your updates is keeping us here. Hope this gets resolved in the next few days. Good luck.
Thanks for explaining everything and hope that the ZOHO team will bounce back from this. Keep up the great service .
Thank you for keeping up updated and explaining the situation. Praying for the tech team, and those impacted by the flooding in Chennai. Thanks Zoho! JD
I stand with you and Awesome job guys. Keep strong!!
Emails are important for our business. But screw the delay in emails! Noooo problem. The most important thing right now is stay safe Zoho Team! I hope everything are okay and stay strong!
We are praying for you. Please stay safe. From Singapore.
We are so happy to hear your families are all safe. We shall rely our mobile devices while you deal with this situation. Take care and stay safe.
My best luck wishes to all people having to deal with this disastrous situation, I sincerely hope everyone will stay safe. Thanks a lot for keeping us updated, I really appreciate.
solve the urgent problem of emails because companies are losing money because of problems occasioned by these units, we want to know whether the problem will continue and speed up pc not solve this very day the inconvenience caused by more than two days for not having a contingency plan in its main base, you will have to take responsibility for the loss of money that each customer has not getting their emails, since today an email to a customer means money. We look forward to your answers and solutions. Best Regards and God Bless!
Charlie, as a business owner I hear your frustration however while Zoho staff are working through their contingency plans (and keeping us well informed of their progress) you too could move to plan B now that you know what the problem is and perhaps ensure all of your important money-making correspondence is done via alternative means?? On another more important note, my thoughts and prayers go out to the people affected by the floods.
We had a problem with the incoming emails yesterday and just saw this announcement. Praying for Chennai and hope Zoho will recover soon.
My sincerest prayers are with all of the Zoho Team and Chennai. You are a wonderful company. Stay well.
I'm very sorry to hear that, I hope everyone and everything is safe there. sent my deepest sympathy to all who are suffering. Wish Zoho team and familly is safe and sound..
Agree with everyone. People first. My employer also leverages strategic vendors in Chennai. All I can say is there are some very brave and dedicated folks there that have been working around the clock to limit impacts to the worldwide markets they support. While there will be opportunities to improve business continuity in the future. Now, today, they are showing there dedication and value as a global partner. Our prayers are with the people of Chennai and their families!!! For now, we can all do with a little less mail :)
The personal crises of those directly affected by the floods is the most important issue and we wish them well. Your communication with us customers is exemplary - if only more companies saw fit to keep their customers informed in such a clear and honest way.
We at Ecosuds are so glad to hear your families are all safe. We can manage with our mobile devices for a few days while you deal with this situation. Take care.
Please tell me how I can unsubsribe to the 100 emails that I am getting regaridng this horrible situation. I am sorry, but I take care of an elderly person and my time is valuable. I just do not want to see 100 plus emails in my zoho account. So obviously it's working? I actually did a test to post an ad on Craigslist and it sent through, I got the confirmation and it is posted Yay! I just can't handle the hundreds of emails coming to me anymore.. Oh wait, I see a box below: Notify me of folow-up comments by email and Notify me of new posts by email. I think I figured it out! Maybe it will end! :)
I'm praying for the Families who have been hit by such a disaster. Although, i'm glad that on the fact that everybody managed to escape unharmed.
Hearts go out to everyone at Zoho and Chennai. Safety first and worry about emails later. You guys are doing such a terrific job and for you guys to have the time to update us and show your support even under those conditions just goes to show how great of a company you are. Stay safe. Emails can wait.
Thank you Zoho team for keeping us in the loop. Please know that we at MTBparks.com stand behind you 100% and wish the best for your staff and families.
Uso zoho en mi pequeño negocio desde hace pocas semanas, pero estoy mas que contento con el servicio, espero resuelvan sus problemas pronto y que todo salga bien!. Saludos desde Venezuela.
Good luck to you all, be strong!
Don't worry email, peoples lives are at stake here, that's more important but thank you for the update.
Honestly, I was pretty frustrated at first with the email slowdown/shutdown & its effect on my business, but I see from the news that my problems are petty compared to what you're going through. I will need to see my emails eventually, of course, so I trust they are just piling up in cyberspace until you guys get back online, and not lost forever...? For now, though, I'm actually beginning to enjoy the break. How often do you get a day without constant interruptions anymore? I can't believe how much I've gotten done today... and how much lower my blood pressure is! In fact, take your time. Take care of what's really important first. My clients & I can wait a few days, until you & your families are safe. - Tim
Hi Tim You have wrote nice mail... Thanks Raja
You made me laugh today! Thank you.
Thanks for keeping us up to date on the situation. Hope everyone is keeping safe in Chennai!
This flood is just another bar to hurdle for the Indian people who's nation is rising as the next economic power. Remember Manjhi who cracked a mountain? This flood is not even a hill.
We here are big fans of Zoho and it's services. Because of that, for us, a few delayed emails is not critical. We want everyone at Zoho to be safe and to be able to continue to give us the amazing service and services that you provide. People on here who think their emails being sent at an exact time are more important than the lives of people who provide them a service need to do a reality check. Stay strong, Zoho team!
I am sorry to hear about the disaster but glad to hear everyone is safe! Thank you for the update. Your service is wonderful.
My best wishes for the people of Chennai and I hope all you can return to normal life soon. Also I wanted to share this article I found online which might be interesting for your people: It might be a little radical, but usually there is some certain facts behind this type of articles.http://qz.com/563396/chennai-floods-are-not-a-natural-disaster-theyve-been-created-by-greedy-town-planners-and-dumb-engineers/
Very sad to hear. Best of luck and a speedy recovery to those in and around the Chennai office.
Desejamos a empresa Zoho e seus colaboradores toda sorte e que se recuperem e que se fortaleça ainda mais!
Hope everyone is ok-- prayers for those affected..
Zoho has been very good to me in Australia, now we have to support you wonderful people, it works both ways, I have found Zoho to be the best email ever and good luck with getting back on track soon
Recover soon, especially Zoho Team comes first. We will take the necessary measures to overcome the impasse. What is important is the sincerity to address the problems. Greetings and my solidarity with the zoho team and the people of Chennai.
I found out my e-mail is not working as I was awaiting news on a competition in Chennai and realized it isn't working than I than saw your blog-post and couldn't believe it , didn't know you were based in Chennai before. My heart goes to you and I hope everyone stays strong and well throughout this disaster.
Be safe guys, that comes first! Good luck and godspeed
We send our prayers out to all members of staff at Zoho and especially to residents of Chennai at this difficult period.
Godspeed relief to you good people. Wherever we are in the world, we're in this together.
Thanks for the information and my best wishes for you all in Chennai. I am very sorry and I hope you will stay safe.
first people. Although there may be difficulties in traffic of our emails, nothing compares with the living and working conditions of all employees currently zoho's face, so there's only one word for them, we are with you, have courage and strength above all mental, I'm sure the zoho will help you, even if it's with good words of encouragement ... hugs.
Screw the email . I hope your town is ok . I just looked up the flood on google. WOW , Stay safe zoho team !!!!
Good thing everybody is in one piece... keep well guys - you have all our support!
Our staff at Miami, FL and Guatemala city send our thoughts and hope Chennai comes stronger and more united after this. All the best guys
Is there anyway I can forward all of my emails to another email client just temporarily?
So sorry to hear about this, my thoughts are with you! Keep safe, best wishes.
I wish the best for your teamwork and your people and family. I'm really sorry
Hello and Comfort words from Sao Paulo, Brazil!
Wishing you a speedy change of weather for the near future so your staff may be safe and dry. Thank you for thinking of your customers during this time as well, nature is not something any one can 100% anticipate however.
We, from Ollive, hope you all are safe and the best for your employers and their families! We extremely sorry for you all! Could you tell us how Brazil may help you?
Sorry to hear of the major problems and terrible conditions. Thanks for bringing us up to date on why things have suddenly become problematic. Good luck getting operations restored. Thanks again.
Keep your heads up! Keep moving forward! with this you will overcome any obstacle that may come at you!
We at AXATECH hope every one be fine, life is a priority and family's stability as well. Best wishes to all you in Zoho be safe.
Good luck guys, stay strong.. much love for chennai from california
We are really sad about this distaster situation and we all know that we don't have any control in natual calamities but we all can pray for their safe . I will keep all chennai people in my prayer for them to bring back in safe. .
More excuses and reasons why they can not simply provide reliable service . The people running this company are nothing more than a bunch of clowns on a bus going nowhere.
u dare speak about the people of chennai suffering how dare you call us clowns.
Our thoughts are with you guys and with all the people affected. The team @ FormITV
My thoughts are with you guys!
Take care guys! Thank you for keeping us posted
Please keep safe and do not worry about us right now!
Wow! So sorry to hear this team. Thank you so much for the update and thanks for working on this.
ZOHO and all the people who make it happen rock. Our prayers are with you and all the people affected in Chennai. Keep safe, and we will all be thankful when you are back up and running.
Best wishes to all those affected, and thanks for the updates. Your in our thoughts and prayers.
We hope that all zoho staff and their families in Chennai india be okay
People first. Hope everyone is okay! Just wondering, will our emails actually make it in/out or will they just be lost forever? In other words, are they backed up somewhere so we will get them (or have it sent out) in due time? Thank you, prayers.
Eddie, your mails will not be lost. There is some delay in delivering emails (which is being looked into).
Best wishes to all of your employees and their families from us here in Guatemala at Paso Llano Magazine. We will stand by you - we have been happy with your services!
Desde mexico, el brindamos ayuda, suerte muchachos estamos con ustedes
Sorry to hear about this I hope all your employees are safe and that things get better for them quickly.
Really so sad,Stay strong Chennai
Dios guarde y proteja a su pueblo. Saludos desde PERU
Thanks for the update. So sorry to hear of this news, our prayers are with the families.
Why are we being informed that incoming and outgoing mail are delayed in a "Support staff" announcement? UPDATE: Dec 3 1200 Pacific Time We have identified the issues with our Mail – both incoming and outgoing messages – and are working on it. All other services are up and running normally. I am a Zoho premium Mail customer. So all of my services (incoming and outgoing email) are not running normally at all. Your email servers are in California. The fact that they are not working has nothing to do with flooding in Chennai. And if it did, you would be guilty of putting mail servers in an Indian flood plain. But you are not guilty of this. You are, however, guilty of mismanaging your servers. And that mismanagement is independent of this flood.
Thinking of you guys too. Hoping for the best and the gods are there for you.
So sorry to hear this news. Wish you all well and glad to hear that the team is safe. All the best.
ZOHO force team, we are in prayer for everyone in Chennai. Greetings from Peru
I hope you all are safe and I hope for the best for the employees and their families!
Please take care guys.
solicitamos soporte urgente. ya que tenemos un dia completo sin recibir correos, y nuestra empresa es imprecindible el correo electronico ya que tenemos oficinas en todo latinoamerica y es URGENTE.
Desde PERÚ enviamos fuerza a zoho y sus empleados.
We at Agrega hope everyone is safe and sound!
Keep strong!
Our thoughts and prayers are with your teams and their families..
My thoughts are with you guys!
Thanks for the feedback
@ the Zoho team and all those in Chennai. While I am far away, our dev team is in Chennai as well and we are well aware of the escalating crisis. We wish everyone the best of luck and hope that the DRC is operational! rgds all!
I understand how difficult life is for most Indians and the extreme poverty and suffering that exists there, even at the best of times, so my prayers are especially with those in Chennai with the extra pressures. Shalom Yishai
Heard about this disastrous situation on national BBC News here in UK this morning. Best wishes for all to keep safe.
I am extremely sorry to hear about this disaster and sent my deepest sympathy to all who are suffering. While your people are struggling with too much rain, the resulting flooding and everything dying because of drowning, the people in my country are suffering because of too little rain, droughts and everything dying from thirst. Wouldn't it have been nice if you could email us some of your water?
We at Gloo are thinking of the Zoho employees and their families in Chennai and hoping everyone stays safe.