All services including the Zoho website are currently (starting 8:25AM PST) inaccessible due to a network issue. We are currently looking into it. Will keep you posted. Please follow us on our Twitter account for updates.
Update (8:46AM): It looks like we have a power failure in Equinix co-location provider. It effects multiple customers in that data center.
Update (9:15AM): The power is back up in the data center. We are working to restore the services. Will keep you updated.
Update (9:32AM): The power is up. We are making sure there is no data corruption before we bring the services back up.
Update (9:50AM): We are running sanity checks before we open the services to everyone. Services will be up one by one after the checks.
Update (10:35AM): We are still working on bringing the services back up. Mail & CRM will be restored first.
Update (10:55AM): Our Accounts service (accounts.zoho.com) is up. Working on restoring Apps.
Update (11:30AM): We are still working on restoring our services. We sincerely apologize for the inconvenience.
Update (11:55AM): The reason it is taking time is because the database clusters lost sync internally. It is taking time to sync them back. Once done, the services will be restored.
Update (12:10PM): Zoho Support is Up. Due to Mail dependency, the mail part will work after Zoho Mail is restored.
Update (12:15PM): Restoring database clusters is taking time due to inconsistencies. Databases went out of sync due to power failure. To restore services completely, it might take few hours. We sincerely apologize for this down time.
Update (12:50PM): An explanation on why the restore is taking time is explained in this blog post.
Update (1:35PM): Zoho Mail is partially restored. Users in affected clusters will be able to access it after we restore that cluster. For now, most Zoho Mail users should be able to access Zoho Mail. Also, note that Zoho Support is also up.
Update (1:40PM): Zoho Analytics is now restored.
Update (1:48PM): Zoho Writer is now restored.
Update (2:25PM): Zoho Calendar & Contacts are restored. We still are working on fully restoring Writer. Will keep you posted.
Update (2:55PM): Zoho Writer is back up. Zoho Discussions & Zoho Books are restored.
Update (3:15PM): About 60% of database clusters have been restored currently for Zoho CRM. We will be making these available first. Users on these clusters will be able to access their data. As and when other clusters are restored, we will bring them online.
Update (3:50PM): Zoho CRM is partially restored. Around 80% of users should have access to their data. Other 20% will be restored soon. Will keep you updated.
Update (4:00PM): Zoho Invoice is restored.
So far, the following services are restored: Zoho CRM, Books, Invoice, Mail, Support, Reports, Discussions, Calendar & Writer are restored. Others will follow soon.
Update (4:20PM): Zoho Projects and Wiki are currently restored.
Update (4:28PM): Zoho Recruit & People are now restored.
Update (4:35PM): Zoho CRM is fully restored. Zoho Show & Bug Tracker are also restored. Zoho Docs is partially restored.
Update (4:45PM): Zoho Mobile service is now restored.
Update (4:52PM): Zoho Sheet & Docs are partially restored
Update (5:35PM): Zoho Chat & Share are currently restored.
Update (5:56PM): Zoho Creator is now restored.
Update (6:12PM): Zoho Meeting is now restore.
All Zoho Applications are currently restored. There are a few backend services currently being restored to full service. We are doing it in the backend.
We will do a detailed post with the postmortem report. Again, apologize for the outage.
I'm trying to locate the postmortem blog post on this, but can't find it. What's interesting is that it was a power outage, and I would think that UPS's are used to send timely db shutdowns so that things are consistent. Where exactly did things break down? This is a good learning experience for all, and certainly brings up questions like:Are you redundant in two geographically separated data centers?
How long can your data center run without power?Plenty of other questions like these, but I would love to see the postmortem report.
I hope a postmortem report is sent to all the "admins" of all the hosted domains as well as the "regular" account holders.
With today's cloud availability it's very easy for large service providers to provision data centers around the world. There really should be NO reason for any PAID service to have an outage for much more than an hour - or however long it takes to propagate DNS records pointing to the backup server.
during the outage several test emails sent by me to my Zoho hosted email domain bounced back undeliverable! That is what backup Mail exchangers are for!!! Whats MX2.zoho.com for if it's going to go down with MX"1"???
I have been a Zoho customer since 2008 and this is the first major outage that I can remember. Eight hours (or whatever it was) over all those years is still six nines and you can't ask for better than that unless you are NASA. While I do not always agree on some of their feature choices in CRM I completely trust in their abilities to manage a network. I am 99.999999% sure that this will never happen again.
With current technology, there is absolutely no reason for any website to every go offline. Zoho should have redundancy across several servers around the word. Amazon's EC2 Cloud will do just that. It's time to elevate the service or you'll be left behind. This is the second time since becoming a subscriber in the a couple of months ago that my file, tasks, email, etc, have put me out of business for the day. One one time and I am out of here.
As a brand spanking new user, this experience has been sobering to say the least. Not having access to client files, calendars, sales numbers, invoices, etc is debilitating. Wondering how users can at least have "something" in the vent his happens again. Is it possible to have an offline version of Zoho so we aren't completely disabled?
Yes....Zoho being down is a major problem for all users. Why, because we use zoho for our mission critical business applications. Proof that Zoho provides great services.The cost of lost time and sending invoices...etc and customer communications across the globe is a huge number no doubt. Proof that Zoho is not an SMB providing simple solutions, but rather a global player in the market place.But when you think of an average customer having say 10 mail users, 10 CRM users at a cost of approx $270 per month.........and not having to contact anyone, do anything or pay a few hundred/thousand dollars an hour plus for a team to fix this problem.........It's priceless. A typical in server costs about 9K per year for a maint. contract plus the cost of the applications.Will Zoho perform a postmortem meeting.......absolutely.
Heads will roll I'm sure.
Topics like
geo location clustering
Minimum battery backup requirements
Raid levels
DB record locking interval, roll back segments, redo logs, extents, SAN redundancy
etc...will be discussed.Action plans will be developed and implemented.You and I not having to deal with this problem directly..........AWESOME!CISCO, IBM, ATT, Google, FaceBook, Twitter etc....have all had problems like this.
Accept they are not as transparent.Inconvenienced...yes. Lost faith...no This is the first time in three years that I have seen this happen.
How about redundant power? Any decent data center has redundant power, UPS and multiple power grids. Unless you're going cheap or the data center is lame, power shouldn't be an issue...
out of curiosity, how many people here are actually paying for the service? I'm not, which is why I guess I don't feel the need to light torches and bring out the pitchforks. Considering this is the first time I've ever seen this happen I'm giving these guys some room to breathe and cheer them on as they bust their humps to get this resolved. Good luck guys!
You should pay for this.
We had a huge loss because of this. How can a company like Google tie up with you?
You should be prepared for this. Your support also sucks. There is small IP issue which Iam facing and you guys cannot fix that. I sent 10 emails and whenever i Call the support number people say we"ll get back to you and they never do.
Zoho Sucks
I appreciate your transparency in the face of an angry mob and hope this outage provides the lessons necessary on how to avoid a repeat.
Needed to bill a customer today $1,500 so I can have some working capital. Unable to stay in business without cash. Zoho should have redundancy and emails to all customers. I always worry about leaving my valuable information and ability to conduct business in someone else's hands. That said, my computer could barf as well, leaving me in the lurch.
Nice New Zealand is up and running. Thanks guys at Zoho for your hard work :)
Ok hope you guys get it sorted soon. Anyway I like Zoho its the best cloud platform out their not even Google comes close. Maybe Exquinix need to get their asses fired or Zoho needs to get their own inhouse servers running Cisco UCS blade servers, UPS backup and diesel generators. The problem here then is having to rely on Exquinix cocolation servers in which Zoho have no real control over, and are subject to their failures.
This is horrible!! Come on guys! Guess I need to find another solution besides zoho. Unbelievable!!
Apologize for this. The power failure caused database inconsistencies. That is causing the delay.
Cloud is still traditional. There is not some magical place out there in internet land where things are stored with no power and no servers. The "cloud" is just a location or cluster of locations that have servers that run on power that can be shut down like anything else.
@CaryApologize for this. Yes, we are going to do a postmortem on this and will publish it in Zoho blogs. We are going to look at all the systems closely.
This is absolutely unacceptable. Whatever your reasons are, there is no reason this many businesses should have a service outage for this long.This is proof of a lack of redundancy. One area went out of power, which means you had no backup for the users in that area. This is absolutely horrible IT planning.
I hope you will post a post mortem explaining what happened and what will be done (and when) so that this situation can be avoided in the future.
My biggest disapointment and surprise is that the server architecture IS NOT cloud based but rather traditionnal which isn't what it's said in some of your promotional videos nor what some reviewers said about Zoho.
Its unacceptable this situation. First no countermeasures and failover system. I can't access my account since 3 hours ago. My clients got desperate. No invoicing, no reports, nothing. It is important to keep credibility for this cloud solutions. A lot of people are angry.
@JimWe do have a weekly backup and restore tests etc. When it comes to power, it is handled by the colocation provider - Equinix. They do some of these tests. They are trying to figure out what went wrong.
We just changed our company email over to zoho this morning on my recommendation. It's been down for almost 4 hours. :(
Dear Zoho,I'm a big fan of your services, especially ZOHO CRM. I'm constantly letting others know about ZOHO CRM especially when it's a small to mid size company, in my opinion is your perfect market.HOWEVER, I find it very hard to imagine that redundancy doesn't just mean having multiple redundant servers in one location. If you have redundancy it should be setup at multiple locations globally. The original idea of the Internet in the first place was continued data communication for the military (DARPA funded, Not invented by Al Gore) even if one location goes down that the data is rerouted through the system. The actual first military operation to prove it's original purpose (would you believe) was Sadam Husein during the Operation Desert Storm campaign (but that's a whole other story not to get into at the moment). Come on guys! If you have redundancy in multiple locations how can you be down for so long? In any case, I love ZOHO CRM, I think you guys have done a tremendous job with the applications. Keep up the great work but figure out a way that your servers have true redundancy at multiple locations, globally.Best Regards,Paul Ahlefeld
Performance Consultant
Dale Carnegie Training
We have a huge back up of estimates that we need to get to clients to confirm jobs this is going to cause us allot of over time for staff to be able to catch up. I really hope that you will look closely at your systems and try and prevent this from happening again. I appreciate the updates it helps elevate some of our anger but this is very disapointing
Not happy that it has been down all day, we rely on the CRM for all kinds of our business operations from sales to service.I will admit that it is being handled as best it can be, and I'm sure the Zoho leadership is not having a good day.However, it still hurts my wallet.I would love to have some kind of local snapshot capability so we could at least access customer phone numbers during a Zoho or internet service outage.
It is West Coast US.
Does Zoho not have a weekly staged drill for power outage and weekly power backup tests. This prepares the staff for the actual real event when it does happen, instead of just relying on the backup to kickin in the hope it will restore order. With 5 million customers I find it hard to believe its a power outage or this is either pure ignorance, negligence, cover up, or Zoho has been hacked. if you had protocols and procedures for staff to follow, services would have been restored along time ago. From New Zealand.
the times that are shown... is that East Coast US?
Thank you for keeping us posted with the status of the issue. This really makes a difference in how your customers view your product. Keep up the good work and fight the good fight.Regards,
I am waiting on an urgent email and cant access it. I use this for business and this is the second time in as many months that service has been interrupted during business hours.
Thanks for keeping us posted. We only use zoho for a few things and obviously we want it fixed too! but it's great to know the status is easily accessible through your blog. it's a much better situation than when our cable/internet/phones go down, which has happened at least 3 times recently.thanks
bk
@TonyWe do have multiple locations. We couldn't fail over. Will provide more details in a followup post.
@simon
Apologize for this. We are trying to restore the services as soon as possible.
You guys need to get set up in multiple locations with fail over and quick. A 15 minute interruption is one thing but over three hours is totally unacceptable. We took CRM out of our offices and to Zoho to protect us against this sort of thing. Not happy.
We do have backups. The issue is, power outage took down all our apps. Bring them back up requires a lot of dependency checks. That is taking time.No data will be lost.
Maybe you guys at Zoho can get this order right.Can I get a dozen donuts and a super-sized slurpee!
OK, Thanks - any idea how long now? It is nearly 7.30pm here in the UK, and I should have gone home two hours ago, but cannot leave until I have logged into zoho
I find it hard to believe that a company as huge as Zoho.com does not have backup servers set in place. Even the dr.s office around the corner has them. I have been pinging zoho.com for hours with no response. You guys better hurry up, cause I got deadlines. It would be the biggest epic fail in crm history if you guys lost everybodies data!!!
@Simon
Our Accounts service has to be up before restoring Mail. It is needed to login into your mail account.
If mail and crm are the first to be brought up, how come accounts is now up and mail not?
@BillThis is a power failure in Equinix taking all our services down (along with other companies). This is the first time such a thing happened in the past 7 years. We did have an application related outage on few of our apps. Unfortunately, power outage is out of our control. We are trying to get details on why Equinix power backup didn't kick in.
Totally Agree with you. Google has been down more often in the last 3 years !
I love Zoho, but this is a pain.... had so much lead work to do.
Actually, this is the second time I can recall an interruption of service. I hope this is fixed soon; we rely on you guys (maybe a little too much?). I appreciate the frequent updates; it helps.
@Dan,CRM & Mail wil be the first services to be restored.
No updates since 9:50am. It's now 1:40pm EST and we still cannot access Zoho CRM. We are unable to operate out business today until this is fixed! Please advise asap.
We do have multiple data centers. We will provide a detailed postmortem on what happened and we are unable to switch.
I was a little annoyed when I first tried to log in this morning and found zoho to be down. But then I realize this was the first time this has happened...EVER in the 3 or so years I've been using it, so I guess you guys have earned a little bit of slack :-)Hope you figure it out soon, good luck!
@ Equinix does have a backup UPS and a Generator. We are waiting for an explanation from them on why this happend with dual backup.
when we can back to zoho, just let me know. Thanks.
We just cant sign in.
This is very helpful. I went to your website, quickly found your blog, and I see up-to-the-minute updates regarding the power outage. Thank you so much.
Thank you for the updates, will be waiting your post-mortem on this on what number of data centers you have, why they do not have backup power etc...
Are you saying that you have only one location with zoho apps infrastructure ? I guess you should think about BCP and DRP, I don't want to be affected as a result of power failure in one data centre, not in the XXI century.
wow - sure woulda thunk a juggernaut (?) such as Zoho would have had in place automatic load-balancing and fail-over and redundancy and such. As a user, it is disappointing to learn they don't - especially in this way. oops
Twitter feed providing answers, thanks.
Hi. This is a major issue for my company right now. I have people calling me to ask where their reports are for meetings they are going to very soon...
Can you please provide us more information on this "network issue?"
-What caused it
-What is an estimated timeline for resolution?
Thanks.