Have you ever had the same email sent multiple times to the same customer during a campaign? Ever had the same lead assigned to multiple sales reps, and had to deal with an irate prospect? If you have, then you know that’s a surefire way for losing existing and even potential customers. One of the primary culprits for mistakes like those is duplicate data: records with the same email or same phone numbers, but stored under different names.
Let’s look at some of the ways that duplicate data—and bad data in general—can affect your organization.
- Higher consumption of resources
- Higher maintenance cost
- Higher spam counts
- Higher churn rates
- Lower customer satisfaction and retention
- Invalid reports
- Lowered productivity
- Loss of revenue
Research by Experian Information solutions shows that businesses are losing 14% of their revenue due to bad data. This equates to a staggering $258.278 million just in the UK alone. But revenue is not the only aspect of a business to take a hit due to rogue data: The research also showed that around 21% of UK businesses have taken a reputation hit as well.
If you wish to conduct your business smoothly without unwanted expenses, get rid of all duplicates! They are detrimental to your growth.
An organization spends on average about $14 million on data quality. Kissmetrics breaks down how much bad data, which includes duplicates, costs the company in terms of revenue, and the difference it makes to have good quality data to work with.
It‘s unreasonable for organizations to spend so much money on data quality. That’s why we’ve made it our mission to ensure that Zoho CRM is capable of inherently assisting an organization in maintaining data quality.
We’ve approached duplicates like a disease. For any affliction, there are two ways you can deal with it. You can either prevent it before it affects youor, in case you’ve already been affected, the only thing you can do is remedy it. It’s the same with duplicate data.
The prevention
We’ve introduced something called Unique Fields. You can choose to mark particular fields as unique (ex: name, phone number, etc.) through the layout setting option. Every time a new record with the same variable in any of the fields marked unique is being pushed into the CRM, a prompt pops up saying that a duplicate is about to be created and the action will not be allowed. This is a preventive measure in Zoho CRM to make sure no duplicates are generated.
The remedy
To deal with duplicates already present in the CRM, we have the de-duplicate option in the Leads, Contacts, Vendors and Accounts module. The de-duplication function is present under the “more options” button. The process starts once you decided upon the fields you wish to check for duplicates.
An automated de–duplication process will run in the background, merging records with completely identical variables in the selected fields. Once it’s done, you’ll be notified that the process is complete.
De–duplication in Zoho CRM used to be a manual process, but now we’ve made sure you don’t have to waste time sorting through all those records. So now, the only manual de–duplication you need to worry about is when there are conflicts across a particular field. A prompt will show up after the process is completed to let you know that duplicates with conflicting records were found.
This is the only manual de–duplication you need to do since it’s your call on which records need to be merged. After you’ve thought about it, select which of the conflicting records you want to merge and click “Merge”.
Duplicate data are a pain to deal with once inside your system. Apart from improving your efficiency, maintaining good data quality can have a direct impact on your revenue. The chart from Ringlead below shows that a strong organization can generate about 66% more revenue than an average organization based on the quality of data alone.
(Source: Ringlead)
Maintain a clean database with Zoho CRM. Get started today! Your feedback on what you think can be improved would be of great assistance to us. Please feel free to leave your thoughts in the comments below.
Question: I have a lot of duplicate leads. Our leads are primary students at a university. We collect their data at many different points through their college experience (I recognize this is the cause of my duplicates). Our leads frequently change one or two fields attached to them, (ex: their major or graduation date) and we come up with duplicates that have slightly different information. Is there any way to deduplicate these leads and choose to always take the newer information entered? For example, a freshman enters in all of their contact information and indicates they are studying computer science. As a sophomore, they come to an event of ours and fill out their information again. This time, however, they indicate their major is biology. Is there any way that I can automatically merge this duplicate and rewrite the old major (computer science) with the new major (biology) without having to accept and select every single duplicate? Thank you for your time!
Could Zoho learn to tell the difference between 3334445555, 333-444-5555, 333.444.5555 and (333) 444-5555 so when checking for duplicates (phones, addresses, names, etc.) if could intelligently suggest possible duplicates and cut down on manual checking?
If we deduplicate on the Accounts module will all related records be associated with the master record, e.g. Potential records? If so, does that include custom module records? Example. Accounts Record 1 - XYZ Co. Record 2 - XYZ Plc. New Accounts master record is deduped to XYZ Plc. Potentials. Potential 1 Account is XYZ Co. What will be the related Accounts record be after the deup? Custom module 1 Custom id field 1 Account XYZ Co. What will be the related Accounts record be after the deup?
Good day, Allan. The Account 'XYZ Plc.' will have all the related records of merged account 'XYZ co'. Including the custom module.
give me back the option to manually dedup. I like the auto dedupe, but the link that comes with the failed, only lasts for xx days. I would want to keep that link/list and work at it.
Improve your customer service. Your customer service reps take a long time to respond in chat, even after the conversation has started. Also, some make mistakes because they aren't very knowledgeable about the system. It is so frustrating to the point that I sometimes consider switching to another CRM.
CAn you please create a de-duplicate option for the "Deals" tab. My pipeline is twice the size it should be.
Hi, Peter. We're working hard to provide the deduplication for every module. But could we know how you ascertained that your pipeline looks bloated due to duplicates?
I would suggest that you just create a checkbox next to the list of opportunities under an account, and create a Merge feature to combine them, allowing the user to select which conflicting values to store on the resulting merged record.
I like that this will run in the background - but would like to see more variables for deduplication of accounts. One account could have same name and phone number but different locations. This would then merge the accounts but really need to remain separate. It would seem that address should be a critical match factor.
Hi, Stacie. To address your concern, it would not auto-merge since the auto-merge only occurs for completely identical records. Records with variables like you have mentioned would throw a "conflicts present" result. Hope that clears things up.
Prevention is the first step in managing dupes. Our company does not sell to an individual, rather we sell to a company. A user will search for the name of a company within the Accounts module. All records that have a partial of the "search" word will be displayed. IF that results display could be sorted (on any of the results column headers) users would be able to find an existing record quicker rather than scrolling 10 records at a time to find if it is already in the database. The current results grid has no sorting feature.
Hi, Fred. Yes, the global search result page doesn't have a sort feature right now. We'll look into the feasibility of adding that feature in.
Would like to be able to download contact information to a *.csv or sync with my Google Contacts or at least be able to share a contact's information with another person. I'd like my contact's name to show up on my android phone when they call without having to enter the data in both ZOHO & Google Contacts.
i just tried this tool... And afterwards was told that I merged 439 duplicates. How do i see what 439 duplicates have been merged?
Hi Sonia, As of now, you can't see them. We are working on to show the master records ( alone ) on which the mirror duplicates are merged automatically. I will keep posted once it is done.
Hi team, love the function however, you really need to extend this to custom modules, we are a ZOHO partner and build crm systems for the insurance industry, we use custom modules for clients, claims, renewals, policies, endorsements, etc. The duplicate problem is not only restricted to leads and accounts, so an extension to custom modules would be very beneficial. kind regards GJ
Glad to have your feedback GJ! But there are scalability challenges. So, we started off with our basic modules. We will definitely open this option to custom modules once we are ready with our scalability challenges. I will keep you posted on this.
I'd love to see what you've built. Looking for something for my father-in-law's insurance agency.
Will de-duplicate work only within the LEAD ..OR... CONTACTS module? Or, will de-duplicate work across the two modules?
Hey Todd! The de-duplicate work with leads and contacts separately. Not across the modules. However, the duplication check will work for leads and converted leads (contacts).
We would like a pop up (or notification) if we are duplicating a field, but still have the option to create the duplicate lead. Is there something that would do this for us?
Sorry, Lorrie. As of now, there is no way that you a get notification of duplication and get away with that. We don't provide that option. Either allow duplicates or restrict duplicates.
I agree with Gary...please give the first & last name option! My customers move between companies thus the company and email change. Only the name stays the same. Thank you!
I would like to see the first and last name as an option.
Good day Gary. We can't give the combination of the first and last name. I understand your requirement. Any combination of fields is not possible as of now. But, it is because of the technical constraint. We can't promise anything on this.
How do I do this? Is there a step by step process?
Hey, Patty. Please have a look at this It details how the feature can be used.
How does the automatic duplication process determine ownership of the record, when the records have different owners? Any dupe files with different ownership, should by default be flagged for manual resolution. The records should dump into a common file which can then be sorted through, to determine proper ownership before merging. The sales team will go bonkers if any of their accounts are switched by automation to a different owner..
Hi, Mark. As of now, the mirror records are determined by leaving the owner field. Very rarely that you will end up having with "ALL THE FIELD VALUES ALIKE AND THE OWNERS, DIFFERENT. So, we don't consider the owner name as a differential factor, when all the field values are same. If you have any practical-glitch, please let us know. We will see how we can sort it out.
A very helpful solution would be to be able to give permission to "Merge Duplicates" to specific users/groups. Where an organization has a lot of duplicates, they are not likely going to have their executives or IT personnel process them. It would be someone that takes up less resources for their time, who is still familiar with the accounts.
Hey there, Rachael! The profiles which have "Find and Merge" permission alone can do the "Deduplication". You can assign the profile- with this permission enabled- to the users you want and manage.
A very helpful solution would be to be able to give permission to "Merge Duplicates" to specific users/groups. Where an organization has a lot of duplicates, they are not likely going to have their executives or IT personnel process them. It would be someone that takes up less resources for their time, who is still familiar with the accounts.
Existing in my system already. I have duplicates that are both leads and contacts. Do you have any way to help me match these for deduplication?
Hey, Marty. You can't de-dupe the leads and contacts together. It is applicable only to one module. When you actually convert a lead and when a contact already exists, then the conversion process will take care of, to merge with the existing contact or duplicate it.
For the record, it is technically feasible to identify duplicates between Leads and Contacts using a union join between the two tables. Certainly might take a little out-of-the-box thinking here, but you all could do it. We could certainly benefit from that.
Never mind! I see the option in there now. going to test it.Thanks!
"records with completely identical variables" This is still a problem. If the company name varies slightly such as inc or co, Dedupe will not catch it. I'd prefer that it give suggested dedupe options along with automatically merging exact matches.
Yes, agree with you Preston. Most of the duplicate entries in the system happen due to slight mismatch of names (Accounts and Contacts) even when a search is done to check for duplicates before creating a new record. Having a suggested dedupe option would be very useful. That's actually how we search for duplicates...by entering partial names to get a better match.
Can you add this to all the tabs? I want to de-duplicate People, Companies, etc.
It would be better to prevent duplicate from being entered before they are put into the CRM. This new deduplication feature only merges profiles that match exactly. Different users may be putting in different data so there is still a manual process. You should also consider deduping on not only the unique fields but a combination of unique fields and other fields the company knows that is more indicative of a duplicate. In our company we use Contacts for married couples or people that live in the same building. It would be more effective if we the mass deduplication merged contacts that had the same first name, last name, phone and address. This is something that I have been requesting for more than a year and your helpdesk says it's impossible. But until this change is made, your new feature still does not help to clean up our data.
Good day, Nikita. I can't agree more. The combination of unique field is not required. Since it is unique you can't duplicate it. But a provision to mark UNIQUENESS based on the combination of other two or three fields should be supported. But, the reason why we did not provide the combination of fields is purely due to the technical constraint. We are striving hard to cross over technical hurdles. It would take time. We will keep you posted on this.
I've repeatedly suggested that Zoho create a checkbox on records in search results to enable you to merge duplicates right there and then. But you have to select one, then go and find the duplicate again, eliminating select pre-filled fields and checking your and/or logic, just to get back to where you were when you did your initial search. It's no wonder why our people don't clean up duplicates more often. I'm certainly curious about this automated de-dupe function, but don't have a high degree of trust that it will run smoothly and without unintentional error. I'd probably prefer to test it on a copy before running it on our live database.
Good day Steve! This is a nice suggestion. We'll look into the possibilities.
What about duplicate data across Leads and Accounts? For example, "The X Company" is a customer in Accounts module. However, marketing uploaded a new lead list to Leads module that also includes "The X Company" in the list. Then sales calls the lead, not knowing that it is also an Account. How can we prevent this from happening?
We have the same problem, can this be solved by the automated de-dupe function update? i'm curious as well but do not trust that it will run smoothly either...
This is one of our biggest problems and would like to see the remedy across modules and file import prevention as well.
Hey, Brandon! Basically, you would like to prevent the duplicate leads from the already converted leads ( i.e the lead would have qualified and converted as an Account and Contact ). Right now, through API, Webforms, and Manual entry the duplicates will be stopped. However, all the other means of lead source will be covered soon. ( Import, Contact sync, etc., ) But, the deduplication ( remedy ) across the modules can't be done.
Just as with leads & contacts, it is technically feasible to identify duplicates between Leads and Accounts using a union join between the two tables. Again, this will require a little out-of-the-box thinking, but it could be done. This would be a big benefit to a lot of us!
How do others UPDATED multiple Price Books? I have 5 price books and OVER 300 products. Zoho states they only way I can update the 5 price books PER products in one by one manually. I just cannot believe, that there is not a better, easier way!!
Can you override the function that does not allow you to create a duplicate if you need one?
Good day, Lorrie. You can select or deselect the "Do not allow duplicate values" option. Please refer to this forum post and this help documentationCheers, Shashank
This is a great news, and we tested this feature last week. Really amazing. Thanks Zoho.
Happy to know you like the feature. Let us know how it helped you and what we can do to make your Zoho CRM experience better! Have a great day!
Yes! I really wanted the unique field option and now that you have it, you'll make my sales life easier
Glad we could help you Rehaf! Do let us know your thoughts on anything that could be added or improved. Cheers, Shashank.