Topic Out of Memory

Yasir started the topic:
2017-08-17 14:26

Out of Memory


I think your product is lovely. I am trying to de-dup my customer master records using this tool. I am just using demo version to get the look and feel. When I tried to process 20k records, the tool gave me out of memory exception. So I increased the java heap size to 4096m. Now I am able to process 50k records in de-duplication task but when I connect the merge task, I am again getting out of memory exception.

Any thoughts?

deleted user replied:
2017-08-18 08:19
That sounds extreme for so few records. Are the records very large, or are your criteria very wide so they end up as only a very few groups (very large groups are rather expensive, memory-wise)?

Best regards,
vivek replied:
2018-03-14 06:37

Could you please tell me how to increase the java heap size for data cleaner?

Arjan replied:
2018-03-14 07:37
The folder where you installed DataCleaner to contains a DataCleaner.l4j.ini file. You can add something like -Xmx4g to this file to enforce that DataCleaner has a maximum heap size of 4 GB when you started it (using DataCleaner.exe; if you start it using DataCleaner-console.exe, it uses the settings in the DataCleaner-console.l4j.ini file).

On the DataCleaner welcome screen, there is a More menu in the top. If you open this and open the Options dialog and then go to the Memory tab, you can see the actual memory statistics for the running DataCleaner instance.