Topic java.lang.OutOfMemoryError:

YKARTAL started the topic:
2017-05-17 20:56

java.lang.OutOfMemoryError:

Hi,

I have just downloaded trial version and got below error. My data is 497156 lines. Would anybody help me out with this?

Thanks,

Yildiz

Unexpected error!


Message:
java.lang.OutOfMemoryError: Java heap space
Level:
SEVERE
Stack Trace:
Java heap space
java.util.Arrays.copyOf(Unknown Source)
java.util.Arrays.copyOf(Unknown Source)
java.util.ArrayList.grow(Unknown Source)
java.util.ArrayList.ensureExplicitCapacity(Unknown Source)
java.util.ArrayList.ensureCapacityInternal(Unknown Source)
java.util.ArrayList.add(Unknown Source)
org.apache.xmlbeans.impl.store.Xobj.find_all_element_users(Xobj.java:2098)
org.openxmlformats.schemas.spreadsheetml.x2006.main.impl.CTSstImpl.getSiArray(Unknown Source)
org.apache.poi.xssf.model.SharedStringsTable.readFrom(SharedStringsTable.java:123)
org.apache.poi.xssf.model.SharedStringsTable.<init>(SharedStringsTable.java:106)
org.apache.poi.xssf.eventusermodel.XSSFReader.getSharedStringsTable(XSSFReader.java:82)
org.apache.metamodel.excel.XlsxSheetToRowsHandler.<init>(XlsxSheetToRowsHandler.java:87)
org.apache.metamodel.excel.XlsxSpreadsheetReaderDelegate.buildColumns(XlsxSpreadsheetReaderDelegate.java:193)
org.apache.metamodel.excel.XlsxSpreadsheetReaderDelegate.createSchema(XlsxSpreadsheetReaderDelegate.java:114)
org.apache.metamodel.excel.ExcelDataContext.getMainSchema(ExcelDataContext.java:169)
org.apache.metamodel.QueryPostprocessDataContext.getSchemaByNameInternal(QueryPostprocessDataContext.java:480)
org.apache.metamodel.AbstractDataContext.getSchemaByName(AbstractDataContext.java:206)
org.apache.metamodel.AbstractDataContext.getSchemas(AbstractDataContext.java:80)
org.datacleaner.connection.SchemaNavigator.getSchemas(SchemaNavigator.java:52)
org.datacleaner.widgets.properties.SchemaNamePropertyWidget.setDatastore(SchemaNamePropertyWidget.java:76)
com.hi.datacleaner.desktop.welcome.ColumnMeaningMappingPanel.<init>(ColumnMeaningMappingPanel.java:105)
com.hi.datacleaner.desktop.welcome.ColumnMeaningMappingPanel.<init>(ColumnMeaningMappingPanel.java:47)
com.hi.datacleaner.desktop.welcome.AbstractColumnMeaningBasedJobWizard$1.<init>(AbstractColumnMeaningBasedJobWizard.java:34)
com.hi.datacleaner.desktop.welcome.AbstractColumnMeaningBasedJobWizard.startWizard(AbstractColumnMeaningBasedJobWizard.java:34)
com.hi.datacleaner.desktop.welcome.WizardListPanel$2.datastoreSelected(WizardListPanel.java:109)
org.datacleaner.widgets.Dropzone.showFileChooser(Dropzone.java:202)
org.datacleaner.widgets.Dropzone$1.actionPerformed(Dropzone.java:109)
javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
javax.swing.DefaultButtonModel.setPressed(Unknown Source)
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
deleted user replied:
2017-05-18 06:18
H Yildiz,

Please look here for instructions on increasing the assigned memory to DataCleaner.

However, for performance I'd look into converting the file to CSV. Our Excel reader/writer is honestly not great with large files (I'm not sure if anything is with large Excel files, though).

Best regards,
Dennis
vivek replied:
2018-03-14 07:30
Hi Yildiz,

where do I find DataCleaner.l4j' file
Arjan replied:
2018-03-14 07:46
It can be found in the folder where you installed DataCleaner. It isn't called DataCleaner.l4j, but DataCleaner.l4j.ini.
vivek replied:
2018-03-14 08:24
Thanks for reply.
I dowloded datacleaner.zip and extracted but I could not find that file.
I have the following files and the folders in my datacleaner location after extraction.
files
DataCleaner-console.exe
DataCleaner.exe
DataCleaner.jar

Folders:
datastores
jobs
extensions
lib

Could you please help me out from this issue?

Thanks in advance
Arjan replied:
2018-03-14 08:26
Where did you download DataCleaner.zip from?
vivek replied:
2018-03-14 08:28

I am executing the DataCleaner-console.exe file to open DataCleaner to develop jobs.

Is it a right way of installation or is there any other way to use datacleaner.
vivek replied:
2018-03-14 08:30
This is the link where i got datacleaner.zip

https://datacleaner.org/get_datacleaner_ce#
Arjan replied:
2018-03-14 08:31
You can use either DataCleaner-console.exe or DataCleaner.exe to start DataCleaner. If you use DataCleaner-console.exe, you'll have a console where errors are logged. It doesn't change the working of DataCleaner itself.

You didn't install DataCleaner using an installer?
Arjan replied:
2018-03-14 09:02
The community edition doesn't contain the DataCleaner-console.l4j.ini and the DataCleaner.l4j.ini files. You can create a DataCleaner-console.l4j.ini file yourself in the same folder as DataCleaner-console.exe and make sure it looks like this:

 
# Runtime configuration for DataCleaner-console
-Xms256m -Xmx4g

If you then start DataCleaner, it should have a maximum java heap size of 4 GB.
vivek replied:
2018-03-14 09:30
Hi Yildiz,

Thank You.
It is working while opening DataCleaner-console.exe directly

But if I open DataCleaner through PDI kettle (Tools-->Start DataCleaner) , the increased memory is not allocated .It shows default max memory size of 1GB.

Could you please tell me the reason ?

Thanks in advance.
Arjan replied:
2018-03-14 09:37
I'm not sure how PDI kettle starts DataCleaner, so I don't know if it uses DataCleaner-console.exe or DataCleaner.exe (or if it maybe even starts DataCleaner in another manner. You could try to add a DataCleaner.l4j.ini file which is a copy of the DataCleaner-console.l4j.ini file you created, maybe that works. Otherwise, I don't know.
vivek replied:
2018-03-14 11:42
Ok.
Also we have DataCleaner.jar file in the installation folder and we can open DataCleaner through it.

How do i create .ini file for this jar file.Please let me know your suggestions.
Arjan replied:
2018-03-14 13:53
If you run it using the DataCleaner.jar file, you don't use a .ini file, if you run it using a jar file, you typically do this:

java -jar DataCleaner.jar

If you then want to influence the maximum java heap size, you can do this instead:

java -Xmx4g -jar DataCleaner.jar
vivek replied:
2018-03-15 04:03
Ok I will try this.

Thanks a lot.
You are not signed in.
You need to be signed in to participate in the discussion.