Topic Profiler(1.5.1): lots of columns need horiz scroll

sapiorltd started the topic:
2009-05-14 15:10

Profiler(1.5.1): lots of columns need horiz scroll

Folks,
When profiling records structures with 70+ fields (have to use one with 1780 fields) and previewing the columns are not viewable.
Because each is minimally sized and there is no room to expand any of them a horizontal scroll bar is needed.
Regards,
Rob
Kasper Sørensen replied:
2009-05-16 07:45
Hi Rob,

The horisontal scrollbar is a matter of many tastes because it will often mess up the layout depending on the number of columns. The current solution to the problem have been to incorporate a function to swap the axes of the result tables. You can:
  • right click the table and select "swap table axis" (or something like that)
  • or, you can permanently enable swapped axes in the settings dialog.
I hope this is sufficient?
sapiorltd replied:
2009-05-18 09:31
Hi Kasper,

Thanks for responding. Your advice does seem to work for the output of a profile run. It does not work for the initial preview after opening a file.

However if I use my 1780 col file with 11600 rows for a date match the entire tool vanishes after about 15 seconds.

Include the contents of "hs_err_pid592.log" here for your interest. No idea why the java is failing for want of memory. PC has enough spare.

Regards,

Rob
-----------------------
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 131072000 bytes for GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
#
# Internal Error (allocation.inline.hpp:42), pid=592, tid=3700
# Error: GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp
#
# Java VM: Java HotSpot(TM) Client VM (11.3-b02 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x54c22800): VMThread [stack: 0x54cb0000,0x54d00000] [id=3700]

Stack: [0x54cb0000,0x54d00000]
[error occurred during error reporting (printing stack bounds), id 0x80000001]

VM_Operation (0x5614f660): GenCollectForAllocation, mode: safepoint, requested by thread 0x55031000


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x55031000 JavaThread "querythread-1" [_thread_blocked, id=2956, stack(0x56100000,0x56150000)]
0x54ca6800 JavaThread "Swing-Shell" daemon [_thread_blocked, id=3456, stack(0x56350000,0x563a0000)]
0x54f62400 JavaThread "TimerQueue" daemon [_thread_blocked, id=3864, stack(0x56250000,0x562a0000)]
0x003d6800 JavaThread "DestroyJavaVM" [_thread_blocked, id=3728, stack(0x009c0000,0x00a10000)]
0x55b04400 JavaThread "D3D Screen Updater" daemon [_thread_blocked, id=1528, stack(0x55980000,0x559d0000)]
0x55b12400 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=3008, stack(0x558c0000,0x55910000)]
0x550c2c00 JavaThread "AWT-Windows" daemon [_thread_in_native, id=4024, stack(0x55210000,0x55260000)]
0x5509a800 JavaThread "AWT-Shutdown" [_thread_blocked, id=2720, stack(0x551c0000,0x55210000)]
0x550b3800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=1736, stack(0x55170000,0x551c0000)]
0x54c44c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2496, stack(0x54e90000,0x54ee0000)]
0x54c3e800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=3552, stack(0x54e40000,0x54e90000)]
0x54c3d000 JavaThread "Attach Listener" daemon [_thread_blocked, id=1952, stack(0x54df0000,0x54e40000)]
0x54c3bc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3808, stack(0x54da0000,0x54df0000)]
0x54c2c000 JavaThread "Finalizer" daemon [_thread_blocked, id=1984, stack(0x54d50000,0x54da0000)]
0x54c27800 JavaThread "Reference Handler" daemon [_thread_blocked, id=3632, stack(0x54d00000,0x54d50000)]

Other Threads:
=>0x54c22800 VMThread [stack: 0x54cb0000,0x54d00000] [id=3700]
0x54c58400 WatcherThread [stack: 0x54ee0000,0x54f30000] [id=2856]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x003d5818] UNKNOWN - owner thread: 0x54c22800
[0x003d5c78] UNKNOWN - owner thread: 0x55031000

Heap
def new generation total 72576K, used 72576K [0x02ab0000, 0x07970000, 0x07970000)
eden space 64512K, 100% used [0x02ab0000, 0x069b0000, 0x069b0000)
from space 8064K, 100% used [0x07190000, 0x07970000, 0x07970000)
to space 8064K, 0% used [0x069b0000, 0x069b0000, 0x07190000)
tenured generation total 967936K, used 950611K [0x07970000, 0x42ab0000, 0x42ab0000)
the space 967936K, 98% used [0x07970000, 0x419c4d48, 0x419c4e00, 0x42ab0000)
compacting perm gen total 19200K, used 19056K [0x42ab0000, 0x43d70000, 0x52ab0000)
the space 19200K, 99% used [0x42ab0000, 0x43d4c1f0, 0x43d4c200, 0x43d70000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00424000 C:\extras\Java\jre6\bin\javaw.exe
0x7c900000 - 0x7c9b2000 C:\WINDOWS\system32\ntdll.dll
0x7c800000 - 0x7c8f5000 C:\WINDOWS\system32\kernel32.dll
0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
0x77e70000 - 0x77f01000 C:\WINDOWS\system32\RPCRT4.dll
0x7e410000 - 0x7e4a0000 C:\WINDOWS\system32\USER32.dll
0x77f10000 - 0x77f58000 C:\WINDOWS\system32\GDI32.dll
0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL
0x6d710000 - 0x6d723000 C:\extras\KASPER~1\KASPER~1\mzvkbd.dll
0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
0x6d730000 - 0x6d743000 C:\extras\KASPER~1\KASPER~1\mzvkbd3.dll
0x7c340000 - 0x7c396000 C:\extras\Java\jre6\bin\msvcr71.dll
0x6d800000 - 0x6da56000 C:\extras\Java\jre6\bin\client\jvm.dll
0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
0x6bd00000 - 0x6bd0d000 C:\WINDOWS\system32\SYNCOR11.DLL
0x6d290000 - 0x6d298000 C:\extras\Java\jre6\bin\hpi.dll
0x6d7b0000 - 0x6d7bc000 C:\extras\Java\jre6\bin\verify.dll
0x6d330000 - 0x6d34f000 C:\extras\Java\jre6\bin\java.dll
0x6d7f0000 - 0x6d7ff000 C:\extras\Java\jre6\bin\zip.dll
0x6d000000 - 0x6d14a000 C:\extras\Java\jre6\bin\awt.dll
0x73000000 - 0x73026000 C:\WINDOWS\system32\WINSPOOL.DRV
0x77c10000 - 0x77c68000 C:\WINDOWS\system32\msvcrt.dll
0x774e0000 - 0x7761d000 C:\WINDOWS\system32\ole32.dll
0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03\comctl32.dll
0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
0x74720000 - 0x7476b000 C:\WINDOWS\system32\MSCTF.dll
0x552a0000 - 0x552a6000 C:\Program Files\McAfee\SiteAdvisor\saHook.dll
0x755c0000 - 0x755ee000 C:\WINDOWS\system32\msctfime.ime
0x7c9c0000 - 0x7d1d7000 C:\WINDOWS\system32\shell32.dll
0x6d230000 - 0x6d284000 C:\extras\Java\jre6\bin\fontmanager.dll
0x552c0000 - 0x55466000 C:\WINDOWS\system32\d3d9.dll
0x55470000 - 0x55476000 C:\WINDOWS\system32\d3d8thk.dll
0x77c00000 - 0x77c08000 C:\WINDOWS\system32\VERSION.dll
0x554a0000 - 0x554c8000 C:\WINDOWS\system32\rsaenh.dll
0x769c0000 - 0x76a73000 C:\WINDOWS\system32\USERENV.dll
0x5b860000 - 0x5b8b4000 C:\WINDOWS\system32\netapi32.dll
0x6d610000 - 0x6d623000 C:\extras\Java\jre6\bin\net.dll
0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
0x71a50000 - 0x71a8f000 C:\WINDOWS\System32\mswsock.dll
0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll
0x76d60000 - 0x76d79000 C:\WINDOWS\system32\iphlpapi.dll
0x76fb0000 - 0x76fb8000 C:\WINDOWS\System32\winrnr.dll
0x76f60000 - 0x76f8c000 C:\WINDOWS\system32\WLDAP32.dll
0x76fc0000 - 0x76fc6000 C:\WINDOWS\system32\rasadhlp.dll
0x6d630000 - 0x6d639000 C:\extras\Java\jre6\bin\nio.dll
0x605d0000 - 0x605d9000 C:\WINDOWS\system32\mslbui.dll
0x77120000 - 0x771ab000 C:\WINDOWS\system32\OLEAUT32.DLL
0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
0x563a0000 - 0x56665000 C:\WINDOWS\system32\xpsp2res.dll
0x77b40000 - 0x77b62000 C:\WINDOWS\system32\appHelp.dll
0x76fd0000 - 0x7704f000 C:\WINDOWS\system32\CLBCATQ.DLL
0x77050000 - 0x77115000 C:\WINDOWS\system32\COMRes.dll
0x56680000 - 0x566db000 C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\PDFShell.dll
0x78130000 - 0x781cb000 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.1433_x-ww_5cf844d2\MSVCR80.dll
0x71b20000 - 0x71b32000 C:\WINDOWS\system32\MPR.dll
0x75f60000 - 0x75f67000 C:\WINDOWS\System32\drprov.dll
0x71c10000 - 0x71c1e000 C:\WINDOWS\System32\ntlanman.dll
0x71cd0000 - 0x71ce7000 C:\WINDOWS\System32\NETUI0.dll
0x71c90000 - 0x71cd0000 C:\WINDOWS\System32\NETUI1.dll
0x71c80000 - 0x71c87000 C:\WINDOWS\System32\NETRAP.dll
0x71bf0000 - 0x71c03000 C:\WINDOWS\System32\SAMLIB.dll
0x75f70000 - 0x75f79000 C:\WINDOWS\System32\davclnt.dll
0x77920000 - 0x77a13000 C:\WINDOWS\system32\SETUPAPI.dll
0x56700000 - 0x5697e000 C:\WINDOWS\system32\wpdshext.dll
0x56980000 - 0x56b26000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.GdiPlus_6595b64144ccf1df_1.0.2600.3352_x-ww_81af8e88\gdiplus.dll
0x5ad70000 - 0x5ada8000 C:\WINDOWS\system32\UxTheme.dll
0x56b40000 - 0x56b89000 C:\WINDOWS\system32\PortableDeviceApi.dll
0x76c30000 - 0x76c5e000 C:\WINDOWS\system32\WINTRUST.dll
0x77a80000 - 0x77b14000 C:\WINDOWS\system32\CRYPT32.dll
0x77b20000 - 0x77b32000 C:\WINDOWS\system32\MSASN1.dll
0x76c90000 - 0x76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll
0x73d70000 - 0x73d83000 C:\WINDOWS\System32\shgina.dll
0x75970000 - 0x75a67000 C:\WINDOWS\system32\MSGINA.dll
0x76360000 - 0x76370000 C:\WINDOWS\system32\WINSTA.dll
0x74320000 - 0x7435d000 C:\WINDOWS\system32\ODBC32.dll
0x763b0000 - 0x763f9000 C:\WINDOWS\system32\comdlg32.dll
0x56d90000 - 0x56da7000 C:\WINDOWS\system32\odbcint.dll
0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll
0x56db0000 - 0x56e29000 C:\WINDOWS\system32\Audiodev.dll
0x56e30000 - 0x5708c000 C:\WINDOWS\system32\WMVCore.DLL
0x57090000 - 0x570ca000 C:\WINDOWS\system32\WMASF.DLL
0x6d1a0000 - 0x6d1c3000 C:\extras\Java\jre6\bin\dcpr.dll

VM Arguments:
jvm_args: -Xmx1024m -XX:MaxPermSize=256m
java_command: S:\downloads\datacleaner\1.5.1\datacleaner.jar
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=.;C:\extras\IBM\SQLLIB\java\db2java.zip;C:\extras\IBM\SQLLIB\java\db2jcc.jar;C:\extras\IBM\SQLLIB\java\sqlj.zip;C:\extras\IBM\SQLLIB\java\db2jcc_license_cu.jar;C:\extras\IBM\SQLLIB\bin;C:\extras\IBM\SQLLIB\tools\db2XTrigger.jar;C:\extras\IBM\SQLLIB\java\common.jar
PATH=d:\extras\IBMDebug\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\extras\IBM\SQLLIB\BIN;C:\extras\IBM\SQLLIB\FUNCTION;C:\extras\IBM\SQLLIB\SAMPLES\REPL;C:\extras\Java\jre6\bin
USERNAME=Rob Navarro
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 12 Stepping 0, AuthenticAMD



--------------- S Y S T E M ---------------

OS: Windows XP Build 2600 Service Pack 2

CPU:total 1 (1 cores per cpu, 1 threads per core) family 15 model 12 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, mmxext, 3dnow, 3dnowext

Memory: 4k page, physical 2095852k(153004k free), swap 4029568k(2255312k free)

vm_info: Java HotSpot(TM) Client VM (11.3-b02) for windows-x86 JRE (1.6.0_13-b03), built on Mar 9 2009 01:15:24 by "java_re" with MS VC++ 7.1

time: Mon May 18 10:22:30 2009
elapsed time: 333 seconds

Kasper Sørensen replied:
2009-05-18 10:51
Try downloading the platform-independent version of DataCleaner - it will contain an .cmd file instead of the .exe. You can edit this .cmd to allow the JVM to consume more memory by editing (adding to) the command line parameters called when starting up DataCleaner. I think the default is to consume up to 1 gig, but in your case maybe more is needed? Please let us know.
sapiorltd replied:
2009-05-25 15:04
Dear Kasper,

"java -Xmx1324m -XX:MaxPermSize=256m -jar datacleaner.jar"

fails within 10 secs after WinXP reports java.exe reaching ~900mb. Whole app simply vanishes.

"java -Xms512m -Xmx512m -XX:PermSize=256m -jar datacleaner.jar"

fails after ~20 secs after WinXP report java.exe reaching ~600mb. Get a nice unexpected exception report (No try block?) with following copied from Clipboard:

Unexpected exception


Message:
java.lang.OutOfMemoryError: Java heap space
Level:
SEVERE
Stack Trace:
Java heap space
java.lang.reflect.Array.newArray(Native Method)
java.lang.reflect.Array.newInstance(Unknown Source)
java.util.Arrays.copyOf(Unknown Source)
java.util.ArrayList.toArray(Unknown Source)
sun.java2d.d3d.D3DScreenUpdateManager.run(Unknown Source)
java.lang.Thread.run(Unknown Source)

--
Data file is only 22mb in size. It has 1780 columns and around 12000 rows. Why should the app require ~800mb to hold 22mb?

Regards,

Rob
Kasper Sørensen replied:
2009-05-27 15:22
Hmm no this sounds more and more like a bug. I will try to recreate it soon to do a JVM profile to find out why so much memory is consumed. I guess maybe we haven't been ambitious enough in our assumptions as to the size of (and thereby the method of reading) flat files.
sapiorltd replied:
2009-05-27 22:23
Dear Kasper,

To help you here's a dummy dataset generator that's similar to my orig and crashes datacleaner1.5.1 in same way:
--create file called gen_dat.awk--
NR == 1 {
printf("c0001")
for (i=2;i<=1780; ++i)
printf(",c%04d", i)
printf("\n")
}

{ for (i=1; i<=11999; ++i)
{
printf("%d", i+1)
for (j=2; j<= 1780; ++j)
{
if ((j % 54) == 1)
printf(",%d", i+j)
else
printf(",")
}
printf("\n")
}
}
---------
create dataset in Korn shell with:
print a | gawk -f gen_dat.awk > gen_dat.csv

This resulted in the error log:
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 131072000 bytes for GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
#
# Internal Error (allocation.inline.hpp:42), pid=2804, tid=2340
# Error: GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp
#
# Java VM: Java HotSpot(TM) Client VM (11.3-b02 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x54c22800): VMThread [stack: 0x54cb0000,0x54d00000] [id=2340]

Stack: [0x54cb0000,0x54d00000]
[error occurred during error reporting (printing stack bounds), id 0x80000001]

VM_Operation (0x5715f7bc): GenCollectForAllocation, mode: safepoint, requested by thread 0x5510c800


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x5510c800 JavaThread "querythread-1" [_thread_blocked, id=3812, stack(0x57110000,0x57160000)]
0x54f3f800 JavaThread "Swing-Shell" daemon [_thread_blocked, id=3860, stack(0x56370000,0x563c0000)]
0x54c49400 JavaThread "TimerQueue" daemon [_thread_blocked, id=1444, stack(0x56280000,0x562d0000)]
0x003d6800 JavaThread "DestroyJavaVM" [_thread_blocked, id=2868, stack(0x009c0000,0x00a10000)]
0x55524800 JavaThread "D3D Screen Updater" daemon [_thread_blocked, id=2356, stack(0x55e40000,0x55e90000)]
0x55580400 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=1440, stack(0x55ce0000,0x55d30000)]
0x550ac000 JavaThread "AWT-Shutdown" [_thread_blocked, id=3332, stack(0x551c0000,0x55210000)]
0x550b2800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=3112, stack(0x55210000,0x55260000)]
0x550a0000 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=3356, stack(0x55170000,0x551c0000)]
0x54c45000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2172, stack(0x54e90000,0x54ee0000)]
0x54c3e800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=3896, stack(0x54e40000,0x54e90000)]
0x54c3d000 JavaThread "Attach Listener" daemon [_thread_blocked, id=3552, stack(0x54df0000,0x54e40000)]
0x54c3bc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3612, stack(0x54da0000,0x54df0000)]
0x54c2c400 JavaThread "Finalizer" daemon [_thread_blocked, id=716, stack(0x54d50000,0x54da0000)]
0x54c27800 JavaThread "Reference Handler" daemon [_thread_blocked, id=4028, stack(0x54d00000,0x54d50000)]

Other Threads:
=>0x54c22800 VMThread [stack: 0x54cb0000,0x54d00000] [id=2340]
0x54c46400 WatcherThread [stack: 0x54ee0000,0x54f30000] [id=3644]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x003d58c8] UNKNOWN - owner thread: 0x54c22800
[0x003d5d28] UNKNOWN - owner thread: 0x5510c800

Heap
def new generation total 72576K, used 72576K [0x02ab0000, 0x07970000, 0x07970000)
eden space 64512K, 100% used [0x02ab0000, 0x069b0000, 0x069b0000)
from space 8064K, 100% used [0x07190000, 0x07970000, 0x07970000)
to space 8064K, 99% used [0x069b0000, 0x0718fff8, 0x07190000)
tenured generation total 967936K, used 967935K [0x07970000, 0x42ab0000, 0x42ab0000)
the space 967936K, 99% used [0x07970000, 0x42aafff8, 0x42ab0000, 0x42ab0000)
compacting perm gen total 18688K, used 18508K [0x42ab0000, 0x43cf0000, 0x52ab0000)
the space 18688K, 99% used [0x42ab0000, 0x43cc3360, 0x43cc3400, 0x43cf0000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00424000 C:\extras\Java\jre6\bin\javaw.exe
0x7c900000 - 0x7c9b2000 C:\WINDOWS\system32\ntdll.dll
0x7c800000 - 0x7c8f5000 C:\WINDOWS\system32\kernel32.dll
0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
0x77e70000 - 0x77f01000 C:\WINDOWS\system32\RPCRT4.dll
0x7e410000 - 0x7e4a0000 C:\WINDOWS\system32\USER32.dll
0x77f10000 - 0x77f58000 C:\WINDOWS\system32\GDI32.dll
0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL
0x6d710000 - 0x6d723000 C:\extras\KASPER~1\KASPER~1\mzvkbd.dll
0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
0x6d730000 - 0x6d743000 C:\extras\KASPER~1\KASPER~1\mzvkbd3.dll
0x7c340000 - 0x7c396000 C:\extras\Java\jre6\bin\msvcr71.dll
0x6d800000 - 0x6da56000 C:\extras\Java\jre6\bin\client\jvm.dll
0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
0x6bd00000 - 0x6bd0d000 C:\WINDOWS\system32\SYNCOR11.DLL
0x6d290000 - 0x6d298000 C:\extras\Java\jre6\bin\hpi.dll
0x6d7b0000 - 0x6d7bc000 C:\extras\Java\jre6\bin\verify.dll
0x6d330000 - 0x6d34f000 C:\extras\Java\jre6\bin\java.dll
0x6d7f0000 - 0x6d7ff000 C:\extras\Java\jre6\bin\zip.dll
0x6d000000 - 0x6d14a000 C:\extras\Java\jre6\bin\awt.dll
0x73000000 - 0x73026000 C:\WINDOWS\system32\WINSPOOL.DRV
0x77c10000 - 0x77c68000 C:\WINDOWS\system32\msvcrt.dll
0x774e0000 - 0x7761d000 C:\WINDOWS\system32\ole32.dll
0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03\comctl32.dll
0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
0x74720000 - 0x7476b000 C:\WINDOWS\system32\MSCTF.dll
0x552a0000 - 0x552a6000 C:\Program Files\McAfee\SiteAdvisor\saHook.dll
0x755c0000 - 0x755ee000 C:\WINDOWS\system32\msctfime.ime
0x7c9c0000 - 0x7d1d7000 C:\WINDOWS\system32\shell32.dll
0x6d230000 - 0x6d284000 C:\extras\Java\jre6\bin\fontmanager.dll
0x552c0000 - 0x55466000 C:\WINDOWS\system32\d3d9.dll
0x55470000 - 0x55476000 C:\WINDOWS\system32\d3d8thk.dll
0x77c00000 - 0x77c08000 C:\WINDOWS\system32\VERSION.dll
0x554a0000 - 0x554c8000 C:\WINDOWS\system32\rsaenh.dll
0x769c0000 - 0x76a73000 C:\WINDOWS\system32\USERENV.dll
0x5b860000 - 0x5b8b4000 C:\WINDOWS\system32\netapi32.dll
0x6d610000 - 0x6d623000 C:\extras\Java\jre6\bin\net.dll
0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
0x71a50000 - 0x71a8f000 C:\WINDOWS\System32\mswsock.dll
0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll
0x76d60000 - 0x76d79000 C:\WINDOWS\system32\iphlpapi.dll
0x76fb0000 - 0x76fb8000 C:\WINDOWS\System32\winrnr.dll
0x76f60000 - 0x76f8c000 C:\WINDOWS\system32\WLDAP32.dll
0x76fc0000 - 0x76fc6000 C:\WINDOWS\system32\rasadhlp.dll
0x6d630000 - 0x6d639000 C:\extras\Java\jre6\bin\nio.dll
0x605d0000 - 0x605d9000 C:\WINDOWS\system32\mslbui.dll
0x77120000 - 0x771ab000 C:\WINDOWS\system32\OLEAUT32.DLL
0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
0x563c0000 - 0x56685000 C:\WINDOWS\system32\xpsp2res.dll
0x77b40000 - 0x77b62000 C:\WINDOWS\system32\appHelp.dll
0x76fd0000 - 0x7704f000 C:\WINDOWS\system32\CLBCATQ.DLL
0x77050000 - 0x77115000 C:\WINDOWS\system32\COMRes.dll
0x56690000 - 0x566eb000 C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\PDFShell.dll
0x78130000 - 0x781cb000 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.1433_x-ww_5cf844d2\MSVCR80.dll
0x71b20000 - 0x71b32000 C:\WINDOWS\system32\MPR.dll
0x75f60000 - 0x75f67000 C:\WINDOWS\System32\drprov.dll
0x71c10000 - 0x71c1e000 C:\WINDOWS\System32\ntlanman.dll
0x71cd0000 - 0x71ce7000 C:\WINDOWS\System32\NETUI0.dll
0x71c90000 - 0x71cd0000 C:\WINDOWS\System32\NETUI1.dll
0x71c80000 - 0x71c87000 C:\WINDOWS\System32\NETRAP.dll
0x71bf0000 - 0x71c03000 C:\WINDOWS\System32\SAMLIB.dll
0x75f70000 - 0x75f79000 C:\WINDOWS\System32\davclnt.dll
0x77920000 - 0x77a13000 C:\WINDOWS\system32\SETUPAPI.dll
0x566f0000 - 0x5696e000 C:\WINDOWS\system32\wpdshext.dll
0x56970000 - 0x56b16000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.GdiPlus_6595b64144ccf1df_1.0.2600.3352_x-ww_81af8e88\gdiplus.dll
0x5ad70000 - 0x5ada8000 C:\WINDOWS\system32\UxTheme.dll
0x56b30000 - 0x56b79000 C:\WINDOWS\system32\PortableDeviceApi.dll
0x76c30000 - 0x76c5e000 C:\WINDOWS\system32\WINTRUST.dll
0x77a80000 - 0x77b14000 C:\WINDOWS\system32\CRYPT32.dll
0x77b20000 - 0x77b32000 C:\WINDOWS\system32\MSASN1.dll
0x76c90000 - 0x76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll
0x73d70000 - 0x73d83000 C:\WINDOWS\System32\shgina.dll
0x75970000 - 0x75a67000 C:\WINDOWS\system32\MSGINA.dll
0x76360000 - 0x76370000 C:\WINDOWS\system32\WINSTA.dll
0x74320000 - 0x7435d000 C:\WINDOWS\system32\ODBC32.dll
0x763b0000 - 0x763f9000 C:\WINDOWS\system32\comdlg32.dll
0x56d80000 - 0x56d97000 C:\WINDOWS\system32\odbcint.dll
0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll
0x56da0000 - 0x56e19000 C:\WINDOWS\system32\Audiodev.dll
0x56e20000 - 0x5707c000 C:\WINDOWS\system32\WMVCore.DLL
0x57080000 - 0x570ba000 C:\WINDOWS\system32\WMASF.DLL
0x6d1a0000 - 0x6d1c3000 C:\extras\Java\jre6\bin\dcpr.dll

VM Arguments:
jvm_args: -Xmx1024m -XX:MaxPermSize=256m
java_command: S:\downloads\datacleaner\1.5.1-win32\datacleaner.jar
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=.;C:\extras\IBM\SQLLIB\java\db2java.zip;C:\extras\IBM\SQLLIB\java\db2jcc.jar;C:\extras\IBM\SQLLIB\java\sqlj.zip;C:\extras\IBM\SQLLIB\java\db2jcc_license_cu.jar;C:\extras\IBM\SQLLIB\bin;C:\extras\IBM\SQLLIB\tools\db2XTrigger.jar;C:\extras\IBM\SQLLIB\java\common.jar;C:\extras\Java\jre6\lib\ext\QTJava.zip
PATH=d:\extras\IBMDebug\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\extras\IBM\SQLLIB\BIN;C:\extras\IBM\SQLLIB\FUNCTION;C:\extras\IBM\SQLLIB\SAMPLES\REPL;D:\extras\QuickTime\QTSystem\;C:\extras\Java\jre6\bin
USERNAME=Rob Navarro
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 12 Stepping 0, AuthenticAMD



--------------- S Y S T E M ---------------

OS: Windows XP Build 2600 Service Pack 2

CPU:total 1 (1 cores per cpu, 1 threads per core) family 15 model 12 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, mmxext, 3dnow, 3dnowext

Memory: 4k page, physical 2095852k(34748k free), swap 4029568k(2061604k free)

vm_info: Java HotSpot(TM) Client VM (11.3-b02) for windows-x86 JRE (1.6.0_13-b03), built on Mar 9 2009 01:15:24 by "java_re" with MS VC++ 7.1

time: Wed May 27 23:17:27 2009
elapsed time: 79 seconds

Regards,

Rob
Kasper Sørensen replied:
2009-05-28 15:05
Hi again,

Okay I've discovered where the bug is and it's definately because of a very optimistic (some may say naive) way of reading CSV files! Fixing it will not be hard but I'm pretty busy at the moment so if you have any Java skills you're definately invited to help :) If not, it's in the back of my head torturing me (ie. my prof. pride ;)) so as soon as I get the time I can look into it.

Here's the ticket, if you're interested: #291.
Kasper Sørensen replied:
2009-05-31 13:59
Ok, so I've fixed the CSV reading functionality now. Look out for a release soon (I will try to make it within a few weeks or maybe within the week). For now you can download the latest build from our build and test system.

Best,
Kasper
sapiorltd replied:
2009-05-31 22:37
Hi Kasper,

Downloaded the code @ the link you gave. Real data still causes a crash (see hs_err_pid included below). Sample data I provided did work however. It runs *really* slow. 8 mins for 1000 rows on a 2.4GHz CPU.

Curiously it still occupied over 1GB of RAM (screenshot in email - how do I upload a screenshot here?). I thought you'd moved to an "out-of-core" algorithm.

I realise that this leaves me in a tricky spot. Cannot send up real data but would like to help get this fault fixed.

Any suggestions or instrumented code you could prepare for me? Am happy to run and return the outputs.

Regards,

Rob

-----hs_err_pid3496.log-----
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 131072000 bytes for GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
#
# Internal Error (allocation.inline.hpp:42), pid=3496, tid=1976
# Error: GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp
#
# Java VM: Java HotSpot(TM) Client VM (11.3-b02 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x54c32800): VMThread [stack: 0x54cc0000,0x54d10000] [id=1976]

Stack: [0x54cc0000,0x54d10000]
[error occurred during error reporting (printing stack bounds), id 0x80000001]

VM_Operation (0x5716f4a8): GenCollectForAllocation, mode: safepoint, requested by thread 0x5518a400


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x5518a400 JavaThread "querythread-1" [_thread_blocked, id=3536, stack(0x57120000,0x57170000)]
0x54cb5800 JavaThread "Swing-Shell" daemon [_thread_blocked, id=3576, stack(0x56300000,0x56350000)]
0x003a6800 JavaThread "DestroyJavaVM" [_thread_blocked, id=2080, stack(0x009c0000,0x00a10000)]
0x55710c00 JavaThread "D3D Screen Updater" daemon [_thread_blocked, id=1328, stack(0x55e60000,0x55eb0000)]
0x556e3800 JavaThread "TimerQueue" daemon [_thread_blocked, id=2300, stack(0x55d40000,0x55d90000)]
0x556c9400 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=3248, stack(0x55cf0000,0x55d40000)]
0x551d1000 JavaThread "AWT-Windows" daemon [_thread_blocked, id=2980, stack(0x55220000,0x55270000)]
0x551a6c00 JavaThread "AWT-Shutdown" [_thread_blocked, id=1516, stack(0x54fd0000,0x55020000)]
0x551a6400 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=3972, stack(0x54f80000,0x54fd0000)]
0x54c54c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2476, stack(0x54ea0000,0x54ef0000)]
0x54c4e800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=4008, stack(0x54e50000,0x54ea0000)]
0x54c4d000 JavaThread "Attach Listener" daemon [_thread_blocked, id=3352, stack(0x54e00000,0x54e50000)]
0x54c4bc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3092, stack(0x54db0000,0x54e00000)]
0x54c38c00 JavaThread "Finalizer" daemon [_thread_blocked, id=352, stack(0x54d60000,0x54db0000)]
0x54c37800 JavaThread "Reference Handler" daemon [_thread_blocked, id=2848, stack(0x54d10000,0x54d60000)]

Other Threads:
=>0x54c32800 VMThread [stack: 0x54cc0000,0x54d10000] [id=1976]
0x54c56400 WatcherThread [stack: 0x54ef0000,0x54f40000] [id=3680]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x003a5890] UNKNOWN - owner thread: 0x54c32800
[0x003a5cf0] UNKNOWN - owner thread: 0x5518a400

Heap
def new generation total 68096K, used 0K [0x02ac0000, 0x074a0000, 0x07980000)
eden space 60544K, 0% used [0x02ac0000, 0x02ac0000, 0x065e0000)
from space 7552K, 0% used [0x06d40000, 0x06d400a8, 0x074a0000)
to space 7552K, 0% used [0x065e0000, 0x065e0000, 0x06d40000)
tenured generation total 907408K, used 907352K [0x07980000, 0x3efa4000, 0x42ac0000)
the space 907408K, 99% used [0x07980000, 0x3ef96028, 0x3ef96200, 0x3efa4000)
compacting perm gen total 19456K, used 19303K [0x42ac0000, 0x43dc0000, 0x52ac0000)
the space 19456K, 99% used [0x42ac0000, 0x43d99c30, 0x43d99e00, 0x43dc0000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00424000 C:\WINDOWS\system32\java.exe
0x7c900000 - 0x7c9b2000 C:\WINDOWS\system32\ntdll.dll
0x7c800000 - 0x7c8f5000 C:\WINDOWS\system32\kernel32.dll
0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
0x77e70000 - 0x77f01000 C:\WINDOWS\system32\RPCRT4.dll
0x7c340000 - 0x7c396000 C:\extras\Java\jre6\bin\msvcr71.dll
0x6d800000 - 0x6da56000 C:\extras\Java\jre6\bin\client\jvm.dll
0x7e410000 - 0x7e4a0000 C:\WINDOWS\system32\USER32.dll
0x77f10000 - 0x77f58000 C:\WINDOWS\system32\GDI32.dll
0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL
0x6d710000 - 0x6d723000 C:\extras\KASPER~1\KASPER~1\mzvkbd.dll
0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
0x6d730000 - 0x6d743000 C:\extras\KASPER~1\KASPER~1\mzvkbd3.dll
0x6bd00000 - 0x6bd0d000 C:\WINDOWS\system32\SYNCOR11.DLL
0x6d290000 - 0x6d298000 C:\extras\Java\jre6\bin\hpi.dll
0x6d7b0000 - 0x6d7bc000 C:\extras\Java\jre6\bin\verify.dll
0x6d330000 - 0x6d34f000 C:\extras\Java\jre6\bin\java.dll
0x6d7f0000 - 0x6d7ff000 C:\extras\Java\jre6\bin\zip.dll
0x6d000000 - 0x6d14a000 C:\extras\Java\jre6\bin\awt.dll
0x73000000 - 0x73026000 C:\WINDOWS\system32\WINSPOOL.DRV
0x77c10000 - 0x77c68000 C:\WINDOWS\system32\msvcrt.dll
0x774e0000 - 0x7761d000 C:\WINDOWS\system32\ole32.dll
0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03\comctl32.dll
0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
0x74720000 - 0x7476b000 C:\WINDOWS\system32\MSCTF.dll
0x552b0000 - 0x552b6000 C:\Program Files\McAfee\SiteAdvisor\saHook.dll
0x755c0000 - 0x755ee000 C:\WINDOWS\system32\msctfime.ime
0x7c9c0000 - 0x7d1d7000 C:\WINDOWS\system32\shell32.dll
0x6d230000 - 0x6d284000 C:\extras\Java\jre6\bin\fontmanager.dll
0x6d610000 - 0x6d623000 C:\extras\Java\jre6\bin\net.dll
0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
0x6d630000 - 0x6d639000 C:\extras\Java\jre6\bin\nio.dll
0x55b00000 - 0x55ca6000 C:\WINDOWS\system32\d3d9.dll
0x55aa0000 - 0x55aa6000 C:\WINDOWS\system32\d3d8thk.dll
0x77c00000 - 0x77c08000 C:\WINDOWS\system32\VERSION.dll
0x55cb0000 - 0x55cd8000 C:\WINDOWS\system32\rsaenh.dll
0x769c0000 - 0x76a73000 C:\WINDOWS\system32\USERENV.dll
0x5b860000 - 0x5b8b4000 C:\WINDOWS\system32\netapi32.dll
0x71a50000 - 0x71a8f000 C:\WINDOWS\System32\mswsock.dll
0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll
0x76d60000 - 0x76d79000 C:\WINDOWS\system32\iphlpapi.dll
0x76fb0000 - 0x76fb8000 C:\WINDOWS\System32\winrnr.dll
0x76f60000 - 0x76f8c000 C:\WINDOWS\system32\WLDAP32.dll
0x76fc0000 - 0x76fc6000 C:\WINDOWS\system32\rasadhlp.dll
0x605d0000 - 0x605d9000 C:\WINDOWS\system32\mslbui.dll
0x77120000 - 0x771ab000 C:\WINDOWS\system32\OLEAUT32.DLL
0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
0x563a0000 - 0x56665000 C:\WINDOWS\system32\xpsp2res.dll
0x77b40000 - 0x77b62000 C:\WINDOWS\system32\appHelp.dll
0x76fd0000 - 0x7704f000 C:\WINDOWS\system32\CLBCATQ.DLL
0x77050000 - 0x77115000 C:\WINDOWS\system32\COMRes.dll
0x56680000 - 0x566db000 C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\PDFShell.dll
0x78130000 - 0x781cb000 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.1433_x-ww_5cf844d2\MSVCR80.dll
0x71b20000 - 0x71b32000 C:\WINDOWS\system32\MPR.dll
0x75f60000 - 0x75f67000 C:\WINDOWS\System32\drprov.dll
0x71c10000 - 0x71c1e000 C:\WINDOWS\System32\ntlanman.dll
0x71cd0000 - 0x71ce7000 C:\WINDOWS\System32\NETUI0.dll
0x71c90000 - 0x71cd0000 C:\WINDOWS\System32\NETUI1.dll
0x71c80000 - 0x71c87000 C:\WINDOWS\System32\NETRAP.dll
0x71bf0000 - 0x71c03000 C:\WINDOWS\System32\SAMLIB.dll
0x75f70000 - 0x75f79000 C:\WINDOWS\System32\davclnt.dll
0x77920000 - 0x77a13000 C:\WINDOWS\system32\SETUPAPI.dll
0x56700000 - 0x5697e000 C:\WINDOWS\system32\wpdshext.dll
0x56980000 - 0x56b26000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.GdiPlus_6595b64144ccf1df_1.0.2600.3352_x-ww_81af8e88\gdiplus.dll
0x5ad70000 - 0x5ada8000 C:\WINDOWS\system32\UxTheme.dll
0x56b40000 - 0x56b89000 C:\WINDOWS\system32\PortableDeviceApi.dll
0x76c30000 - 0x76c5e000 C:\WINDOWS\system32\WINTRUST.dll
0x77a80000 - 0x77b14000 C:\WINDOWS\system32\CRYPT32.dll
0x77b20000 - 0x77b32000 C:\WINDOWS\system32\MSASN1.dll
0x76c90000 - 0x76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll
0x73d70000 - 0x73d83000 C:\WINDOWS\System32\shgina.dll
0x75970000 - 0x75a67000 C:\WINDOWS\system32\MSGINA.dll
0x76360000 - 0x76370000 C:\WINDOWS\system32\WINSTA.dll
0x74320000 - 0x7435d000 C:\WINDOWS\system32\ODBC32.dll
0x763b0000 - 0x763f9000 C:\WINDOWS\system32\comdlg32.dll
0x56d90000 - 0x56da7000 C:\WINDOWS\system32\odbcint.dll
0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll
0x56db0000 - 0x56e29000 C:\WINDOWS\system32\Audiodev.dll
0x56e30000 - 0x5708c000 C:\WINDOWS\system32\WMVCore.DLL
0x57090000 - 0x570ca000 C:\WINDOWS\system32\WMASF.DLL
0x6d1a0000 - 0x6d1c3000 C:\extras\Java\jre6\bin\dcpr.dll

VM Arguments:
jvm_args: -Xmx1024m -XX:MaxPermSize=256m
java_command: datacleaner.jar
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=.;C:\extras\IBM\SQLLIB\java\db2java.zip;C:\extras\IBM\SQLLIB\java\db2jcc.jar;C:\extras\IBM\SQLLIB\java\sqlj.zip;C:\extras\IBM\SQLLIB\java\db2jcc_license_cu.jar;C:\extras\IBM\SQLLIB\bin;C:\extras\IBM\SQLLIB\tools\db2XTrigger.jar;C:\extras\IBM\SQLLIB\java\common.jar;C:\extras\Java\jre6\lib\ext\QTJava.zip
PATH=d:\extras\IBMDebug\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\extras\IBM\SQLLIB\BIN;C:\extras\IBM\SQLLIB\FUNCTION;C:\extras\IBM\SQLLIB\SAMPLES\REPL;D:\extras\QuickTime\QTSystem\
USERNAME=Rob Navarro
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 12 Stepping 0, AuthenticAMD



--------------- S Y S T E M ---------------

OS: Windows XP Build 2600 Service Pack 2

CPU:total 1 (1 cores per cpu, 1 threads per core) family 15 model 12 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, mmxext, 3dnow, 3dnowext

Memory: 4k page, physical 2095852k(35852k free), swap 4029568k(2182972k free)

vm_info: Java HotSpot(TM) Client VM (11.3-b02) for windows-x86 JRE (1.6.0_13-b03), built on Mar 9 2009 01:15:24 by "java_re" with MS VC++ 7.1

time: Sun May 31 23:18:04 2009
elapsed time: 125 seconds

Kasper Sørensen replied:
2009-06-01 05:51
Hi Rob,

Ok, I will take a look at it as soon as possible, ie. not this week, because I'm at the Java One conference in SF so I probably won't have any awake hours left to dispose of ;-)
sapiorltd replied:
2009-06-02 14:35
Hi Kasper,

There're more signs of trouble. I left your latest version running against the test/generated dataset. It took over 11hrs to fail to process less than 12,000 rows. I attach the crash log below for interest.

Regards,

Rob

------hs_err_pid3496.log--------------
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 131072000 bytes for GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
#
# Internal Error (allocation.inline.hpp:42), pid=3496, tid=1976
# Error: GrET in C:\BUILD_AREA\jdk6_13\hotspot\src\share\vm\utilities\growableArray.cpp
#
# Java VM: Java HotSpot(TM) Client VM (11.3-b02 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

--------------- T H R E A D ---------------

Current thread (0x54c32800): VMThread [stack: 0x54cc0000,0x54d10000] [id=1976]

Stack: [0x54cc0000,0x54d10000]
[error occurred during error reporting (printing stack bounds), id 0x80000001]

VM_Operation (0x5716f4a8): GenCollectForAllocation, mode: safepoint, requested by thread 0x5518a400


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x5518a400 JavaThread "querythread-1" [_thread_blocked, id=3536, stack(0x57120000,0x57170000)]
0x54cb5800 JavaThread "Swing-Shell" daemon [_thread_blocked, id=3576, stack(0x56300000,0x56350000)]
0x003a6800 JavaThread "DestroyJavaVM" [_thread_blocked, id=2080, stack(0x009c0000,0x00a10000)]
0x55710c00 JavaThread "D3D Screen Updater" daemon [_thread_blocked, id=1328, stack(0x55e60000,0x55eb0000)]
0x556e3800 JavaThread "TimerQueue" daemon [_thread_blocked, id=2300, stack(0x55d40000,0x55d90000)]
0x556c9400 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=3248, stack(0x55cf0000,0x55d40000)]
0x551d1000 JavaThread "AWT-Windows" daemon [_thread_blocked, id=2980, stack(0x55220000,0x55270000)]
0x551a6c00 JavaThread "AWT-Shutdown" [_thread_blocked, id=1516, stack(0x54fd0000,0x55020000)]
0x551a6400 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=3972, stack(0x54f80000,0x54fd0000)]
0x54c54c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2476, stack(0x54ea0000,0x54ef0000)]
0x54c4e800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=4008, stack(0x54e50000,0x54ea0000)]
0x54c4d000 JavaThread "Attach Listener" daemon [_thread_blocked, id=3352, stack(0x54e00000,0x54e50000)]
0x54c4bc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3092, stack(0x54db0000,0x54e00000)]
0x54c38c00 JavaThread "Finalizer" daemon [_thread_blocked, id=352, stack(0x54d60000,0x54db0000)]
0x54c37800 JavaThread "Reference Handler" daemon [_thread_blocked, id=2848, stack(0x54d10000,0x54d60000)]

Other Threads:
=>0x54c32800 VMThread [stack: 0x54cc0000,0x54d10000] [id=1976]
0x54c56400 WatcherThread [stack: 0x54ef0000,0x54f40000] [id=3680]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x003a5890] UNKNOWN - owner thread: 0x54c32800
[0x003a5cf0] UNKNOWN - owner thread: 0x5518a400

Heap
def new generation total 68096K, used 0K [0x02ac0000, 0x074a0000, 0x07980000)
eden space 60544K, 0% used [0x02ac0000, 0x02ac0000, 0x065e0000)
from space 7552K, 0% used [0x06d40000, 0x06d400a8, 0x074a0000)
to space 7552K, 0% used [0x065e0000, 0x065e0000, 0x06d40000)
tenured generation total 907408K, used 907352K [0x07980000, 0x3efa4000, 0x42ac0000)
the space 907408K, 99% used [0x07980000, 0x3ef96028, 0x3ef96200, 0x3efa4000)
compacting perm gen total 19456K, used 19303K [0x42ac0000, 0x43dc0000, 0x52ac0000)
the space 19456K, 99% used [0x42ac0000, 0x43d99c30, 0x43d99e00, 0x43dc0000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00424000 C:\WINDOWS\system32\java.exe
0x7c900000 - 0x7c9b2000 C:\WINDOWS\system32\ntdll.dll
0x7c800000 - 0x7c8f5000 C:\WINDOWS\system32\kernel32.dll
0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
0x77e70000 - 0x77f01000 C:\WINDOWS\system32\RPCRT4.dll
0x7c340000 - 0x7c396000 C:\extras\Java\jre6\bin\msvcr71.dll
0x6d800000 - 0x6da56000 C:\extras\Java\jre6\bin\client\jvm.dll
0x7e410000 - 0x7e4a0000 C:\WINDOWS\system32\USER32.dll
0x77f10000 - 0x77f58000 C:\WINDOWS\system32\GDI32.dll
0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL
0x6d710000 - 0x6d723000 C:\extras\KASPER~1\KASPER~1\mzvkbd.dll
0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
0x6d730000 - 0x6d743000 C:\extras\KASPER~1\KASPER~1\mzvkbd3.dll
0x6bd00000 - 0x6bd0d000 C:\WINDOWS\system32\SYNCOR11.DLL
0x6d290000 - 0x6d298000 C:\extras\Java\jre6\bin\hpi.dll
0x6d7b0000 - 0x6d7bc000 C:\extras\Java\jre6\bin\verify.dll
0x6d330000 - 0x6d34f000 C:\extras\Java\jre6\bin\java.dll
0x6d7f0000 - 0x6d7ff000 C:\extras\Java\jre6\bin\zip.dll
0x6d000000 - 0x6d14a000 C:\extras\Java\jre6\bin\awt.dll
0x73000000 - 0x73026000 C:\WINDOWS\system32\WINSPOOL.DRV
0x77c10000 - 0x77c68000 C:\WINDOWS\system32\msvcrt.dll
0x774e0000 - 0x7761d000 C:\WINDOWS\system32\ole32.dll
0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03\comctl32.dll
0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
0x74720000 - 0x7476b000 C:\WINDOWS\system32\MSCTF.dll
0x552b0000 - 0x552b6000 C:\Program Files\McAfee\SiteAdvisor\saHook.dll
0x755c0000 - 0x755ee000 C:\WINDOWS\system32\msctfime.ime
0x7c9c0000 - 0x7d1d7000 C:\WINDOWS\system32\shell32.dll
0x6d230000 - 0x6d284000 C:\extras\Java\jre6\bin\fontmanager.dll
0x6d610000 - 0x6d623000 C:\extras\Java\jre6\bin\net.dll
0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
0x6d630000 - 0x6d639000 C:\extras\Java\jre6\bin\nio.dll
0x55b00000 - 0x55ca6000 C:\WINDOWS\system32\d3d9.dll
0x55aa0000 - 0x55aa6000 C:\WINDOWS\system32\d3d8thk.dll
0x77c00000 - 0x77c08000 C:\WINDOWS\system32\VERSION.dll
0x55cb0000 - 0x55cd8000 C:\WINDOWS\system32\rsaenh.dll
0x769c0000 - 0x76a73000 C:\WINDOWS\system32\USERENV.dll
0x5b860000 - 0x5b8b4000 C:\WINDOWS\system32\netapi32.dll
0x71a50000 - 0x71a8f000 C:\WINDOWS\System32\mswsock.dll
0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll
0x76d60000 - 0x76d79000 C:\WINDOWS\system32\iphlpapi.dll
0x76fb0000 - 0x76fb8000 C:\WINDOWS\System32\winrnr.dll
0x76f60000 - 0x76f8c000 C:\WINDOWS\system32\WLDAP32.dll
0x76fc0000 - 0x76fc6000 C:\WINDOWS\system32\rasadhlp.dll
0x605d0000 - 0x605d9000 C:\WINDOWS\system32\mslbui.dll
0x77120000 - 0x771ab000 C:\WINDOWS\system32\OLEAUT32.DLL
0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
0x563a0000 - 0x56665000 C:\WINDOWS\system32\xpsp2res.dll
0x77b40000 - 0x77b62000 C:\WINDOWS\system32\appHelp.dll
0x76fd0000 - 0x7704f000 C:\WINDOWS\system32\CLBCATQ.DLL
0x77050000 - 0x77115000 C:\WINDOWS\system32\COMRes.dll
0x56680000 - 0x566db000 C:\Program Files\Common Files\Adobe\Acrobat\ActiveX\PDFShell.dll
0x78130000 - 0x781cb000 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.1433_x-ww_5cf844d2\MSVCR80.dll
0x71b20000 - 0x71b32000 C:\WINDOWS\system32\MPR.dll
0x75f60000 - 0x75f67000 C:\WINDOWS\System32\drprov.dll
0x71c10000 - 0x71c1e000 C:\WINDOWS\System32\ntlanman.dll
0x71cd0000 - 0x71ce7000 C:\WINDOWS\System32\NETUI0.dll
0x71c90000 - 0x71cd0000 C:\WINDOWS\System32\NETUI1.dll
0x71c80000 - 0x71c87000 C:\WINDOWS\System32\NETRAP.dll
0x71bf0000 - 0x71c03000 C:\WINDOWS\System32\SAMLIB.dll
0x75f70000 - 0x75f79000 C:\WINDOWS\System32\davclnt.dll
0x77920000 - 0x77a13000 C:\WINDOWS\system32\SETUPAPI.dll
0x56700000 - 0x5697e000 C:\WINDOWS\system32\wpdshext.dll
0x56980000 - 0x56b26000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.GdiPlus_6595b64144ccf1df_1.0.2600.3352_x-ww_81af8e88\gdiplus.dll
0x5ad70000 - 0x5ada8000 C:\WINDOWS\system32\UxTheme.dll
0x56b40000 - 0x56b89000 C:\WINDOWS\system32\PortableDeviceApi.dll
0x76c30000 - 0x76c5e000 C:\WINDOWS\system32\WINTRUST.dll
0x77a80000 - 0x77b14000 C:\WINDOWS\system32\CRYPT32.dll
0x77b20000 - 0x77b32000 C:\WINDOWS\system32\MSASN1.dll
0x76c90000 - 0x76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll
0x73d70000 - 0x73d83000 C:\WINDOWS\System32\shgina.dll
0x75970000 - 0x75a67000 C:\WINDOWS\system32\MSGINA.dll
0x76360000 - 0x76370000 C:\WINDOWS\system32\WINSTA.dll
0x74320000 - 0x7435d000 C:\WINDOWS\system32\ODBC32.dll
0x763b0000 - 0x763f9000 C:\WINDOWS\system32\comdlg32.dll
0x56d90000 - 0x56da7000 C:\WINDOWS\system32\odbcint.dll
0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll
0x56db0000 - 0x56e29000 C:\WINDOWS\system32\Audiodev.dll
0x56e30000 - 0x5708c000 C:\WINDOWS\system32\WMVCore.DLL
0x57090000 - 0x570ca000 C:\WINDOWS\system32\WMASF.DLL
0x6d1a0000 - 0x6d1c3000 C:\extras\Java\jre6\bin\dcpr.dll

VM Arguments:
jvm_args: -Xmx1024m -XX:MaxPermSize=256m
java_command: datacleaner.jar
Launcher Type: SUN_STANDARD

Environment Variables:
CLASSPATH=.;C:\extras\IBM\SQLLIB\java\db2java.zip;C:\extras\IBM\SQLLIB\java\db2jcc.jar;C:\extras\IBM\SQLLIB\java\sqlj.zip;C:\extras\IBM\SQLLIB\java\db2jcc_license_cu.jar;C:\extras\IBM\SQLLIB\bin;C:\extras\IBM\SQLLIB\tools\db2XTrigger.jar;C:\extras\IBM\SQLLIB\java\common.jar;C:\extras\Java\jre6\lib\ext\QTJava.zip
PATH=d:\extras\IBMDebug\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\extras\IBM\SQLLIB\BIN;C:\extras\IBM\SQLLIB\FUNCTION;C:\extras\IBM\SQLLIB\SAMPLES\REPL;D:\extras\QuickTime\QTSystem\
USERNAME=Rob Navarro
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 12 Stepping 0, AuthenticAMD



--------------- S Y S T E M ---------------

OS: Windows XP Build 2600 Service Pack 2

CPU:total 1 (1 cores per cpu, 1 threads per core) family 15 model 12 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, mmxext, 3dnow, 3dnowext

Memory: 4k page, physical 2095852k(35852k free), swap 4029568k(2182972k free)

vm_info: Java HotSpot(TM) Client VM (11.3-b02) for windows-x86 JRE (1.6.0_13-b03), built on Mar 9 2009 01:15:24 by "java_re" with MS VC++ 7.1

time: Sun May 31 23:18:04 2009
elapsed time: 125 seconds

Kasper Sørensen replied:
2009-06-02 17:55
Just a small tip off the top of my head...

Can you tell me which profiles you are using? Specifically, the Value Distribution profile is known to be the most memory-insufficient one, so try removing it from the job if you are using it. Also make sure you only use the pattern finder on columns where you expect to see a reasonable number of patterns because storing thousands of patterns in memory can be quite expensive.

PS: You don't need to attach/print out the log any more, except if I ask for it :)
sapiorltd replied:
2009-06-03 09:01
Sure. I use only the "Date mask matcher". But I need it to check all the columns for me. Imagine selecting columns manually with 1780 columns!

Rob
Kasper Sørensen replied:
2009-06-12 17:30
Hi Rob,

OK - I've been digging deep into this bug for some time now and I'm pretty positive I got the bug out of there now! At least it runs well on my system when I loop through a 200 meg. CSV file. Try optaining the latest datacleaner from the nightly build site and let me know if it works!
Kasper Sørensen replied:
2009-06-13 18:17
Hi sapior,

As an early follow-up, I went ahead and did a small performance benchmark that might catch your interest:

http://kasper.eobjects.dk/2009/06/performance-benchmark-datacleaner.html

In short - if it's possible for you, it would greatly improve performance if you could cut down on the number of columns.
sapiorltd replied:
2009-06-15 08:39
Hi Kasper,

Yes this version now works but took around 2hrs. The results were then output to a Dojo Html file that also worked very slowly (kept timing out and asking if script should continue to be executed).

I see your other comments about selecting columns to reduce the number. It seems clear that unless I do reduce the col count the tool takes too long.

Thanks again for fixing the bug I stumbled across!

Regards,

Rob
Kasper Sørensen replied:
2009-06-15 10:53
Glad it worked!

The dojo-issue is not really DataCleaner related, but pertains to your browsers ability to handle all that Javascript. Basicly you have two options for handling the issues:

1. Use a faster browser (AFAIK google chrome is the fastest browser when it comes to javascript)
2. Create your own export format that produces a more simple "pure HTML" output. You can read about this in the datacleaner docs.
You are not signed in.
You need to be signed in to participate in the discussion.