Very frequent JVM crash using jsmile on linux

The engine.
Mohsin Beg
Posts: 11
Joined: Tue Feb 12, 2008 2:46 am

Very frequent JVM crash using jsmile on linux

Post by Mohsin Beg »

I am using jsmile in an application server environment. Every couple of hours the application server jvm core dumps. I am not sharing/reusing DiagNetwork objects across threads. I am using the jsmile for linux released on 04.16.2009

Opening the jvm core in gdb reveals the below. Can anyone advise what might be the issue here or is this a bug ?
(gdb) where
#0 0xf6fe87a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0xf6eb2825 in raise () from /lib/tls/libc.so.6
#2 0xf6eb4289 in abort () from /lib/tls/libc.so.6
#3 0xf6ee6cda in __libc_message () from /lib/tls/libc.so.6
#4 0xf6eed56f in _int_free () from /lib/tls/libc.so.6
#5 0xf6eed94a in free () from /lib/tls/libc.so.6
#6 0xa73e4aa1 in operator delete () from /usr/lib/libstdc++.so.6
#7 0xa7509cf5 in Java_smile_DiagNetwork_updateNative () from /myhost/product/ias10g/lib/libjsmile.so
#8 0xf2f2afe0 in ?? ()
#9 0xa6850110 in ?? ()
#10 0xa527ccf4 in ?? ()
#11 0x0000000a in ?? ()
#12 0xf1d4d8a0 in ?? ()
#13 0xa527ccc0 in ?? ()
#14 0x00000000 in ?? ()
The environment details are as follows.
[myhost@user home]$ java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) Server VM (build 14.0-b16, mixed mode)
[myhost@user home]$ uname -a
Linux myhost 2.6.9-78.0.13.0.1.ELhugemem #1 SMP Wed Jan 14 19:35:00 EST 2009 i686 athlon i386 GNU/Linux
[myhost@user home]$ g++ -v
Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.6/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-java-awt=gtk --host=i386-redhat-linux
Thread model: posix
gcc version 3.4.6 20060404 (Red Hat 3.4.6-10.0.1)
[myhost@user home]$ ls -lag libstdc++.so.6.0.3
-rwxr-xr-x 1 root 792920 Jun 12 2008 libstdc++.so.6.0.3

[myhost@user lib]$ file libstdc++.so.6.0.3
libstdc++.so.6.0.3: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), stripped
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Re: Very frequent JVM crash using jsmile on linux

Post by shooltz[BayesFusion] »

Mohsin Beg wrote:I am using jsmile in an application server environment. Every couple of hours the application server jvm core dumps. I am not sharing/reusing DiagNetwork objects across threads. I am using the jsmile for linux released on 04.16.2009
During the 'couple of hours' timeframe, do you constantly run code in SMILE, or it's more like bursts of activity coming from client requests?
Mohsin Beg
Posts: 11
Joined: Tue Feb 12, 2008 2:46 am

Post by Mohsin Beg »

The crash was not seen during bursts (many parallel diag requests) but during low load activity (i.e. one diag request).

Also jsmile is responding to diag requests at random as they come in from our monitoring systems (i.e. not user initiated), hence is unpredictable at best.

Sincerely,

-Mohsin
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Mohsin Beg wrote:The crash was not seen during bursts (many parallel diag requests) but during low load activity (i.e. one diag request).
Can you run your Java application in Windows? I could prepare the native part of jSMILE built with debug checks.
Mohsin Beg
Posts: 11
Joined: Tue Feb 12, 2008 2:46 am

Post by Mohsin Beg »

Unfortunately no, since this is a pre-production environment and all test and production environments are linux only. Can I turn something on my side. I can put any debug statements in my code if you want.

Btw here is the excerpt from jvm pid.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xf6eecbdb, pid=23197, tid=1773140896
#
# JRE version: 6.0_14-b08
# Java VM: Java HotSpot(TM) Server VM (14.0-b16 mixed mode linux-x86 )
# Problematic frame:
# C [libc.so.6+0x61bdb]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- T H R E A D ---------------

Current thread (0x61b50800): JavaThread "pool-9-thread-82" [_thread_in_native, id=25718, stack(0x69aaf000,0x69b00000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000c

Registers:
EAX=0x63160000, EBX=0xf6fb5ff4, ECX=0x66000010, EDX=0x00000000
ESP=0x69afd3e4, EBP=0x69afd454, ESI=0x631655c8, EDI=0x631660a8
EIP=0xf6eecbdb, CR2=0x0000000c, EFLAGS=0x00210202

Top of Stack: (sp=0x69afd3e4)
0x69afd3e4: f6fb7858 00000000 f6fb7844 f6fb7848
0x69afd3f4: f6fb7820 f6faabf1 f6faabf1 f6faabf1
0x69afd404: f6faabf1 f6fb5ff4 f6fb7820 00000000
0x69afd414: 69afd4dc f6eed9b4 00000300 00000de0
0x69afd424: 66000048 00000000 66000034 66000018
0x69afd434: 66000010 69afd538 00000018 09501508
0x69afd444: 00000520 f6fb5ff4 66000010 00000000
0x69afd454: 69afd51c f6eedc30 f6faac43 00000084

Instructions: (pc=0xf6eecbdb)
0xf6eecbcb: 45 a4 89 45 ac eb 50 01 45 cc 8b 57 08 8b 47 0c
0xf6eecbdb: 39 7a 0c 0f 85 5c 01 00 00 39 78 08 0f 85 53 01

Stack: [0x69aaf000,0x69b00000], sp=0x69afd3e4, free space=312k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x61bdb]
C [libc.so.6+0x62c30]
C [libc.so.6+0x64aa1] malloc+0x81
C [libstdc++.so.6+0x9de87] _Znwj+0x27
C [libstdc++.so.6+0x9df99] _Znaj+0x1d
C [libjsmile.so+0x1c6868] _ZN12DSL_intArray7SetSizeEi+0xd8
C [libjsmile.so+0x1e2ff9] _ZN12DSL_rNetworkC1ER11DSL_networkb+0x79
C [libjsmile.so+0x1e7631] _Z8RelevantP11DSL_networkbb+0xd1
C [libjsmile.so+0x1f8579] _ZN45_GLOBAL__N_tsclustering.cxx_BF3F6D62_C97927F012DoClusteringEP11DSL_networkPbi+0x29
C [libjsmile.so+0x1f9c3e] _Z12TsClusteringP11DSL_network+0x1e
C [libjsmile.so+0x1cff58] _ZN11DSL_network15CallBNAlgorithmEv+0x28
C [libjsmile.so+0x1d4b65] _ZN11DSL_network13UpdateBeliefsEv+0x255
C [libjsmile.so+0x1be90e] _ZN23DSL_fastEntropyAssessor5SolveEv+0x49e
C [libjsmile.so+0xe5b2d] Java_smile_DiagNetwork_updateNative+0x5d
j smile.DiagNetwork.updateNative(I)Lsmile/DiagResults;+0
j smile.DiagNetwork.update()Lsmile/DiagResults;+5
j com.oracle.rootcause.diagnostic.impl.CausalDiagnostic.update()Lsmile/DiagResults;+4

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j smile.DiagNetwork.updateNative(I)Lsmile/DiagResults;+0
j smile.DiagNetwork.update()Lsmile/DiagResults;+5
j com.oracle.rootcause.diagnostic.impl.CausalDiagnostic.update()Lsmile/DiagResults;+4
J com.oracle.rootcause.diagnostic.impl.DiagnosticRunner.doDiagnosticRunForHost(Lcom/oracle/rootcause/diagnostic/impl/DiagnosticRunInfo;Lcom/oracle/ch/core/engine/metadata/DomainComponent;)Lcom/oracle/rootcause/diagnosticrun/recorder/IDiagnosticRunRecorder;
J com.oracle.rootcause.diagnostic.impl.DiagnosticRunner.runDiagnostic()Lcom/oracle/rootcause/diagnosticrun/recorder/IDiagnosticRunRecorder;
j com.oracle.rootcause.diagnostic.impl.DiagnosticRunner.call()Lcom/oracle/rootcause/diagnosticrun/recorder/IDiagnosticRunRecorder;+1
j com.oracle.rootcause.diagnostic.impl.DiagnosticRunner.call()Ljava/lang/Object;+1
j java.util.concurrent.FutureTask$Sync.innerRun()V+30
j java.util.concurrent.FutureTask.run()V+4
j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j java.util.concurrent.FutureTask$Sync.innerRun()V+30
j java.util.concurrent.FutureTask.run()V+4
j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+59
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Mohsin Beg wrote:Unfortunately no, since this is a pre-production environment and all test and production environments are linux only. Can I turn something on my side. I can put any debug statements in my code if you want.
I'll prepare the Linux build with extra debug checks then. Can you write some code which puts an artificial pressure on jSMILE? It would be best if the crash become easily reproducible.
Btw here is the excerpt from jvm pid.log
Interesting. The stack here is points to crash in DSL_intArray::SetSize, but the info in your post at the top of the topic suggests an issue with operator delete/free().
Mohsin Beg
Posts: 11
Joined: Tue Feb 12, 2008 2:46 am

Post by Mohsin Beg »

Hi,

The core.25595 (at the top of this post) was from
Jul 18 2009 07:42
no pid file produced at this time

and the hs_err_pid23197.log was from
Jul 20 2009 10:03
no core produced at this time

Sorry, for the confusion.

If you don't mind, can you please email me at mohsin.beg@oracle.com when the build is available so I may try it out.

Sincerely,

-Mohsin
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

If you don't mind, can you please email me at mohsin.beg@oracle.com when the build is available so I may try it out.
I'm working on the debug build at the moment; I'll ping you when it's ready.
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Debug build is ready. I have sent you an email with the link to downloadable file.
Mohsin Beg
Posts: 11
Joined: Tue Feb 12, 2008 2:46 am

Post by Mohsin Beg »

Hi,

Looks like core-dumps were happening - we did not see them since an oc4j deamon was restarting the server. The ulimit was set for coredumpsize of 0. This has now been changed to see what happens with debug libjsmile.so (that you sent).

Without the core-dump the only message we see is something like this in the log files. (I am going to start our server with the right ulimit)
*** glibc detected *** free(): invalid next size (fast): 0x5de1ca38 ***
Any suggestions in the meanwhile ?

Sincerely,

-Mohsin[/quote]
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Mohsin Beg wrote:Any suggestions in the meanwhile ?
Looks like pointer mismatch or memory overwrite somewhere. I was hoping for an early assert() from SMILE built without -DNDEBUG.

Is your application multithreaded? If so, are you using jSMILE objects from multiple threads?
Last edited by shooltz[BayesFusion] on Fri Jul 31, 2009 2:19 pm, edited 1 time in total.
Mohsin Beg
Posts: 11
Joined: Tue Feb 12, 2008 2:46 am

Post by Mohsin Beg »

Hi,

The diagnetwork objects are not shared across threads.
Each thread has it's own object.

Meanwhile, the jvm crashed again. A core just got produced.
[myuser@myhost home]$ gdb java core.20677
GNU gdb Red Hat Linux (6.3.0.0-1.159.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `/psecho/product/ias10g/jdk/bin/java -server -mx2048M -ms512M -Xrunjdwp:transpor'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libcwait.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libcwait.so
Reading symbols from /lib/tls/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/jli/libjli.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/bin/../jre/lib/i386/jli/libjli.so
Reading symbols from /lib/libdl.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/server/libjvm.so...
(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/server/libjvm.so
Reading symbols from /lib/tls/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/tls/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libverify.so...
(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libverify.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libjava.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libjava.so
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/native_threads/libhpi.so...
(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/native_threads/libhpi.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libjdwp.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libjdwp.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libnpt.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libnpt.so
Reading symbols from /lib/libnss_files.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libzip.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libzip.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libdt_socket.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libdt_socket.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libnet.so...
(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libnet.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libmanagement.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libmanagement.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libnio.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libnio.so
Reading symbols from /lib/libnss_dns.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /psecho/product/ias10g/lib/libjsmile.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/lib/libjsmile.so
Reading symbols from /usr/lib/libstdc++.so.6...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/libawt.so...(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/libawt.so
Reading symbols from /psecho/product/ias10g/jdk/jre/lib/i386/headless/libmawt.so...
(no debugging symbols found)...done.
Loaded symbols for /psecho/product/ias10g/jdk/jre/lib/i386/headless/libmawt.so
#0 0xf6fe87a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
and
(gdb) where
#0 0xf6fe87a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0xf6eb2825 in raise () from /lib/tls/libc.so.6
#2 0xf6eb4289 in abort () from /lib/tls/libc.so.6
#3 0xf6ee6cda in __libc_message () from /lib/tls/libc.so.6
#4 0xf6eed56f in _int_free () from /lib/tls/libc.so.6
#5 0xf6eed94a in free () from /lib/tls/libc.so.6
#6 0x5eca0aa1 in operator delete () from /usr/lib/libstdc++.so.6
#7 0x5eca0aed in operator delete[] () from /usr/lib/libstdc++.so.6
#8 0x5efa94a3 in DSL_stringArray::CleanUp () from /psecho/product/ias10g/lib/libjsmile.so
#9 0x5eedd091 in DSL_stringArray::~DSL_stringArray () from /psecho/product/ias10g/lib/libjsmile.so
#10 0x5ef7e5b8 in DSL_userProperties::~DSL_userProperties () from /psecho/product/ias10g/lib/libjsmile.so
#11 0x5ef94f03 in DSL_nodeInfo::~DSL_nodeInfo$delete () from /psecho/product/ias10g/lib/libjsmile.so
#12 0x5ef93b03 in DSL_node::CleanUp () from /psecho/product/ias10g/lib/libjsmile.so
#13 0x5ef950a5 in DSL_node::~DSL_node$delete () from /psecho/product/ias10g/lib/libjsmile.so
#14 0x5ef8bf99 in DSL_network::CleanUp () from /psecho/product/ias10g/lib/libjsmile.so
#15 0x5ef86b3d in DSL_network::~DSL_network$delete () from /psecho/product/ias10g/lib/libjsmile.so
#16 0x5ee9a91f in Java_smile_Network_deleteNative () from /psecho/product/ias10g/lib/libjsmile.so
#17 0xf2f2afe0 in ?? ()
#18 0x08100110 in ?? ()
#19 0x6211fb24 in ?? ()
#20 0x09d13180 in ?? ()
#21 0x00000000 in ?? ()
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Mohsin Beg wrote:The diagnetwork objects are not shared across threads. Each thread has it's own object.
Core dump suggests (but doesn't prove :) ) the object lifetime issue with non-diag network object. Do you have single Smile.Network shared between threads? Also, is network structure modified by the program after you create/load it from file?

I'll try to write small test app to check if I can reproduce the issue with .
shooltz[BayesFusion]
Site Admin
Posts: 1461
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Mohsin Beg wrote:The diagnetwork objects are not shared across threads. Each thread has it's own object.
I can't reproduce the issue on my machine. The call stack at crash contains the frames which can only be executed when Smile.Network object is finalized; this could only happen if all DiagNetwork object based on this network are not accessible.

Can you give me some more info on lifetimes of your Smile.Network and Smile.DiagNetwork objects?
Mohsin Beg
Posts: 11
Joined: Tue Feb 12, 2008 2:46 am

Post by Mohsin Beg »

We can reproduce the problem at will and it is due to user properties on fault nodes.

Basically if the user properties are very long (several KB) there is a problem. We have had to remove all user properties on the model to see the behaviour. If it does not crash, then we have a workaround (not really acceptable, but liveable). I think the smile code is assuming some default length for those.
(gdb) x/x 0x5ca9a120
0x5ca9a120: 0x00000000

(gdb) x/36s 0x5ca9a120-512
0x5ca99f20: " top of the report:<br>\n<pre>\n Shared Pool Statistics
Begin End\n", ' ' <repeats 30 times>, "------ ------\n", ' ' <repeats
13 times>, "Memory Usage %: 33.14 34.44\n <b>% SQL with executions>1:
95"...
0x5ca99fe8: ".55 95.46</b>\n % Memory for SQL w/exec>1: 91.67
91.69\n</pre> \nIn the above data the % SQL with executions > 1 is 95.46%.
When this value is below 60%, you can set this observation to FOUND.<b"...
0x5ca9a0b0: "r>\n<br>\n</body>\n</html>\n`\017"
0x5ca9a0cb: ""
0x5ca9a0cc: "\034"
0x5ca9a0ce: ""
0x5ca9a0cf: ""
0x5ca9a0d0: "\200\031(]tered_DB"
0x5ca9a0dd: "ation"
0x5ca9a0e3: ""
0x5ca9a0e4: "\021"
0x5ca9a0e6: ""
0x5ca9a0e7: ""
0x5ca9a0e8: "à÷Á\\8"
0x5ca9a0ee: "¨\\\210\017"
0x5ca9a0f3: ""
0x5ca9a0f4: "\034"
0x5ca9a0f6: ""
0x5ca9a0f7: ""
0x5ca9a0f8: ""
0x5ca9a0f9: "2¦]"
0x5ca9a0fd: "æÁ\\/"
0x5ca9a102: "ar"
0x5ca9a105: "ry"
0x5ca9a108: "°/"
0x5ca9a10b: ""
0x5ca9a10c: "\024"
0x5ca9a10e: ""
0x5ca9a10f: ""
0x5ca9a110: "HÅ\207ó\001"
0x5ca9a116: ""
0x5ca9a117: ""
0x5ca9a118: "\200\026;]\001"
0x5ca9a11e: ""
0x5ca9a11f: ""
0x5ca9a120: ""
.... 0 after this one
and
x/2x 0x5ca9a118 shows 0x5d3b1680 0x00000001
Post Reply