My Technical Works
A crash can cause the JVM to terminate. A Crash or even a fatal error can cause a Crash. A bug in the API, a bug in third party library that we use or bug in a native api can cause these JVM crashes. JVM can crash due to the external factors like resource exhaustion etc.
In this article we will see how we can analyze a JVM crash. Whenever a JVM is crashed a fatal error log file is generated. The file is named as hs_err_pid.log (where is the process id of the process). Most cases this file is created in the same working directory and in some other cases the file may be created in the temporary location. The file is created in the /tmp location in Linux and in directory specified by TEMP environment variable in windows.
The header in the log file tells us about the problem. This is printed to standard output. An example log file looks as
The first line tells that the JVM crashed due to an Unexpected signal. The second line gives some more details like,
EXCEPTION_ACCESS_VIOLATION: This was crashed with an Access violation exception and this is the signal name.
This is represented as a Library name and offset. It’s possible to inspect the instructions that caused the crash without a debugger or core file. Just use a disassembler to dump instructions near the offset.
The fatal error log will also give us the Thread details. The thread section in the log file gives us information about the thread that crashed. There may be cases where multiple threads crash, at this point only one thread is printed.
Current thread (0x0d0eb000): The Thread pointer is a pointer to the internal representation to the JVM internal thread structure.
This indicates that the exception code is 0xc0000005 (ACCESS_VIOLATION), and it occurred when the thread attempted to read address 0x00000000.
These register gives us the context of the register at the time of the fatal error and these will change depending on the underlying architecture.
After the register values, you will see the top of stack and the instructions/opcodes near the crashing pc. The error log prints 32 bytes which can be decoded using a disassembler to see the instructions around the location of the crash. These can be seen as
The next line of information will be the thread stack trace. This includes the addresses of the base and the top of the stack, the current stack pointer, and the amount of unused stack available to the thread. This is followed by the stack frames and up to 100 frames are printed. For C/C++ frames the library name may also be printed.
The Native Stack trace gives us the native thread information showing all function calls leaving the Java calls. The second stack is Java frames which prints the java frames including the in-lined methods. It skips the native frames. Sometimes according to the crash it will not print the native thread details but can print the java details.
Some of the fatal error log file will generate the Process Information too. The process section gives us information about the process which includes the thread details and memory usage. We can see the information as,
In the above line we can see, The thread list includes the threads that the VM is aware of. This includes all Java threads and some VM internal threads, but does not include any native threads created by the user application that have not attached to the VM
The next line gives us the VM state. This normally tells us the state of the VM. Some normal VM states include
2. at safepoint – indicates that all threads are blocked in the VM waiting for a special vm operation to complete
3. synchronizing – indicates that a special vm operation is required and the VM is waiting for all threads in the vm to block.
The next line gives us the Mutex/Monitor Information. These are not java Object locks but JVM internal locks. For each lock it prints the name of the lock, its owner, and the addresses of a VM internal mutex structure and its OS lock. In this case the lock is owned by None.
After the heap details, the next section tells us about the memory regions at the time of crash. A Memory map is also generated which is use-full when debugging some crashes, as it can tell you what libraries are actually being used, their location in memory, as well as the location of heap, stack and guard pages. This memory map section will be different depending on the Operating Systems. On Linux the process memory map (/proc/
r-xp: Permission on the region , we have Read,Write,eXecute,Private and Share.
The Java heap details can be obtained with the map listed as heap. This gives us the the actual memory regions reserved for heap with attributes rwxp .
The last section is about the System which gives us the information about the system statistics when the crash occurred like
CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 13 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, lzcnt, tsc
flags : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up unfair_spinlock pni cx16 hypervisor lahf_lm abm sse4a
The output is operating system specific but in general includes the operating system version, CPU information, and summary information about the memory configuration.
On linux there are 3 types of threading libraries which areinuxthreads (fixed stack), linuxthreads (floating stack) and NPTL. In our case it’s the NTPL. Knowing this type is use-full when we see crashes that come due to the threading library.
The ulimit gives us the information about the limits that are set for various elements. In the above line
CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 13 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, lzcnt, tsc
The CPU information is also provided with the fatal error log and last the memory information is provided as
If both physical memory and swap space are almost full, there’s good reason to suspect the crash was due to running out of memory.
vm_info: Java HotSpot(TM) Client VM (11.0-b16) for windows-x86 JRE (1.6.0_11-b03), built on Nov 10 2008 02:15:12 by «java_re» with MS VC++ 7.1
The Final Section is the Vm_info.This is a version string embedded in libjvm.so/jvm.dll. Every JVM has its own unique vm_info string. When you are in doubt about whether the hs_err*.log was generated by a particular JVM, check the version string.
20 comments :
Very informative, thanks. Please post some details on how to deduce the garbage heap collection details. Reply Delete
Hi! I could have sworn I’ve visited this site before but after browsing through many
of the posts I realized it’s new to me. Nonetheless,
I’m definitely happy I came across it and I’ll be bookmarking
it and checking back often! Reply Delete
Hey I know this is off topic but I was wondering if you knew of any widgets I could
add to my blog that automatically tweet my newest
twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping
maybe you would have some experience with something like this.
Please let me know if you run into anything. I truly enjoy reading your blog and I look
forward to your new updates. Reply Delete
A dua
hung
chuôm
hàng đầu
Bạc
đình công
Bàn bạc
tuổi teen
bao bọc
phải
phong cách
mách nhỏ
ngày nay
mẫu mã
quanh đó
không tính
bí quyết
xem nhẹ
xếp đặt
bởi thế
bớt xén
bự chảng
bước ngoặt
kinh doanh
quý khách
các
cái
cơ bản
nguồn gốc
cần thiết
cẩn thận
đớp
khó khăn
gắt
chắc chắn
loại vải
chế tạo
tầm giá
chi tiết
chính hãng
chọn
chống thấm
quan tâm
chú ý
đi lại
dùng cho
xây dựng thương hiệu
sở hữu
công nghệ
công ty
cộng
shop
vô cùng
cười
cứu giúp
rộng rãi
đông đảo
hầu hết
đặc biệt
lẩn thẩn
mê say
bộ dạng
đập phá
đau ốm
thứ 1
toàn bộ
tiết kiệm
đem lại
tới
trẻ trung
đi ra
thế mạnh
điều tra
đồ vật
đơn thuần
ngừng
tiêu dùng
dùng
êm dịu
kém chất lượng
ràng buộc
vừa rồi
gan dạ
quen thuộc
gần gũi
sắp
gặp
gấp rút
ví như
giải thích
hạn chế
hạn chế
kết hợp
hết lòng
tiên tiến
tuyệt vời
tương đối
thua sút
kém
rốt cục
hậu quả
khai mạc
quan niệm
tổng quát
khi không
lúc
làm
ko
khuyến mãi
khiếu nại
lang bạt
làng
điều khiển
rét mướt
chếnh choáng
lào xào
thuận tiện
thu hút
lớn
quái đản
miêu tả
sắm
bắt
tài năng
nặng nề
nấu nướng
té
sửng sốt
nhãn hàng
thời gian nhanh
nhốn nháo
sôi động
vượt trội
nức danh
oai nghiêm
ổn định
pha chế
thấp
phẳng phiu
vươn lên là
nguyên tắc
phù hợp
PR
qua chuyện
ưa chuộng
rách rưới
rậm rì
rẻ mạt
rõ rệt
bao la
tinh khiết
thông minh
sáng sủa
lịch sự
sâu sắc
hoảng
tái mét
tai vạ
tài xế
tật nguyền
tang chứng
tàu bay
thoải mái
lương y
thừa thãi
nối tiếp
Dù vậy
mờ mịt
đầm đìa
va vấp
cầu xin
vắng ngắt
lặt vặt
xa cách
lãng phí
xâm chiếm
xanh rờn
xấu xa
xin xỏ
hoàn thành
xu thế
dự định
ỷ lại
trang phục
y chang
yên lặng
yêu đương
yếu Reply Delete
We have JVM running on windows, can you please tell me if the «Swap» memory is the same as the paging file for windows? Reply Delete