Check for memory leaks

How to troubleshoot server performance problems that are caused by memory leaks.

Introduction

If the execution time of your server application significantly increases, or if the operating system performance seems to decrease significantly, this can indicate a memory leak. This means that virtual memory is being allocated but is not returned when it is no longer needed. Eventually, the application or the system runs out of memory, and the application terminates abnormally.

One common indication of a memory leak is the java.lang.OutOfMemoryError. This error is thrown when there is insufficient memory to allocate an object in the Java heap or in a particular area of the heap. The garbage collector cannot make any further memory available to store a new object, and the heap cannot be expanded further.

When the java.lang.OutOfMemoryError error is thrown, and a stack trace is also written in the server log file.

An early step to diagnose an OutOfMemoryError is to determine what actually causes the error:

• Does it mean that the Java heap is full?
• Does it mean that the native heap is full?

The following sections explain some of the possible error messages, with reference to the detail part of the message.

• Exception in thread "main": java.lang.OutOfMemoryError: Java heap space
• Exception in thread "main": java.lang.OutOfMemoryError: PermGen space
• Exception in thread "main": java.lang.OutOfMemoryError: Requested array size exceeds VM limit
• Exception in thread "main": java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space?
• Exception in thread "main": java.lang.OutOfMemoryError: <reason><stack trace> (Native method)

censhare Server behaviour

Within the censhare Application Server's default release, the JVM command-line option -XX:+HeapDumpOnOutOfMemoryError that is set in the launcher.xml automatically creates a heap dump at ~/css/*.hprof if an OutOfMemoryError is thrown.

Ruleset for Heapdump analysis

Before you request a heapdump analysis, check the following rules:

1. The system is not EOL/EOS.

2. For test systems, if a restart does not solve the problem, we only analyze the second Out-of-Memory.

3. Check if there was a heap dump analysis task for the system in the past 3 months and if the recommendations were implemented.

Transmit necessary analysis data

If there is no configuration issue regarding too small Java heap space or in a particular area of the heap, we need to further analyze this problem.

Please transmit the following information:

1. The corresponding censhare server log files.

2. The ~/css/*.hprof file and include the timestamp of the file creation in the ticket.

3. Check if there are other OutOfMemory messages in the server logs with the following command: 

   CODE

   grep -ir outofmemory work/logs/server-0.*.log

4. Compress  the logs with the following command: 

   CODE

   tar -czf name-of-target.tar.gz source.hprof

   Before compressing, ensure that enough disk space is available!

Information for censhare developers in Germany, Ukraine and India:

To transmit or share the compressed file, see the options and instructions in Upload heapdump files to dev-analyze system (partner login required).

Incident Management

Until the real cause is analyzed, restart the censhare Application Server.

Problem Management

The transmitted heap dump must be analyzed with a profiler like YourKit. Usually, this is done by 3rd Level Support.