Detecting memory leaks in Linux

So after a couple of weeks you notice that your beefed-up Linux box is running dry on memory. Then you restart the box and the same thing happens again after another couple of weeks. If this sort of thing happened to you, then your system is suffering from a potential memory leak.

In a previous article we discussed how to find out the exact memory usage of a Linux process using 'pmap'. Therefore if we use pmap across a long enough time period, we can find out whether the memory usage of the process is increasing, indicating a memory leak. The figure below indicates a typical memory leak:
For a single process this is very easy to accomplish with pmap - the challenge starts when we have no idea which process is leaking. Therefore we need a script to scan all processes and gathering all their memory usage over time. As such I devised the following script which accomplishes this task:

# Author: James Attard []
# Date: 17/01/2012
# This script will check the real memory each process is consuming to determine any memory leaks.
# Get list of processes
ps -ef | grep "myprocess" | grep -v grep | awk '{print $2 ", " $8}' > /tmp/.oraps
numlines=`wc -l < /tmp/.pslist`
while read line ; do
done < /tmp/.oraps
if [ ! -e /tmp/memory_usage.csv ];
        echo "Timestamp (Date Hour), PID, Process Name, Real Memory Used (KB)"
for i in `seq 0 $(($numlines - 1))`
        pid=`echo ${processline[$i]} | awk '{print $1}'`
        echo `date +"%D %H"`, ${processline[$i]}, `pmap -d $pid | grep mapped | awk '{print $4}'` | sed s/.$//  >> /tmp/memory_usage.csv
done < /tmp/.oraps

The script is pretty straightforward - it will produce a list of all processes (in the above example, we are listing all processes which have string 'myprocess'). For each process, the memory usage is computed and saved in a csv file. If this script is put in an hourly cronjob, we will end up with a list of processes's memory usage over hourly interval periods.

Finally load the csv in a pivot table and produce a graph. These steps can be highlighted in the screenshots below:
This is the csv file produced by the script. Column 1 shows the timestamp of the sample, column 2 is the process ID, column 3 is the process name and last column shows the real memory usage of the process.
Load the csv in your favorite spreadsheet software and produce a Pivot Chart.
Column 1 must be the x-axis, column 2 the Series, and last column should be the y-axis.

The generated graph will have all the processes memory usage superimposed on each other. The process which is consuming an ever increasing memory over time might be leaking memory. In this case process 1184 is the culprit!

1 comment:

Anonymous said...

# Get a list of process and set
correcly the process file

ps -ef | grep "MII" | grep -v grep | awk '{print $2 ", " $8}' > /tmp/.pslist

# Echo table header in output file

echo "Timestamp (Date Hour), PID, Process Name, Real Memory Used (KB)" > /tmp/memory_usage.csv

#There been a comma at the end of the pid, filtered out with sed. Works perfect!

pid=`echo ${processline[$i]} | awk '{print $1}' | sed -e 's/[^0-9]//'`
Work very well