Friday, June 17, 2011

Linux as seen by matplotlib - the view from 2000 metres

In my previous post I shared some python code that extracts Linux process-file-system data into python objects. In this post I will augment the code from the previous post with some new code that generates rolling plots of process file system data.

I've been playing around with generating detailed overviews of my running Linux system. Existing tools, such as top or htop, do a pretty good job of displaying the processes consuming the highest resources, but I'm looking for an at-a-glance overview of all processes and their impact on the system.   I've used this goal as a reason to learn a little more about the python plotting tool matplotlib.

The plot to the right is generated by a matplotlib python script producing something similar to the LibreOffice plot in my previous blog entry.  The process-file-system extraction code from the previous blog posting is used to extract CPU, vsize, and RSS data.  Unlike the previous example, the script refreshes the plot every few seconds.   This new plot uses a hash to pick a colour for each user and also annotates each point with the PID, command, and username.

Plotting every process results in a very cluttered graph  To reduce the clutter I have restricted to plot data to the "interesting" processes - those that have changing values.  Interest decays with inactivity, so processes that become static eventually disappear from the graph.

I wasn't happy with the bubble plot because clutter made it hard to get an overview of all process activity, which was the whole point of the exercise.  Time for a different tack.

The plot in the screen-capture on the right is an attempt at an at-a-glance view of every process and thread  This new script periodically extracts stat, status, and IO data from the Linux process file-system.  The script calculates changes in the data between refresh cycles and plots indications of the changes. There is a point for each process and thread.  Points are grouped by user and ordered by process start time.  The size of each point indicates relative resident set size.  The colours of each point represent the process's state during the cycle:

  1. Light grey: no activity
  2. Red: consumed CPU
  3. Green: perfomed IO
  4. Orange: Consumed CPU and performed IO
  5. Yellow: exited.

Changes in RSS are exaggerated so that even small changes in RSS are visible as a little "throb" in the point size.  The plot updates once every three seconds or when ever the space bar is pressed.  By grouping by username and start time, the placement of points remains fairly static and predictable from cycle to cycle.

In matplotlib it is relatively easy to trap hover and click events.  I've included code to trap hover and display a popup details for the process under the mouse.   In  the screen-capture above, I've hovered over the Xorg X-Windows server.  I also trap right-click and pin the popup so that will remain updating even after the mouse is moved away.

The the number of memory hogs visible on the plot is a bit exaggerated because the plot includes threads as well as processes (threads share memory).  This plot is somewhat similar to the Hinton Diagram seen in the matplotlib example pages, which may also be an interesting basis for viewing a collection of processes or servers.  (My own efforts pre-dates my discovering the Hinton example, so the implementation details differ substantially.)


Finally, I thought I'd include a 3D plot. I'm not sure it's that useful, but adding an extra dimension is food for thought.  The script plots cumulative-CPU and RSS over the duration of the scripts run time.  It uses a heuristic to select "interesting" processes.  In this particular screen capture I shut down my eclipse+android IDE - first set of terminated lines - and then restarted the same IDE (steep climbing lines).  The CPU and memory consumed by the plotting script can be seen as the diagonal line across the bottom time-CPU axis.


Matplotlib is a really useful plotting library, when ever I thought I'd exhausted what was possible, a Google-search would usually prove me wrong.  

The code


plot_bubble_cpu_vsize_rss.py


#!/usr/bin/env python
#
# Bubble Plot: cpu (x), vsize(y), rss(z) 
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import matplotlib
matplotlib.use('GTkAgg')
import pylab
import gobject
import sys
import os
import math
from optparse import OptionParser
import linuxprocfs as procfs

SC_CLK_TCK=float(os.sysconf(os.sysconf_names['SC_CLK_TCK']))
ZOMBIE_LIVES=2 # Not a true UNIX zombie, just a plot zombie.
INTERESTING_DEFAULT=3 # Don't display a process if it is boring this many times in a row.
INTERVAL = 2

_user_colors = {}

def _user_color(username):
    if username in _user_colors:
        color = _user_colors[username]
    else: 
        # For a username, generate a psuedo random (r,g,b) values in the 0..1.0 range 
        # E.g. for root base this on hash(root), hash(oot), and hash(ot) - randomise
        # each hash further - seems to work better than the straight hash 
        color = tuple([math.fabs(hash(username[i:]) * 16807 % 2147483647)/(2147483647 * 1.3) for i in (0, 2, 1)])
        #color = tuple([abs(hash(username[i:]))/(sys.maxint*1.3) for i in (2, 1, 0)])
        _user_colors[username] = color
        print username,  color
    return color

class TrackingData(object):

    def __init__(self, info):
        self.previous = info
        self.current = info
        self.color = _user_color(info.username)
        self.text = '%d %s %s' % (info.pid, info.username, info.stat.comm )
        self.interesting = 0
        self.x = 0
        self.y = info.stat.vsize 
        self.z = info.stat.rss / 10
        self.zombie_lives = ZOMBIE_LIVES

    def update(self, info):
        self.current = info
        oldx = self.x
        oldy = self.y
        oldz = self.z
        self.x = ((info.stat.stime + info.stat.utime) - (self.previous.stat.stime + self.previous.stat.utime)) /SC_CLK_TCK
        self.y = info.stat.vsize 
        self.z = info.stat.rss / 10
        if self.x == oldx and self.y == oldy and self.z == oldz: 
            if self.interesting > 0:
                self.interesting -= 1  # if interesting drops to zero, stop plotting this process
        else:
            self.interesting = INTERESTING_DEFAULT
        self.previous = self.current
        self.zombie_lives = ZOMBIE_LIVES
        
    def is_alive(self):
        self.zombie_lives -= 1
        return self.zombie_lives > 0


class PlotBubbles(object):

    def __init__(self, sleep=INTERVAL, include_threads=False):
        self.subplot = None
        self.datums = {}
        self.sleep = sleep * 1000
        self.include_threads = include_threads

    def make_graph(self):
        all_procs = procfs.get_all_proc_data(self.include_threads)
        y_list = []
        x_list = []
        s_list = []
        color_list = []
        anotations = []
 
        for proc_info in all_procs:
            if not proc_info.pid in self.datums:
                data = TrackingData(proc_info)
                self.datums[proc_info.pid] = data
            else:
                data = self.datums[proc_info.pid]
                data.update(proc_info)
            if data.interesting > 0: # Only plot active processes
                x_list.append(data.x)
                y_list.append(data.y)
                s_list.append(data.z)
                color_list.append(data.color)
                anotations.append((data.x, data.y, data.color, data.text))
            if not data.is_alive():
                del self.datums[proc_info.pid]
        if self.subplot == None:
            figure = pylab.figure()
            self.subplot = figure.add_subplot(111)
        else:
            self.subplot.cla()
        if len(x_list) == 0: # Nothing to plot - probably initial cycle
            return True
        self.subplot.scatter(x_list, y_list, s=s_list, c=color_list, marker='o', alpha=0.5)
        pylab.xlabel(r'Change in CPU (stime+utime)', fontsize=20)
        pylab.ylabel(r'Total vsize', fontsize=20)
        pylab.title(r'Process CPU, vsize, and RSS')
        pylab.grid(True)
        gca = pylab.gca()
        for x, y, color, text in anotations:
            gca.text(x, y, text, alpha=1, ha='left',va='bottom',fontsize=8, rotation=33, color=color)
        pylab.draw()
        return True

    def start(self):
        self.scatter = None
        self.make_graph()
        pylab.connect('key_press_event', self)
        gobject.timeout_add(self.sleep, self.make_graph)
        pylab.show()

    def __call__(self, event):
      self.make_graph()

if __name__ == '__main__':
    usage = """usage: %prog [options|-h]
    Plot change in CPU(x), total vsize(y) and total RSS.
    """
    parser = OptionParser(usage)
    parser.add_option("-t", "--threads", action="store_true", dest="include_threads", help="Include threads as well as processes.")
    parser.add_option("-s", "--sleep", type="int", dest="interval", default=INTERVAL, help="Sleep seconds for each repetition.")

    (options, args) = parser.parse_args()

    PlotBubbles(options.interval, options.include_threads).start()


% python plot_bubble_cpu_vsize_rss.py -h
Usage: plot_bubble_cpu_vsize_rss.py [options|-h]
    Plot change in CPU(x), total vsize(y) and total RSS.

Options:
  -h, --help            show this help message and exit
  -t, --threads         Include threads as well as processes.
  -s INTERVAL, --sleep=INTERVAL
                        Sleep seconds for each repetition.

plot_grid.py


#!/usr/bin/env python
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import matplotlib
matplotlib.use('GTkAgg')
import pylab
import time
import gobject
import math
import os
import linuxprocfs as procfs
from optparse import OptionParser

SC_CLK_TCK=float(os.sysconf(os.sysconf_names['SC_CLK_TCK']))
LIMIT=1000
INTERVAL=3
DEFAULT_COLS=30
BASE_POINT_SIZE=300
MIN_SIZE=20
ZOMBIE_LIVES=3 # Not a true UNIX zombie, just a plot zombie
MAX_RSS = 10
# Don't spell colour two ways - conform with pylab
DEFAULT_COLORS=['honeydew','red','lawngreen','orange','yellow','lightyellow','white']

class ProcessInfo(object):
    def __init__(self, new_procdata):
        self.data = new_procdata
        self.previous = None
        self.alive = True
        self.info = '%d %s %s' % (new_procdata.pid, new_procdata.username, new_procdata.stat.comm )
        self.username = new_procdata.username
        self.color = 'blue'
        self.zombie_lives = ZOMBIE_LIVES
    
    def update(self, new_procdata):
        self.previous = self.data
        self.data = new_procdata
        self.zombie_lives = ZOMBIE_LIVES
        
    def report(self):
        return '%s\nstate=%s\nutime=%f\nstime=%f\nrss=%d\nreads=%d\nwrites=%d' % \
             (self.info, 
              self.data.status.state if not self.is_zombie() else 'exited', 
              self.data.stat.utime/SC_CLK_TCK, 
              self.data.stat.stime/SC_CLK_TCK,
              self.data.stat.rss,
              self.data.io.read_bytes,
              self.data.io.write_bytes)
        
    def is_alive(self):
        self.zombie_lives -= 1
        return self.zombie_lives > 0
    
    def is_zombie(self):
        return self.zombie_lives < ZOMBIE_LIVES - 1
    
    def sort_key(self):
        return (self.username, self.data.stat.starttime, self.data.pid)

class Activity_Diagram(object):

    def __init__(self, sleep=INTERVAL, max_cols=DEFAULT_COLS, point_size=BASE_POINT_SIZE, include_threads=True, colors=''):
        self.process_info = {}
        self.pos_index = {}
        self.subplot = self.label = self.hover_tip = self.hover_tip_data = None
        self.hover_tip_sticky = False
        self.include_threads = include_threads
        self.sleep = sleep
        self.max_cols = max_cols
        self.point_size = point_size
        # Extend colors to same length as default, merge together colors and default, choose non blank values
        self.normal_color, self.cpu_color, self.io_color, self.cpuio_color, self.exit_color, self.tip_color, self.bg_color = \
            [ (c if c != '' and c != None else d) for c,d in map(None, colors, DEFAULT_COLORS)]

    def start(self):
        self.start_time = time.time()
        self._create_new_plot()
        pylab.connect('motion_notify_event', self)
        pylab.connect('button_press_event', self)
        pylab.connect('key_press_event', self)
        gobject.timeout_add(self.sleep * 1000, self._create_new_plot)
        pylab.show()

    def _create_new_plot(self):
        if self.subplot == None:
            figure = pylab.figure()
            figure.set_frameon(True)
            self.subplot = figure.add_subplot(111, axis_bgcolor=self.bg_color)
        else:
            self.subplot.cla()    
        plot = self.subplot
        x_vals, y_vals, c_vals, s_vals, namelabels = self._retrieve_data()
        plot.scatter(x_vals, y_vals, c=c_vals, s=s_vals) #, label=data.info)
        plot.axis('equal')
        plot.set_xticks([])
        plot.set_yticks(namelabels[0])
        plot.set_yticklabels(namelabels[1], stretch='condensed')
        self._create_tip(plot) # Refresh the tip the user is currently looking at
        pylab.title('Activity: size=RSS changes; red=CPU; green=IO; orange=CPU and IO.')
        pylab.draw()
        #print "ping"
        return True
    
    def _retrieve_data(self):
        # Update from procfs
        for proc in procfs.get_all_proc_data(include_threads=self.include_threads):
            if not proc.pid in self.process_info:
                data = ProcessInfo(proc)
                self.process_info[proc.pid] = data
            else:
                data = self.process_info[proc.pid]
                data.update(proc)
        max_rss = 0
        for info in self.process_info.values():
            if max_rss < info.data.stat.rss: 
                max_rss = info.data.stat.rss
        # Compute values for plotting
        x_vals = []; y_vals = []; c_vals = []; s_vals = []
        usernames = [[],[]]
        col = 0; row = 0
        self.pos_index = {}
        previous = None
        for info in sorted(self.process_info.values(), key=lambda info: info.sort_key()):
            if not info.is_alive():
                del self.process_info[info.data.pid]
            else:
                if (not previous or previous.username != info.username):
                    if col != 0: 
                        col = 0; row += 1
                    print info.username, -row
                    usernames[0].append(-row)
                    usernames[1].append(info.username)
                self.pos_index[(col,-row)] = info
                x_vals.append(col)
                y_vals.append(-row) # Invert ordering
                c_vals.append(self._decide_color(info))
                s_vals.append(self._decide_size(info,  max_rss))
                col += 1
                if col == self.max_cols:
                    col = 0; row += 1
                previous = info
        return (x_vals, y_vals, c_vals, s_vals, usernames)
                
    def _decide_color(self,info): 
        if info.is_zombie():
            return self.exit_color
        if info.previous == None:
            delta_cpu = delta_io = 0     
        else:
            delta_cpu = (info.data.stat.utime + info.data.stat.stime) - (info.previous.stat.utime + info.previous.stat.stime)
            delta_io = (info.data.io.read_bytes + info.data.io.write_bytes) - (info.previous.io.read_bytes + info.previous.io.write_bytes)
        color = self.normal_color
        if delta_io > 0:
            color = self.io_color
        if delta_cpu > 0 or info.data.stat.state == 'R':
            if delta_io > 0:
                color = self.cpuio_color
            else:
                color = self.cpu_color
        return color
 
    def _decide_size(self, info,  max_rss):    
        rss = info.data.stat.rss
        delta_rss = rss - (info.previous.stat.rss if info.previous else rss)
        # A relative proportion of the base dot size
        size = max(MIN_SIZE, self.point_size * rss / max_rss)
        if delta_rss > 0: # Temporary throb to indicate change
            size += max(20,size/4) 
        elif delta_rss < 0:
            size -= max(20,size/4)
        return size

    def _create_tip(self, axes, x=None, y=None, data=None, toggle_sticky=False):
        if data:
            if not self.hover_tip_sticky or toggle_sticky:
                if self.hover_tip: self.hover_tip.set_visible(False)
                self.hover_tip = axes.text(x, y, data.report(), bbox=dict(facecolor=self.tip_color, alpha=0.85), zorder=999)
                self.hover_tip_data = (x,y,data)
            if toggle_sticky: self.hover_tip_sticky = not self.hover_tip_sticky
        elif self.hover_tip_data:
            x,y,data = self.hover_tip_data
            self.hover_tip = axes.text(x, y, data.report(), bbox=dict(facecolor=self.tip_color, alpha=0.85), zorder=999)

    def _clear_tip(self):
        if self.hover_tip and not self.hover_tip_sticky:
            self.hover_tip.set_visible(False) # will be free'ed up by next plot draw 
            self.hover_tip = self.hover_tip_data = None       

    def __call__(self, event):
        #print event.name
        if event.name == 'key_press_event':
            self._create_new_plot()
        else: 
            if (event.name == 'motion_notify_event' or event.name == 'button_press_event') and event.inaxes:
                point = (int(event.xdata + 0.5), int(event.ydata - 0.5))
                if point in self.pos_index:  # On button click let tip stay open without hover
                    self._create_tip(event.inaxes, event.xdata, event.ydata, self.pos_index[point], event.name == 'button_press_event')
            else:
                self._clear_tip() 
            pylab.draw()

if __name__ == '__main__':
    
    usage = """usage: %prog [options|-h]
    Plot RSS, CPU and IO, with hover and click for details.
    RSS size is plotted as a circle - the circle will temporarily
    jump up and down in size to indicate a growing or shrinking 
    RSS - the steady state size is a relative size indicator.
    """
    parser = OptionParser(usage)
    parser.add_option("-p", "--no-threads", action="store_true", dest="no_threads", help="Exclude threads, only show processes.")    
    parser.add_option("-s", "--sleep", type="int", dest="interval", default=INTERVAL, help="Sleep seconds for each repetition.")
    parser.add_option("-n", "--columns", type="int", dest="columns", default=DEFAULT_COLS, help="Number of columns in each row (maximum).")
    parser.add_option("-d", "--point-size", type="int", dest="point_size", default=BASE_POINT_SIZE, help="Dot point size (expressed as square area).")
    parser.add_option("-c", "--colors", type="string", dest="colors", default='', 
                      help="Colors for normal,cpu,io,cpuio,exited,tip,bg comma separated. " + 
                      "Only supply the ones you want to change e.g. -cwhite,,blue - " + 
                      " Defaults are " + ','.join(DEFAULT_COLORS))

    (options, args) = parser.parse_args()
    Activity_Diagram(options.interval, options.columns, options.point_size, not options.no_threads, options.colors.split(',')).start()


% python plot_grid.py -h
Usage: plot_grid.py [options|-h]
    Plot RSS, CPU and IO, with hover and click for details.
    RSS size is plotted as a circle - the circle will temporarily
    jump up and down in size to indicate a growing or shrinking 
    RSS - the steady state size is a relative size indicator.

Options:
  -h, --help            show this help message and exit
  -p, --no-threads      Exclude threads, only show processes.
  -s INTERVAL, --sleep=INTERVAL
                        Sleep seconds for each repetition.
  -n COLUMNS, --columns=COLUMNS
                        Number of columns in each row (maximum).
  -d POINT_SIZE, --point-size=POINT_SIZE
                        Dot point size (expressed as square area).
  -c COLORS, --colors=COLORS
                        Colors for normal,cpu,io,cpuio,exited,tip,bg comma
                        separated. Only supply the ones you want to change
                        e.g. -cwhite,,blue -  Defaults are
                        honeydew,red,lawngreen,orange,yellow,lightyellow,white

plot_cpu_vsize_time_line_3d.py


#!/usr/bin/env python
#
# Plot a limited time line in 3D for CPU and RSS
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import matplotlib
matplotlib.use('GTkAgg')
import pylab
import time
import gobject
import gtk
import random
import string
import math
import mpl_toolkits.mplot3d.axes3d as axes3d
from optparse import OptionParser

import linuxprocfs as procfs


HZ=1000.0
LIMIT=1000
INTERVAL=5
ZOMBIE_LIVES=5
MAX_INTERESTING=5.0

class ProcessInfo(object):

    def __init__(self, new_procdata):
        self.base = new_procdata
        self.previous = new_procdata
        self.data = new_procdata
        self.text = '%d %s %s' % (new_procdata.pid, new_procdata.username, new_procdata.stat.comm )
        self.color = '#%6.6x' % (((self.base.pid * 16807 % 2147483647)/(2147483647 * 1.3)) % 2**24 )
        self.color = tuple([(self.base.pid * i * 16807 % 2147483647)/(2147483647 * 1.3) for i in (7, 13, 23)])
        self.xvals = []
        self.yvals = []
        self.zvals = []
        self.zombie_lives = ZOMBIE_LIVES
        self.interesting = 0
        self.visible = False

    def update(self, new_procdata):
        self.previous = self.data
        self.data = new_procdata
        self.zombie_lives = 5
        x = self.data.time_stamp
        y = ((self.data.stat.utime - self.base.stat.utime) + (self.data.stat.stime - self.base.stat.stime)) / HZ
        z = float(new_procdata.stat.rss)
        if len(self.xvals) > LIMIT:
            del self.xvals[0]
            del self.yvals[0]
            del self.zvals[0]
        self.xvals.append(x)
        self.yvals.append(y)
        self.zvals.append(z)
        if self.data.stat.utime - self.previous.stat.utime > 10.0 or self.data.stat.stime - self.previous.stat.stime > 10.0 or abs(self.data.stat.rss - self.previous.stat.rss) > 15000:
            self.visible = True
            self.interesting = MAX_INTERESTING

class Activity_Diagram(object):

    def __init__(self, remove_dead=True):
        self.process_info = {}
        self.start_time = None
        self.subplot = None
        self.remove_dead = remove_dead

    def make_graph(self):
        for proc in procfs.get_all_proc_data(include_threads=True):
            if not proc.pid in self.process_info:
                data = ProcessInfo(proc)
                self.process_info[proc.pid] = data
            else:
                data = self.process_info[proc.pid]
                data.update(proc)
        if self.subplot == None:
            figure = pylab.figure()
            self.subplot = figure.add_subplot(111, projection='3d')
        else:
            self.subplot.cla()
        for pid, data in self.process_info.items():
            if data.zombie_lives <= 0 and self.remove_dead:
                del self.process_info[pid]  # dead for a while now - remove
            data.zombie_lives -= 1
            if data.visible:
                if len(data.xvals) > 0: 
                    alpha =data.interesting / MAX_INTERESTING  # boring processes fade away
                    self.subplot.plot(data.xvals, data.yvals, data.zvals, color=data.color, alpha=alpha, linewidth=2)
                    self.subplot.text(data.xvals[-1], data.yvals[-1], data.zvals[-1], data.text, alpha=alpha, fontsize=6, color = data.color if data.zombie_lives >= ZOMBIE_LIVES - 1 else 'r')
            data.interesting -= 1 if data.interesting > 2 else 0 # Losing interest
        self.subplot.set_xlabel(r'time', fontsize=10)
        self.subplot.set_ylabel(r'cpu', fontsize=10)
        self.subplot.set_zlabel(r'rss', fontsize=10)
        pylab.title('Cumulative CPU and RSS over %f minutes' % ((time.time() - self.start_time) /60.0))
        pylab.draw()
        return True

    def start(self,  sleep_secs=INTERVAL):
        self.start_time = time.time()
        self.make_graph()
        pylab.connect('key_press_event', self)
        gobject.timeout_add(INTERVAL*1000, self.make_graph)
        pylab.show()

    def __call__(self, event):
      self.make_graph()

def boo():
    print 'boo'

if __name__ == '__main__':
    usage = """usage: %prog [options|-h]
    Plot cumulative-CPU and RSS over the script run time.
    """
    parser = OptionParser(usage)
    parser.add_option("-s", "--sleep", type="int", dest="sleep_secs", default=INTERVAL, help="Sleep seconds for each repetition.")
    (options, args) = parser.parse_args()
    Activity_Diagram(remove_dead=False).start(sleep_secs=options.sleep_secs)

% python plot_cpu_rss_time_lines_3d.py -h
Usage: plot_cpu_rss_time_lines_3d.py [options|-h]
    Plot cumulative-CPU and RSS over the script run time.
    

Options:
  -h, --help            show this help message and exit
  -s SLEEP_SECS, --sleep=SLEEP_SECS
                        Sleep seconds for each repetition.

Notes


Even if you don't know any python, you can still have a play by running these scripts from the command line. Just save them into a folder giving them the appropriate file-names and also save the linuxprocfs.py from my previous post. Make sure you've installed python-matplotlib for you distribution of Linux (it's a standard offering for openSUSE, so it's in the repo).

Consult the notes from the previous blog entry for some notes on the Linux process file-system.

The matplotlib Web-site ( http://matplotlib.sourceforge.net/ ) contains plenty of documentation and examples, plus Google will track down heaps more advice and examples.

This blog page uses SyntaxHighlighter by Alex Gorbatchev.   You can easily copy and paste a line-number free version of  the code by selecting view-source icon in the mini-tool-bar that appears on the top right of the source listing (if javascript is enabled).


Friday, June 10, 2011

Linux proc stat, status, io to CSV via python


I've written a python script that extracts data from the Linux Process File-system into python objects. I've included an option to dump data to CSV. The CSV can be directly loaded into tools such LibreOffice Calc for analysis and plotting.  The screen-capture to the right shows a LibreOffice Calc plot of data for processes running on my desktop.  For each process the plot shows total-CPU-time (x-axis), allocated virtual memory size (y-axis), and resident set size (bubble area).

My python script can be run from the command line and includes a variety of command line options including --help, for example:

% python linuxprocfs.py -h
Usage: linuxprocfs.py [options] [pid...]
    Output CSV for procfs stat, status or io data for given thread/process pid's or
    for all processes and threads if no pid's are supplied.

Options:
  -h, --help            show this help message and exit
  -s, --stat            Output csv for pid/stat files.
  -S, --status          Output csv for pid/status files.
  -i, --io              Output csv for pid/io files.
  -t, --titles          Output a title line.
  -r, --repeat          Repeat until interrupted.
  -w WAIT, --sleep=WAIT
                        Sleep seconds for each repetition.
  -p, --processes       Show all processes, but not threads.


On my desktop, LibraOffice Calc is starting to struggle when plotting large amounts of data. It might be better to process and the data further and plot it using a dedicated plotting tool - which is what I will describe next time.

The code


#!/usr/bin/env python
#
# Copyright (C) 2011: Michael Hamilton
# The code is LGPL (GNU Lesser General Public License) ( http://www.gnu.org/copyleft/lesser.html )
#

from __future__ import with_statement
import re
import os
import glob
import string
import pwd
import csv
import sys
import time
from optparse import OptionParser

PROC_FS_ROOT = '/proc'
INT_RE_SPEC = '[+-]*\d+'
INT_RE = re.compile(INT_RE_SPEC + '$')
CSV_LINE_TERMINATOR='\n'


# Default parser that deals with a multi-line file where each line 
# is a "tag: value" pair 
class _ProcBase(object):

    _split_re = re.compile(':\s+')

    def __init__(self, path=None, filename=None):
        self.error = None
        if path and filename:
            self.parseProcFs(path, filename)

    def parseProcFs(self, path, filename):
        pid = os.path.basename(path)
        self.pid = int(pid) if INT_RE.match(pid) else pid
        try:
            with open(path + '/' + filename) as proc_file:
                for line in proc_file.read().splitlines():
                    sort_key, value = _ProcBase._split_re.split(line)
                    self.__dict__[string.lower(sort_key)] = int(value) if INT_RE.match(value) else value
        except IOError as ioerr:
            self.handle_error('IOError %s/%s - %s' % (path, filename, ioerr))

    def handle_error(self, message):
        self.error = message
        print >> sys.stderr, self.error

    def keys(self):
        return sorted(self.__dict__.keys())

    def csv(self, file, header=True):
        if not self.error:
            if header:
                csv.writer(file, lineterminator=CSV_LINE_TERMINATOR).writerow(self.keys())
            csv.DictWriter(sys.stdout,  self.keys(), lineterminator=CSV_LINE_TERMINATOR).writerow(self.__dict__)

# Parser for space separated Values on one line- e.g. "12 comm 123456 111 a 12"
class _SpaceSeparatedParser(object):

    def __init__(self):
        self._keys = []
        self._re_spec = ''
        self._regexp = None

    def _add_item(self, sort_key, rexp_str):
        self._regexp = None
        self._keys.append(sort_key)
        if rexp_str:
            self._re_spec += rexp_str % sort_key
        return self
    def int_item(self, sort_key):
        return self._add_item(sort_key, '(?P<%s>' + INT_RE_SPEC + ')\s')
    def comm_item(self, sort_key):
        return self._add_item(sort_key, '[(](?P<%s>[^)]+)[)]\s')
    def string_item(self, sort_key):
        return self._add_item(sort_key, '(?P<%s>\w+)\s')
    def nonparsed_item(self, sort_key):  # Create property sort_key only, but don't parse it
        return self._add_item(sort_key, None)

    def keys(self):
        return self._keys;
    def parse(self, line):
        if not self._regexp:
            self._regexp = re.compile(self._re_spec)
        return self._regexp.match(line)

class ProcStat(_ProcBase):

    _parser = _SpaceSeparatedParser().\
        int_item('pid').\
        comm_item('comm').\
        string_item('state').\
        int_item('ppid').\
        int_item('pgrp').\
        int_item('session').\
        int_item('tty_nr').\
        int_item('tpgid').\
        int_item('flags').\
        int_item('minflt').\
        int_item('cminflt').\
        int_item('majflt').\
        int_item('cmajflt').\
        int_item('utime').\
        int_item('stime').\
        int_item('cutime').\
        int_item('cstime').\
        int_item('priority').\
        int_item('nice').\
        int_item('num_threads').\
        int_item('itrealvalue').\
        int_item('starttime').\
        int_item('vsize').\
        int_item('rss').\
        int_item('rlim').\
        int_item('startcode').\
        int_item('endcode').\
        int_item('startstack').\
        int_item('kstkesp').\
        int_item('kstkeip').\
        int_item('signal').\
        int_item('blocked').\
        int_item('sigignore').\
        int_item('sigcatch').\
        int_item('wchan').\
        int_item('nswap').\
        int_item('cnswap').\
        int_item('exit_signal').\
        int_item('processor').\
        int_item('rt_priority').\
        int_item('policy').\
        int_item('delayacct_blkio_ticks').\
        int_item('guest_time').\
        int_item('cguest_time').\
        nonparsed_item('error')

    def __init__(self, path):
        _ProcBase.__init__(self)
        if path:
            self.parseProcFs(path)

    def parseProcFs(self, path):
        path = path + '/stat'
        try:
            with open(path) as stat_file:
                for line in stat_file: # Only one line in file
                    if line and line != '':
                        self.parse(line)
                        self.error = None
                    else:
                        self.error = 'Empty line'
        except IOError as ioerr:
            self.handle_error('IOError %s - %s' % (path, ioerr))


    def parse(self, line):
        # Dynamically (at run time) add properties to this instance representing
        # each stat value.  E.g. add the pid value as a field called self.pid
        split_line = ProcStat._parser.parse(line);
        if split_line:
            # Update the properties of the Stat instance with integer or
            # string values as appropriate.
            for sort_key, value in split_line.groupdict().items():
                self.__dict__[sort_key] = int(value) if INT_RE.match(value) else value
        else:
            self.handle_error('Failed to match:' + line)

    def keys(self):
        return ProcStat._parser.keys()



class ProcStatus(_ProcBase):

    def __init__(self, path):
        _ProcBase.__init__(self, path, 'status')
        if not self.error:
            self.uid = [ int(uid) for uid in string.split(self.uid,'\t')]

class ProcIO(_ProcBase):

    def __init__(self, path):
        _ProcBase.__init__(self, path, 'io')


class ProcInfo(object):

    def __init__(self, path):
        self.time_stamp = time.time()
        self.meta = {}
        self.stat = ProcStat(path)
        self.status = ProcStatus(path)
        self.io = ProcIO(path)
        self.username = pwd.getpwuid(self.status.uid[0]).pw_name if not self.hasErrors() else 'nobody'
        self.pid = int(path.split('/')[-1])

    def hasErrors(self):
        return self.stat.error or self.status.error or self.io.error

def get_all_proc_data(include_threads=False, root=PROC_FS_ROOT):
    if include_threads:
        results = [ProcInfo(task_path) for task_path in glob.glob(root + '/[0-9]*/task/[0-9]*')]
    else:
        results = [ProcInfo(task_path) for task_path in glob.glob(root + '/[0-9]*')]
    return [info for info in results if not info.hasErrors()]

def get_proc_info(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcInfo(root + '/' + pid + ('task/' + threadid) if threadid else '')
def get_proc_stat(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcStat(root + '/' + pid + ('task/' + threadid) if threadid else '')
def get_proc_status(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcStatus(root + '/' + pid + ('task/' + threadid) if threadid else '')
def get_proc_io(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcIO(root + '/' + pid + ('task/' + threadid) if threadid else '')

if __name__ == '__main__':

    usage = """usage: %prog [options] [pid...]
    Output CSV for procfs stat, status or io data for given thread/process pid's or
    for all processes and threads if no pid's are supplied."""
    parser = OptionParser(usage)
    parser.add_option('-s', '--stat', action='store_true', dest='do_stat', help='Output csv for pid/stat files.')
    parser.add_option('-S', '--status', action='store_true', dest='do_status', help='Output csv for pid/status files.')
    parser.add_option('-i', '--io', action='store_true', dest='do_io', help='Output csv for pid/io files.')
    parser.add_option('-t', '--titles', action='store_true', dest='output_titles', help='Output a title line.')
    parser.add_option('-r', '--repeat', action='store_true', dest='repeat', help='Repeat until interrupted.')
    parser.add_option('-w', '--sleep', type='int', dest='wait', default=5, help='Sleep seconds for each repetition.')
    parser.add_option('-p', '--processes', action='store_true', dest='processes_only', help='Show all processes, but not threads.')

    (options, args) = parser.parse_args()
    header = options.output_titles

    if len(args) == 0:
        args = [ '[0-9]*' ] # match all processes or threads
    elif options.processes_only:
        print >> sys.stderr, 'ignoring -p, showing requested processes and threads instead.'
        options.processes_only = False

    while True:
        for pid in args:
            for path in glob.glob(PROC_FS_ROOT + ('/' if options.processes_only else '/[0-9]*/task/') + pid):
                if options.do_stat or (not options.do_status and not options.do_io):
                    ProcStat(path).csv(sys.stdout, header=header)
                if options.do_status:
                    ProcStatus(path).csv(sys.stdout, header=header)
                if options.do_io:
                    ProcIO(path).csv(sys.stdout, header=header)
                header = False
        if not options.repeat:
            break
        time.sleep(options.wait)


Notes



Documentation for the Linux proc file-system can be found in the Linux proc (section 5) manual page (man 5 proc). The files I wanted to parse either contain several lines, with one value per line (status file and io file), or a single line, with multiple values per line (stat file). My linuxprocfs script contains some generalised code that should cope with basic parsing of both types of file and could be a basis for parsing other files in the procfs. The procfs is a prisoner of its history and suffers a bit from inconsistencies in it's syntax.

The linuxprocfs.py python script is coded for python 2.7 which includes all the dependent modules including the csv and options parsing modules.  I'm running an OpenSUSE 11.4 desktop, but I imagine the script will run on any of the modern Linux distributions.  The script may issue warnings to standard-error if processes disappear while it is traversing the procfs, these are normal and only diagnostic.

I have come across one other python interface to procfs, python-linux-procfs by Arnaldo Carvalho de Melo. It's source base is larger, and it decodes more details.  I will be looking into whether anything from my script is worth merging into this other version.

This post is dusting off some work I'd parked a couple of years back.  It's quite pleasant to return to python and its libraries - together they do more to close the gap between idea and implementation than any programming environment I've tired.

This blog page uses SyntaxHighlighter by Alex Gorbatchev.   You can easily copy and paste a line-number free version of  the code by selecting view-source icon in the mini-tool-bar that appears on the top right of the source listing (if javascript is enabled). If you click inside a source listing you will be able to use the arrow keys to scroll sideways.