Friday, December 16, 2011

RPM Changelogs for Recent Updates 10x Faster

In my previous post I presented a generalised rpm changelog summary script. I've now tidied up the implementation and added a couple of new options.

One thing bugging me was that the script exec'ed rpm for each package. Even though UNIX process creation is relatively inexpensive, the programs being exec'ed take time to initialise themselves, they have to open files, read configs, create internal structures, etc. The cumulative initialisation costs can be substantial. For example, the old makewhatis script that used to ship which many Linux distro's exec'ed gawk for every manual page, this took 30 minutes on a 486DX66. It was so annoying I rewrote it to exec gawk less often, and the the run time dropped to 1.5 minutes. The improved version is still included man-1.6g. Given how many machines were once running this script, the reduction in Carbon emissions may have been significant ;-)

By taking advantage of rpm's --queryformat option I've changed the rpmChangelogs script to exec rpm for 100 rpm arguments at a time.  This is about 10 times faster for large runs.  For example, when I generated a summary dating back to my upgrade from OpenSUSE 11.4 to 12.1, the run time reduced from about 50 seconds down to 5 seconds.

I've added an option to include the description of the package. And I've added and option to accept the rpm names from the command line instead of just doing the most recently installed ones.

Here is the syntax summary for the new version:

python rpmChangelogs.py -h
Usage: rpmChangelogs.py [options] [rpm...] 

Report change log entries for recently installed (-i) rpm's or for the rpm's
specified on the command line.

Options:
  -h, --help            show this help message and exit
  -i INSTALLDAYS, --installed-since=INSTALLDAYS
                        Include anything installed up to INSTALLDAYS days ago.
  -c CHANGEDAYS, --changed-since=CHANGEDAYS
                        Report change log entries from up to CHANGEDAYS days
                        ago.
  -d, --description     Include each rpm's description in the output.

Except for the optional addition of the description, the output is the same as the previous OpenSUSE only script.
My python is a little rusty - I just spent months doing Java - so I've also gone back over it and tried to tidy up the code.

The code


(Once you've expanded the code, hover over the code area to bring up options that make it easier to copy or print - requires javascript to be enabled.)
#!/usr/bin/env python
#
# rpmChangelogs.py 
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
# 
# Updated 2013/03/18: now uses seconds from 1970 to avoid localisation issues with the dates output by rpm.
#
import subprocess
from datetime import datetime,  timedelta
from optparse import OptionParser

maxArgsPerCommand=100

optParser = OptionParser(
            usage='Usage: %prog [options] [rpm...] ', 
            description="Report change log entries for recently installed (-i) rpm's or for the rpm's specified on the command line.")
optParser.add_option('-i',  '--installed-since',  dest='INSTALLDAYS', type='int', default=1,  help='Include anything installed up to INSTALLDAYS days ago.')
optParser.add_option('-c',  '--changed-since',  dest='CHANGEDAYS', type='int', default=60,  help='Report change log entries from up to CHANGEDAYS days ago.')
optParser.add_option('-d',  '--description',  dest='DESC', action='store_true', default=False,  help="Include each rpm's description in the output.")
(options, args) = optParser.parse_args()

installedSince = datetime.now() - timedelta(days=options.INSTALLDAYS)
changedSince = datetime.now() - timedelta(days=options.CHANGEDAYS)
showDesc = options.DESC

if len(args) > 0:
    recentPackages = args
else:
    queryProcess = subprocess.Popen(['rpm', '-q', '-a', '--last'], shell=False, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    recentPackages = []
    for queryLine in queryProcess.stdout:
        (name, dateStr) = queryLine.split(' ', 1)
        installDatetime = datetime.strptime(dateStr.strip(), '%a %d %b %Y %H:%M:%S %Z')
        if installDatetime < installedSince:
            break
        recentPackages.append(name)
    queryProcess.stdout.close()
    queryProcess.wait()
    if queryProcess.returncode != 0:
        print '*** ERROR (return code was ', queryProcess.returncode,  ')'
    for line in queryProcess.stderr:
        print line, 

# Use one rpm exec to query multiple packages - 10x faster than an exec for each one
marker = '+Package: '
markerLen = len(marker)
for subset in [recentPackages[i:i+maxArgsPerCommand] for i in range(0, len(recentPackages), maxArgsPerCommand)]:
    format = marker + '%{INSTALLTIME} %{NAME}-%{VERSION}-%{RELEASE}\n' + ('%{DESCRIPTION}\n\n+Changelog:\n' if showDesc else '')
    rpmProcess = subprocess.Popen(['rpm', '-q', '--queryformat=' + format, '--changelog'] + subset, shell=False, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    tooOld = False
    for line in rpmProcess.stdout:
        if line.startswith(marker):
            installedDate = datetime.fromtimestamp(float(line[markerLen:line.rfind(' ')]))
            name = line.rsplit(' ',  1)[1]
            print '=================================================='
            print marker,  installedDate, name, 
            print '------------------------------'
            tooOld = False
        else:
            if line.startswith('* ') and len(line) > 17:
                try:
                    changeDate = datetime.strptime(line[:line.rfind(' ')], '* %a %b %d %Y')
                    tooOld = changeDate < changedSince
                except ValueError:
                    pass # not a date - move on
            if not tooOld: 
                print line, 
    rpmProcess.stdout.close()
    rpmProcess.wait()
    if rpmProcess.returncode != 0:
        print '*** ERROR (return code was ', rpmProcess.returncode,  ')'
    for line in rpmProcess.stderr:
        print line, 
    rpmProcess.stderr.close()

Thursday, December 15, 2011

RPM Changelogs for Recent Updates

Note, in a more recent post I've sped up this code ten times.
In my previous post I showed you a script that could report recent changelogs for OpenSUSE packages. Overnight I realised I could generalise this to all RPM based distros. Here is a new generalised version:
% python rpmChangeLogs.py -h
Usage: rpmChangeLogs.py [options]

Report change log entries for recent rpm installs.

Options:
  -h, --help            show this help message and exit
  -i INSTALLDAYS, --installed-since=INSTALLDAYS
                        Include anything installed up to INSTALLDAYS days ago.
  -c CHANGEDAYS, --changedSince=CHANGEDAYS
                        Report change log entries from up to CHANGEDAYS days
                        ago.


The output is the same as the previous OpenSUSE only script.

I've also cleaned up the code around python sub-processes.

The code


#!/usr/bin/env python
#
# rpmChangelogs.py 
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import subprocess
from datetime import date, datetime,  timedelta
from optparse import OptionParser

optParser = OptionParser(description='Report change log entries for recent rpm installs.')
optParser.add_option('-i',  '--installed-since',  dest='INSTALLDAYS', type='int', default=1,  help='Include anything installed up to INSTALLDAYS days ago.')
optParser.add_option('-c',  '--changedSince',  dest='CHANGEDAYS', type='int', default=60,  help='Report change log entries from up to CHANGEDAYS days ago.')
(options, args) = optParser.parse_args()

installedSince = datetime.now() - timedelta(days=options.INSTALLDAYS)
changedSince = datetime.now() - timedelta(days=options.CHANGEDAYS)

queryProcess = subprocess.Popen(['rpm', '-q', '-a', '--last'], shell=False, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
for queryLine in queryProcess.stdout:
    historyRec = str.split(queryLine, ' ', 1)
    installDatetime = datetime.strptime(str.strip(historyRec[1])[4:24], '%d %b %Y %H:%M:%S')
    if installDatetime < installedSince:
        break
    packageName = historyRec[0]
    print '=================================================='
    print '+Package: ',  installDatetime, packageName
    print '------------------------------'
    rpmProcess = subprocess.Popen(['rpm', '-q', '--changelog',  packageName], shell=False, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    for line in rpmProcess.stdout:
        try:
            if line[0] == '*' and line[1] == ' ' and len(line) > 17:
                changeDate = datetime.strptime(line[6:17], '%b %d %Y')
                if changeDate < changedSince:
                    break
        except ValueError:
            pass # not a date - move on
        print line, 
    rpmProcess.stdout.close()
    rpmProcess.wait()
    if rpmProcess.returncode != 0:
        print '*** ERROR (return code was ', rpmProcess.returncode,  ')'
    for line in rpmProcess.stderr:
        print line, 
    rpmProcess.stderr.close()
queryProcess.stdout.close()
queryProcess.wait()
if queryProcess.returncode != 0:
    print '*** ERROR (return code was ', queryProcess.returncode,  ')'
for line in queryProcess.stderr:
   print line, 

Wednesday, December 14, 2011

OpenSUSE Changelogs for Recent Updates

Here is a short python script that shows recent portions of the changelogs for recently installed packages. This script is indented for extracting a summary of what has changed after updating my OS to latest packages. Usage is as follows:
% python zyppHist.py -h
Usage: zyppHist.py [options]

Report change log entries for recent installs (zypper/rpm).

Options:
  -h, --help            show this help message and exit
  -i INSTALLDAYS, --installed-since=INSTALLDAYS
                        Include anything installed up to INSTALLDAYS days ago.
  -c CHANGEDAYS, --changedSince=CHANGEDAYS
                        Report change log entries from up to CHANGEDAYS days
                        ago.


Sample output:
python zyppHist.py -i 1 -c 30
==================================================
+Package:  2011-12-14 21:11:12 glibc
------------------------------
* Wed Nov 30 2011 aj@suse.de
- Do not install INSTALL file.

* Wed Nov 30 2011 rcoe@wi.rr.com
- fix printf with many args and printf arg specifiers (bnc#733140)

* Fri Nov 25 2011 aj@suse.de
- Updated glibc-ports-2.14.1.tar.bz2 from ftp.gnu.org.

* Fri Nov 25 2011 aj@suse.com
- Create glibc-devel-static baselibs (bnc#732349).

* Fri Nov 18 2011 aj@suse.de
- Remove duplicated locales from glibc-2.3.locales.diff.bz2

==================================================
+Package:  2011-12-14 21:11:21 splashy
------------------------------
* Thu Dec 08 2011 hmacht@suse.de
- update artwork for openSUSE 12.1 (bnc#730050)

==================================================
+Package:  2011-12-14 21:11:23 libqt4
------------------------------
* Wed Nov 23 2011 llunak@suse.com
- do not assert on QPixmap usage in non-GUI threads
  if XInitThreads() has been called (bnc#731455)

==================================================
+Package:  2011-12-14 21:11:23 libcolord1
------------------------------
* Wed Dec 07 2011 vuntz@opensuse.org
- Update to version 0.1.15:
  + This release fixes an important security bug: CVE-2011-4349.
  + New Features:
  - Add a native driver for the Hughski ColorHug hardware
  - Export cd-math as three projects are now using it
  + Bugfixes:
  - Documentation fixes and improvements
  - Do not crash the daemon if adding the device to the db failed
  - Do not match any sensor device with a kernel driver
  - Don't be obscure when the user passes a device-id to colormgr
  - Fix a memory leak when getting properties from a device
  - Fix colormgr device-get-default-profile

...

The script produces a summary by extracting a list of recent installs from /var/log/zypp/history. For each recent install the script issues an rpm changelog query to obtain each package's changelog. Each changelog is written out line by line and the output is truncated when a date is encountered that is more than the specified number of days in the past.

The code


#!/usr/bin/env python
#
# zypphist.py 
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import csv
import subprocess
from datetime import date, datetime,  timedelta
from optparse import OptionParser

zyppHistFilename = '/var/log/zypp/history'

optParser = OptionParser(description='Report change log entries for recent installs (zypper/rpm).')
optParser.add_option('-i',  '--installed-since',  dest='INSTALLDAYS', type='int', default=1,  help='Include anything installed up to INSTALLDAYS days ago.')
optParser.add_option('-c',  '--changedSince',  dest='CHANGEDAYS', type='int', default=60,  help='Report change log entries from up to CHANGEDAYS days ago.')
(options, args) = optParser.parse_args()

installedSince = datetime.now() - timedelta(days=options.INSTALLDAYS)
changedSince = datetime.now() - timedelta(days=options.CHANGEDAYS)

zyppHistReader = csv.reader(open(zyppHistFilename, 'rb'), delimiter='|')
for historyRec in zyppHistReader:
    if historyRec[0][0] != '#' and historyRec[1] == 'install':
        installDate = datetime.strptime(historyRec[0], '%Y-%m-%d %H:%M:%S')
        if installDate >= installedSince:
            packageName = historyRec[2]
            print '=================================================='
            print '+Package: ',  installDate, packageName
            print '------------------------------'
            rpmProcess = subprocess.Popen(['rpm', '-q', '--changelog',  packageName], shell=False, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
            rpmProcess.wait()
            if rpmProcess.returncode != 0:
                print '*** ERROR (return code was ', rpmProcess.returncode,  ')'
            for line in rpmProcess.stderr:
                print line, 
            for line in rpmProcess.stdout:
                try:
                    if line[0] == '*' and line[1] == ' ' and len(line) > 17:
                        changeDate = datetime.strptime(line[6:17], '%b %d %Y')
                        if changeDate < changedSince:
                            break
                except ValueError:
                    pass # not a date - move on
                print line, 
            rpmProcess.stdout.close()
            rpmProcess.stderr.close()

Monday, July 25, 2011

Collectfs - a trash collecting userspace file system for Linux

Collectfs is a FUSE userspace filesystem that provides add-on trash collection for a directory hierarchy. The purpose of collectfs is to protect a project hierarchy by providing a fairly universal no-clobber mechanism:
  • The history of changes is preserved.
  • Missteps in using rm, mv, cat, etc are non-permanent.
  • It works seamlessly with standard development tools.
It is not intended as a replacement for revision control or backups. The intention is to protect you during the between-times, when you're not covered by these other tools.

Any file that is overwritten by remove (unlink), move, link, symlink, or open-truncate is relocated to a trash directory (mount-point/.trash/). Removed files are date-time stamped so that edit history is maintained (a version number is appended if the same file is collected more than once in the same second).

Usage is quite straight forward, for example:
# Use collectfs to mount the real folder onto a mount point (any other folder) 
% collectfs myProject myWorkspace
# Now the mount point mirrors the original but with trash collection
% cd myWorkspace
% vi main.c
% indent main.c
% ls .trash
main.c.2011-07-24.14:48:20
main.c.2011-07-24.14:59:39
% diff .trash/main.c.2011-07-24.14:59:39 main.c
...
% mv .trash/main.c.2011-07-24.14:59:39 main.c
% ls .trash
main.c.2011-07-24.14:48:20
main.c.2011-07-24.15:00:37
# To unmount (stop using) the virtual filesystem...
% cd ..
% fusermount -u myWorkspace
It's easy to build it from source. The source is available at: http://code.google.com/p/collectfs/
Thanks to Thomas Spahni (vodoo on the http://forums.opensuse.org/) the openSUSE build services now has a collectfs package. (I also used Thomas's much more concise description of collectfs to rewrite the first paragraph of this blog post)

I'm currently using collectfs to help write collectfs. It augments my development environment with a local-history feature similar to that provided by eclipse. But unlike eclipse, KDE or gnome, the protection is implemented in the filesystem layer and applies to any tool I care to employ.

Thoughts on a deeper level... I don't think eclipse, KDE, or gnome should be in the business of collecting trash and providing undelete. The proper place to do this is in the filesystem. But this would be a very radical departure for a UNIX filesystem, probably too radical to feature in main-stream UNIX filesystem development. Such features do not fit well with established practice, and raise concerns about security and privacy (data coming back to haunt you). I do think that if you want to think beyond Linux/KDE/gnome, concepts such as trash, undelete, file-versioning are all things that would be worth consideration for a true desktop OS. I believe GNU Hurd was originally to have file-versioning similar to VMS - I considered using fuse to implement VMS-style file-versioning:

xyz.txt;1 xyz.txt;2 xyz.txt;3 ... xyz.txt;N
open(xyz.txt) == open(xyz.txt;N)

but I never liked the clutter in my VMS directories, so I went for timestamped trash instead.

Friday, June 17, 2011

Linux as seen by matplotlib - the view from 2000 metres

In my previous post I shared some python code that extracts Linux process-file-system data into python objects. In this post I will augment the code from the previous post with some new code that generates rolling plots of process file system data.

I've been playing around with generating detailed overviews of my running Linux system. Existing tools, such as top or htop, do a pretty good job of displaying the processes consuming the highest resources, but I'm looking for an at-a-glance overview of all processes and their impact on the system.   I've used this goal as a reason to learn a little more about the python plotting tool matplotlib.

The plot to the right is generated by a matplotlib python script producing something similar to the LibreOffice plot in my previous blog entry.  The process-file-system extraction code from the previous blog posting is used to extract CPU, vsize, and RSS data.  Unlike the previous example, the script refreshes the plot every few seconds.   This new plot uses a hash to pick a colour for each user and also annotates each point with the PID, command, and username.

Plotting every process results in a very cluttered graph  To reduce the clutter I have restricted to plot data to the "interesting" processes - those that have changing values.  Interest decays with inactivity, so processes that become static eventually disappear from the graph.

I wasn't happy with the bubble plot because clutter made it hard to get an overview of all process activity, which was the whole point of the exercise.  Time for a different tack.

The plot in the screen-capture on the right is an attempt at an at-a-glance view of every process and thread  This new script periodically extracts stat, status, and IO data from the Linux process file-system.  The script calculates changes in the data between refresh cycles and plots indications of the changes. There is a point for each process and thread.  Points are grouped by user and ordered by process start time.  The size of each point indicates relative resident set size.  The colours of each point represent the process's state during the cycle:

  1. Light grey: no activity
  2. Red: consumed CPU
  3. Green: perfomed IO
  4. Orange: Consumed CPU and performed IO
  5. Yellow: exited.

Changes in RSS are exaggerated so that even small changes in RSS are visible as a little "throb" in the point size.  The plot updates once every three seconds or when ever the space bar is pressed.  By grouping by username and start time, the placement of points remains fairly static and predictable from cycle to cycle.

In matplotlib it is relatively easy to trap hover and click events.  I've included code to trap hover and display a popup details for the process under the mouse.   In  the screen-capture above, I've hovered over the Xorg X-Windows server.  I also trap right-click and pin the popup so that will remain updating even after the mouse is moved away.

The the number of memory hogs visible on the plot is a bit exaggerated because the plot includes threads as well as processes (threads share memory).  This plot is somewhat similar to the Hinton Diagram seen in the matplotlib example pages, which may also be an interesting basis for viewing a collection of processes or servers.  (My own efforts pre-dates my discovering the Hinton example, so the implementation details differ substantially.)


Finally, I thought I'd include a 3D plot. I'm not sure it's that useful, but adding an extra dimension is food for thought.  The script plots cumulative-CPU and RSS over the duration of the scripts run time.  It uses a heuristic to select "interesting" processes.  In this particular screen capture I shut down my eclipse+android IDE - first set of terminated lines - and then restarted the same IDE (steep climbing lines).  The CPU and memory consumed by the plotting script can be seen as the diagonal line across the bottom time-CPU axis.


Matplotlib is a really useful plotting library, when ever I thought I'd exhausted what was possible, a Google-search would usually prove me wrong.  

The code


plot_bubble_cpu_vsize_rss.py


#!/usr/bin/env python
#
# Bubble Plot: cpu (x), vsize(y), rss(z) 
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import matplotlib
matplotlib.use('GTkAgg')
import pylab
import gobject
import sys
import os
import math
from optparse import OptionParser
import linuxprocfs as procfs

SC_CLK_TCK=float(os.sysconf(os.sysconf_names['SC_CLK_TCK']))
ZOMBIE_LIVES=2 # Not a true UNIX zombie, just a plot zombie.
INTERESTING_DEFAULT=3 # Don't display a process if it is boring this many times in a row.
INTERVAL = 2

_user_colors = {}

def _user_color(username):
    if username in _user_colors:
        color = _user_colors[username]
    else: 
        # For a username, generate a psuedo random (r,g,b) values in the 0..1.0 range 
        # E.g. for root base this on hash(root), hash(oot), and hash(ot) - randomise
        # each hash further - seems to work better than the straight hash 
        color = tuple([math.fabs(hash(username[i:]) * 16807 % 2147483647)/(2147483647 * 1.3) for i in (0, 2, 1)])
        #color = tuple([abs(hash(username[i:]))/(sys.maxint*1.3) for i in (2, 1, 0)])
        _user_colors[username] = color
        print username,  color
    return color

class TrackingData(object):

    def __init__(self, info):
        self.previous = info
        self.current = info
        self.color = _user_color(info.username)
        self.text = '%d %s %s' % (info.pid, info.username, info.stat.comm )
        self.interesting = 0
        self.x = 0
        self.y = info.stat.vsize 
        self.z = info.stat.rss / 10
        self.zombie_lives = ZOMBIE_LIVES

    def update(self, info):
        self.current = info
        oldx = self.x
        oldy = self.y
        oldz = self.z
        self.x = ((info.stat.stime + info.stat.utime) - (self.previous.stat.stime + self.previous.stat.utime)) /SC_CLK_TCK
        self.y = info.stat.vsize 
        self.z = info.stat.rss / 10
        if self.x == oldx and self.y == oldy and self.z == oldz: 
            if self.interesting > 0:
                self.interesting -= 1  # if interesting drops to zero, stop plotting this process
        else:
            self.interesting = INTERESTING_DEFAULT
        self.previous = self.current
        self.zombie_lives = ZOMBIE_LIVES
        
    def is_alive(self):
        self.zombie_lives -= 1
        return self.zombie_lives > 0


class PlotBubbles(object):

    def __init__(self, sleep=INTERVAL, include_threads=False):
        self.subplot = None
        self.datums = {}
        self.sleep = sleep * 1000
        self.include_threads = include_threads

    def make_graph(self):
        all_procs = procfs.get_all_proc_data(self.include_threads)
        y_list = []
        x_list = []
        s_list = []
        color_list = []
        anotations = []
 
        for proc_info in all_procs:
            if not proc_info.pid in self.datums:
                data = TrackingData(proc_info)
                self.datums[proc_info.pid] = data
            else:
                data = self.datums[proc_info.pid]
                data.update(proc_info)
            if data.interesting > 0: # Only plot active processes
                x_list.append(data.x)
                y_list.append(data.y)
                s_list.append(data.z)
                color_list.append(data.color)
                anotations.append((data.x, data.y, data.color, data.text))
            if not data.is_alive():
                del self.datums[proc_info.pid]
        if self.subplot == None:
            figure = pylab.figure()
            self.subplot = figure.add_subplot(111)
        else:
            self.subplot.cla()
        if len(x_list) == 0: # Nothing to plot - probably initial cycle
            return True
        self.subplot.scatter(x_list, y_list, s=s_list, c=color_list, marker='o', alpha=0.5)
        pylab.xlabel(r'Change in CPU (stime+utime)', fontsize=20)
        pylab.ylabel(r'Total vsize', fontsize=20)
        pylab.title(r'Process CPU, vsize, and RSS')
        pylab.grid(True)
        gca = pylab.gca()
        for x, y, color, text in anotations:
            gca.text(x, y, text, alpha=1, ha='left',va='bottom',fontsize=8, rotation=33, color=color)
        pylab.draw()
        return True

    def start(self):
        self.scatter = None
        self.make_graph()
        pylab.connect('key_press_event', self)
        gobject.timeout_add(self.sleep, self.make_graph)
        pylab.show()

    def __call__(self, event):
      self.make_graph()

if __name__ == '__main__':
    usage = """usage: %prog [options|-h]
    Plot change in CPU(x), total vsize(y) and total RSS.
    """
    parser = OptionParser(usage)
    parser.add_option("-t", "--threads", action="store_true", dest="include_threads", help="Include threads as well as processes.")
    parser.add_option("-s", "--sleep", type="int", dest="interval", default=INTERVAL, help="Sleep seconds for each repetition.")

    (options, args) = parser.parse_args()

    PlotBubbles(options.interval, options.include_threads).start()


% python plot_bubble_cpu_vsize_rss.py -h
Usage: plot_bubble_cpu_vsize_rss.py [options|-h]
    Plot change in CPU(x), total vsize(y) and total RSS.

Options:
  -h, --help            show this help message and exit
  -t, --threads         Include threads as well as processes.
  -s INTERVAL, --sleep=INTERVAL
                        Sleep seconds for each repetition.

plot_grid.py


#!/usr/bin/env python
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import matplotlib
matplotlib.use('GTkAgg')
import pylab
import time
import gobject
import math
import os
import linuxprocfs as procfs
from optparse import OptionParser

SC_CLK_TCK=float(os.sysconf(os.sysconf_names['SC_CLK_TCK']))
LIMIT=1000
INTERVAL=3
DEFAULT_COLS=30
BASE_POINT_SIZE=300
MIN_SIZE=20
ZOMBIE_LIVES=3 # Not a true UNIX zombie, just a plot zombie
MAX_RSS = 10
# Don't spell colour two ways - conform with pylab
DEFAULT_COLORS=['honeydew','red','lawngreen','orange','yellow','lightyellow','white']

class ProcessInfo(object):
    def __init__(self, new_procdata):
        self.data = new_procdata
        self.previous = None
        self.alive = True
        self.info = '%d %s %s' % (new_procdata.pid, new_procdata.username, new_procdata.stat.comm )
        self.username = new_procdata.username
        self.color = 'blue'
        self.zombie_lives = ZOMBIE_LIVES
    
    def update(self, new_procdata):
        self.previous = self.data
        self.data = new_procdata
        self.zombie_lives = ZOMBIE_LIVES
        
    def report(self):
        return '%s\nstate=%s\nutime=%f\nstime=%f\nrss=%d\nreads=%d\nwrites=%d' % \
             (self.info, 
              self.data.status.state if not self.is_zombie() else 'exited', 
              self.data.stat.utime/SC_CLK_TCK, 
              self.data.stat.stime/SC_CLK_TCK,
              self.data.stat.rss,
              self.data.io.read_bytes,
              self.data.io.write_bytes)
        
    def is_alive(self):
        self.zombie_lives -= 1
        return self.zombie_lives > 0
    
    def is_zombie(self):
        return self.zombie_lives < ZOMBIE_LIVES - 1
    
    def sort_key(self):
        return (self.username, self.data.stat.starttime, self.data.pid)

class Activity_Diagram(object):

    def __init__(self, sleep=INTERVAL, max_cols=DEFAULT_COLS, point_size=BASE_POINT_SIZE, include_threads=True, colors=''):
        self.process_info = {}
        self.pos_index = {}
        self.subplot = self.label = self.hover_tip = self.hover_tip_data = None
        self.hover_tip_sticky = False
        self.include_threads = include_threads
        self.sleep = sleep
        self.max_cols = max_cols
        self.point_size = point_size
        # Extend colors to same length as default, merge together colors and default, choose non blank values
        self.normal_color, self.cpu_color, self.io_color, self.cpuio_color, self.exit_color, self.tip_color, self.bg_color = \
            [ (c if c != '' and c != None else d) for c,d in map(None, colors, DEFAULT_COLORS)]

    def start(self):
        self.start_time = time.time()
        self._create_new_plot()
        pylab.connect('motion_notify_event', self)
        pylab.connect('button_press_event', self)
        pylab.connect('key_press_event', self)
        gobject.timeout_add(self.sleep * 1000, self._create_new_plot)
        pylab.show()

    def _create_new_plot(self):
        if self.subplot == None:
            figure = pylab.figure()
            figure.set_frameon(True)
            self.subplot = figure.add_subplot(111, axis_bgcolor=self.bg_color)
        else:
            self.subplot.cla()    
        plot = self.subplot
        x_vals, y_vals, c_vals, s_vals, namelabels = self._retrieve_data()
        plot.scatter(x_vals, y_vals, c=c_vals, s=s_vals) #, label=data.info)
        plot.axis('equal')
        plot.set_xticks([])
        plot.set_yticks(namelabels[0])
        plot.set_yticklabels(namelabels[1], stretch='condensed')
        self._create_tip(plot) # Refresh the tip the user is currently looking at
        pylab.title('Activity: size=RSS changes; red=CPU; green=IO; orange=CPU and IO.')
        pylab.draw()
        #print "ping"
        return True
    
    def _retrieve_data(self):
        # Update from procfs
        for proc in procfs.get_all_proc_data(include_threads=self.include_threads):
            if not proc.pid in self.process_info:
                data = ProcessInfo(proc)
                self.process_info[proc.pid] = data
            else:
                data = self.process_info[proc.pid]
                data.update(proc)
        max_rss = 0
        for info in self.process_info.values():
            if max_rss < info.data.stat.rss: 
                max_rss = info.data.stat.rss
        # Compute values for plotting
        x_vals = []; y_vals = []; c_vals = []; s_vals = []
        usernames = [[],[]]
        col = 0; row = 0
        self.pos_index = {}
        previous = None
        for info in sorted(self.process_info.values(), key=lambda info: info.sort_key()):
            if not info.is_alive():
                del self.process_info[info.data.pid]
            else:
                if (not previous or previous.username != info.username):
                    if col != 0: 
                        col = 0; row += 1
                    print info.username, -row
                    usernames[0].append(-row)
                    usernames[1].append(info.username)
                self.pos_index[(col,-row)] = info
                x_vals.append(col)
                y_vals.append(-row) # Invert ordering
                c_vals.append(self._decide_color(info))
                s_vals.append(self._decide_size(info,  max_rss))
                col += 1
                if col == self.max_cols:
                    col = 0; row += 1
                previous = info
        return (x_vals, y_vals, c_vals, s_vals, usernames)
                
    def _decide_color(self,info): 
        if info.is_zombie():
            return self.exit_color
        if info.previous == None:
            delta_cpu = delta_io = 0     
        else:
            delta_cpu = (info.data.stat.utime + info.data.stat.stime) - (info.previous.stat.utime + info.previous.stat.stime)
            delta_io = (info.data.io.read_bytes + info.data.io.write_bytes) - (info.previous.io.read_bytes + info.previous.io.write_bytes)
        color = self.normal_color
        if delta_io > 0:
            color = self.io_color
        if delta_cpu > 0 or info.data.stat.state == 'R':
            if delta_io > 0:
                color = self.cpuio_color
            else:
                color = self.cpu_color
        return color
 
    def _decide_size(self, info,  max_rss):    
        rss = info.data.stat.rss
        delta_rss = rss - (info.previous.stat.rss if info.previous else rss)
        # A relative proportion of the base dot size
        size = max(MIN_SIZE, self.point_size * rss / max_rss)
        if delta_rss > 0: # Temporary throb to indicate change
            size += max(20,size/4) 
        elif delta_rss < 0:
            size -= max(20,size/4)
        return size

    def _create_tip(self, axes, x=None, y=None, data=None, toggle_sticky=False):
        if data:
            if not self.hover_tip_sticky or toggle_sticky:
                if self.hover_tip: self.hover_tip.set_visible(False)
                self.hover_tip = axes.text(x, y, data.report(), bbox=dict(facecolor=self.tip_color, alpha=0.85), zorder=999)
                self.hover_tip_data = (x,y,data)
            if toggle_sticky: self.hover_tip_sticky = not self.hover_tip_sticky
        elif self.hover_tip_data:
            x,y,data = self.hover_tip_data
            self.hover_tip = axes.text(x, y, data.report(), bbox=dict(facecolor=self.tip_color, alpha=0.85), zorder=999)

    def _clear_tip(self):
        if self.hover_tip and not self.hover_tip_sticky:
            self.hover_tip.set_visible(False) # will be free'ed up by next plot draw 
            self.hover_tip = self.hover_tip_data = None       

    def __call__(self, event):
        #print event.name
        if event.name == 'key_press_event':
            self._create_new_plot()
        else: 
            if (event.name == 'motion_notify_event' or event.name == 'button_press_event') and event.inaxes:
                point = (int(event.xdata + 0.5), int(event.ydata - 0.5))
                if point in self.pos_index:  # On button click let tip stay open without hover
                    self._create_tip(event.inaxes, event.xdata, event.ydata, self.pos_index[point], event.name == 'button_press_event')
            else:
                self._clear_tip() 
            pylab.draw()

if __name__ == '__main__':
    
    usage = """usage: %prog [options|-h]
    Plot RSS, CPU and IO, with hover and click for details.
    RSS size is plotted as a circle - the circle will temporarily
    jump up and down in size to indicate a growing or shrinking 
    RSS - the steady state size is a relative size indicator.
    """
    parser = OptionParser(usage)
    parser.add_option("-p", "--no-threads", action="store_true", dest="no_threads", help="Exclude threads, only show processes.")    
    parser.add_option("-s", "--sleep", type="int", dest="interval", default=INTERVAL, help="Sleep seconds for each repetition.")
    parser.add_option("-n", "--columns", type="int", dest="columns", default=DEFAULT_COLS, help="Number of columns in each row (maximum).")
    parser.add_option("-d", "--point-size", type="int", dest="point_size", default=BASE_POINT_SIZE, help="Dot point size (expressed as square area).")
    parser.add_option("-c", "--colors", type="string", dest="colors", default='', 
                      help="Colors for normal,cpu,io,cpuio,exited,tip,bg comma separated. " + 
                      "Only supply the ones you want to change e.g. -cwhite,,blue - " + 
                      " Defaults are " + ','.join(DEFAULT_COLORS))

    (options, args) = parser.parse_args()
    Activity_Diagram(options.interval, options.columns, options.point_size, not options.no_threads, options.colors.split(',')).start()


% python plot_grid.py -h
Usage: plot_grid.py [options|-h]
    Plot RSS, CPU and IO, with hover and click for details.
    RSS size is plotted as a circle - the circle will temporarily
    jump up and down in size to indicate a growing or shrinking 
    RSS - the steady state size is a relative size indicator.

Options:
  -h, --help            show this help message and exit
  -p, --no-threads      Exclude threads, only show processes.
  -s INTERVAL, --sleep=INTERVAL
                        Sleep seconds for each repetition.
  -n COLUMNS, --columns=COLUMNS
                        Number of columns in each row (maximum).
  -d POINT_SIZE, --point-size=POINT_SIZE
                        Dot point size (expressed as square area).
  -c COLORS, --colors=COLORS
                        Colors for normal,cpu,io,cpuio,exited,tip,bg comma
                        separated. Only supply the ones you want to change
                        e.g. -cwhite,,blue -  Defaults are
                        honeydew,red,lawngreen,orange,yellow,lightyellow,white

plot_cpu_vsize_time_line_3d.py


#!/usr/bin/env python
#
# Plot a limited time line in 3D for CPU and RSS
#
# Copyright (C) 2011: Michael Hamilton
# The code is GPL 3.0(GNU General Public License) ( http://www.gnu.org/copyleft/gpl.html )
#
import matplotlib
matplotlib.use('GTkAgg')
import pylab
import time
import gobject
import gtk
import random
import string
import math
import mpl_toolkits.mplot3d.axes3d as axes3d
from optparse import OptionParser

import linuxprocfs as procfs


HZ=1000.0
LIMIT=1000
INTERVAL=5
ZOMBIE_LIVES=5
MAX_INTERESTING=5.0

class ProcessInfo(object):

    def __init__(self, new_procdata):
        self.base = new_procdata
        self.previous = new_procdata
        self.data = new_procdata
        self.text = '%d %s %s' % (new_procdata.pid, new_procdata.username, new_procdata.stat.comm )
        self.color = '#%6.6x' % (((self.base.pid * 16807 % 2147483647)/(2147483647 * 1.3)) % 2**24 )
        self.color = tuple([(self.base.pid * i * 16807 % 2147483647)/(2147483647 * 1.3) for i in (7, 13, 23)])
        self.xvals = []
        self.yvals = []
        self.zvals = []
        self.zombie_lives = ZOMBIE_LIVES
        self.interesting = 0
        self.visible = False

    def update(self, new_procdata):
        self.previous = self.data
        self.data = new_procdata
        self.zombie_lives = 5
        x = self.data.time_stamp
        y = ((self.data.stat.utime - self.base.stat.utime) + (self.data.stat.stime - self.base.stat.stime)) / HZ
        z = float(new_procdata.stat.rss)
        if len(self.xvals) > LIMIT:
            del self.xvals[0]
            del self.yvals[0]
            del self.zvals[0]
        self.xvals.append(x)
        self.yvals.append(y)
        self.zvals.append(z)
        if self.data.stat.utime - self.previous.stat.utime > 10.0 or self.data.stat.stime - self.previous.stat.stime > 10.0 or abs(self.data.stat.rss - self.previous.stat.rss) > 15000:
            self.visible = True
            self.interesting = MAX_INTERESTING

class Activity_Diagram(object):

    def __init__(self, remove_dead=True):
        self.process_info = {}
        self.start_time = None
        self.subplot = None
        self.remove_dead = remove_dead

    def make_graph(self):
        for proc in procfs.get_all_proc_data(include_threads=True):
            if not proc.pid in self.process_info:
                data = ProcessInfo(proc)
                self.process_info[proc.pid] = data
            else:
                data = self.process_info[proc.pid]
                data.update(proc)
        if self.subplot == None:
            figure = pylab.figure()
            self.subplot = figure.add_subplot(111, projection='3d')
        else:
            self.subplot.cla()
        for pid, data in self.process_info.items():
            if data.zombie_lives <= 0 and self.remove_dead:
                del self.process_info[pid]  # dead for a while now - remove
            data.zombie_lives -= 1
            if data.visible:
                if len(data.xvals) > 0: 
                    alpha =data.interesting / MAX_INTERESTING  # boring processes fade away
                    self.subplot.plot(data.xvals, data.yvals, data.zvals, color=data.color, alpha=alpha, linewidth=2)
                    self.subplot.text(data.xvals[-1], data.yvals[-1], data.zvals[-1], data.text, alpha=alpha, fontsize=6, color = data.color if data.zombie_lives >= ZOMBIE_LIVES - 1 else 'r')
            data.interesting -= 1 if data.interesting > 2 else 0 # Losing interest
        self.subplot.set_xlabel(r'time', fontsize=10)
        self.subplot.set_ylabel(r'cpu', fontsize=10)
        self.subplot.set_zlabel(r'rss', fontsize=10)
        pylab.title('Cumulative CPU and RSS over %f minutes' % ((time.time() - self.start_time) /60.0))
        pylab.draw()
        return True

    def start(self,  sleep_secs=INTERVAL):
        self.start_time = time.time()
        self.make_graph()
        pylab.connect('key_press_event', self)
        gobject.timeout_add(INTERVAL*1000, self.make_graph)
        pylab.show()

    def __call__(self, event):
      self.make_graph()

def boo():
    print 'boo'

if __name__ == '__main__':
    usage = """usage: %prog [options|-h]
    Plot cumulative-CPU and RSS over the script run time.
    """
    parser = OptionParser(usage)
    parser.add_option("-s", "--sleep", type="int", dest="sleep_secs", default=INTERVAL, help="Sleep seconds for each repetition.")
    (options, args) = parser.parse_args()
    Activity_Diagram(remove_dead=False).start(sleep_secs=options.sleep_secs)

% python plot_cpu_rss_time_lines_3d.py -h
Usage: plot_cpu_rss_time_lines_3d.py [options|-h]
    Plot cumulative-CPU and RSS over the script run time.
    

Options:
  -h, --help            show this help message and exit
  -s SLEEP_SECS, --sleep=SLEEP_SECS
                        Sleep seconds for each repetition.

Notes


Even if you don't know any python, you can still have a play by running these scripts from the command line. Just save them into a folder giving them the appropriate file-names and also save the linuxprocfs.py from my previous post. Make sure you've installed python-matplotlib for you distribution of Linux (it's a standard offering for openSUSE, so it's in the repo).

Consult the notes from the previous blog entry for some notes on the Linux process file-system.

The matplotlib Web-site ( http://matplotlib.sourceforge.net/ ) contains plenty of documentation and examples, plus Google will track down heaps more advice and examples.

This blog page uses SyntaxHighlighter by Alex Gorbatchev.   You can easily copy and paste a line-number free version of  the code by selecting view-source icon in the mini-tool-bar that appears on the top right of the source listing (if javascript is enabled).


Friday, June 10, 2011

Linux proc stat, status, io to CSV via python


I've written a python script that extracts data from the Linux Process File-system into python objects. I've included an option to dump data to CSV. The CSV can be directly loaded into tools such LibreOffice Calc for analysis and plotting.  The screen-capture to the right shows a LibreOffice Calc plot of data for processes running on my desktop.  For each process the plot shows total-CPU-time (x-axis), allocated virtual memory size (y-axis), and resident set size (bubble area).

My python script can be run from the command line and includes a variety of command line options including --help, for example:

% python linuxprocfs.py -h
Usage: linuxprocfs.py [options] [pid...]
    Output CSV for procfs stat, status or io data for given thread/process pid's or
    for all processes and threads if no pid's are supplied.

Options:
  -h, --help            show this help message and exit
  -s, --stat            Output csv for pid/stat files.
  -S, --status          Output csv for pid/status files.
  -i, --io              Output csv for pid/io files.
  -t, --titles          Output a title line.
  -r, --repeat          Repeat until interrupted.
  -w WAIT, --sleep=WAIT
                        Sleep seconds for each repetition.
  -p, --processes       Show all processes, but not threads.


On my desktop, LibraOffice Calc is starting to struggle when plotting large amounts of data. It might be better to process and the data further and plot it using a dedicated plotting tool - which is what I will describe next time.

The code


#!/usr/bin/env python
#
# Copyright (C) 2011: Michael Hamilton
# The code is LGPL (GNU Lesser General Public License) ( http://www.gnu.org/copyleft/lesser.html )
#

from __future__ import with_statement
import re
import os
import glob
import string
import pwd
import csv
import sys
import time
from optparse import OptionParser

PROC_FS_ROOT = '/proc'
INT_RE_SPEC = '[+-]*\d+'
INT_RE = re.compile(INT_RE_SPEC + '$')
CSV_LINE_TERMINATOR='\n'


# Default parser that deals with a multi-line file where each line 
# is a "tag: value" pair 
class _ProcBase(object):

    _split_re = re.compile(':\s+')

    def __init__(self, path=None, filename=None):
        self.error = None
        if path and filename:
            self.parseProcFs(path, filename)

    def parseProcFs(self, path, filename):
        pid = os.path.basename(path)
        self.pid = int(pid) if INT_RE.match(pid) else pid
        try:
            with open(path + '/' + filename) as proc_file:
                for line in proc_file.read().splitlines():
                    sort_key, value = _ProcBase._split_re.split(line)
                    self.__dict__[string.lower(sort_key)] = int(value) if INT_RE.match(value) else value
        except IOError as ioerr:
            self.handle_error('IOError %s/%s - %s' % (path, filename, ioerr))

    def handle_error(self, message):
        self.error = message
        print >> sys.stderr, self.error

    def keys(self):
        return sorted(self.__dict__.keys())

    def csv(self, file, header=True):
        if not self.error:
            if header:
                csv.writer(file, lineterminator=CSV_LINE_TERMINATOR).writerow(self.keys())
            csv.DictWriter(sys.stdout,  self.keys(), lineterminator=CSV_LINE_TERMINATOR).writerow(self.__dict__)

# Parser for space separated Values on one line- e.g. "12 comm 123456 111 a 12"
class _SpaceSeparatedParser(object):

    def __init__(self):
        self._keys = []
        self._re_spec = ''
        self._regexp = None

    def _add_item(self, sort_key, rexp_str):
        self._regexp = None
        self._keys.append(sort_key)
        if rexp_str:
            self._re_spec += rexp_str % sort_key
        return self
    def int_item(self, sort_key):
        return self._add_item(sort_key, '(?P<%s>' + INT_RE_SPEC + ')\s')
    def comm_item(self, sort_key):
        return self._add_item(sort_key, '[(](?P<%s>[^)]+)[)]\s')
    def string_item(self, sort_key):
        return self._add_item(sort_key, '(?P<%s>\w+)\s')
    def nonparsed_item(self, sort_key):  # Create property sort_key only, but don't parse it
        return self._add_item(sort_key, None)

    def keys(self):
        return self._keys;
    def parse(self, line):
        if not self._regexp:
            self._regexp = re.compile(self._re_spec)
        return self._regexp.match(line)

class ProcStat(_ProcBase):

    _parser = _SpaceSeparatedParser().\
        int_item('pid').\
        comm_item('comm').\
        string_item('state').\
        int_item('ppid').\
        int_item('pgrp').\
        int_item('session').\
        int_item('tty_nr').\
        int_item('tpgid').\
        int_item('flags').\
        int_item('minflt').\
        int_item('cminflt').\
        int_item('majflt').\
        int_item('cmajflt').\
        int_item('utime').\
        int_item('stime').\
        int_item('cutime').\
        int_item('cstime').\
        int_item('priority').\
        int_item('nice').\
        int_item('num_threads').\
        int_item('itrealvalue').\
        int_item('starttime').\
        int_item('vsize').\
        int_item('rss').\
        int_item('rlim').\
        int_item('startcode').\
        int_item('endcode').\
        int_item('startstack').\
        int_item('kstkesp').\
        int_item('kstkeip').\
        int_item('signal').\
        int_item('blocked').\
        int_item('sigignore').\
        int_item('sigcatch').\
        int_item('wchan').\
        int_item('nswap').\
        int_item('cnswap').\
        int_item('exit_signal').\
        int_item('processor').\
        int_item('rt_priority').\
        int_item('policy').\
        int_item('delayacct_blkio_ticks').\
        int_item('guest_time').\
        int_item('cguest_time').\
        nonparsed_item('error')

    def __init__(self, path):
        _ProcBase.__init__(self)
        if path:
            self.parseProcFs(path)

    def parseProcFs(self, path):
        path = path + '/stat'
        try:
            with open(path) as stat_file:
                for line in stat_file: # Only one line in file
                    if line and line != '':
                        self.parse(line)
                        self.error = None
                    else:
                        self.error = 'Empty line'
        except IOError as ioerr:
            self.handle_error('IOError %s - %s' % (path, ioerr))


    def parse(self, line):
        # Dynamically (at run time) add properties to this instance representing
        # each stat value.  E.g. add the pid value as a field called self.pid
        split_line = ProcStat._parser.parse(line);
        if split_line:
            # Update the properties of the Stat instance with integer or
            # string values as appropriate.
            for sort_key, value in split_line.groupdict().items():
                self.__dict__[sort_key] = int(value) if INT_RE.match(value) else value
        else:
            self.handle_error('Failed to match:' + line)

    def keys(self):
        return ProcStat._parser.keys()



class ProcStatus(_ProcBase):

    def __init__(self, path):
        _ProcBase.__init__(self, path, 'status')
        if not self.error:
            self.uid = [ int(uid) for uid in string.split(self.uid,'\t')]

class ProcIO(_ProcBase):

    def __init__(self, path):
        _ProcBase.__init__(self, path, 'io')


class ProcInfo(object):

    def __init__(self, path):
        self.time_stamp = time.time()
        self.meta = {}
        self.stat = ProcStat(path)
        self.status = ProcStatus(path)
        self.io = ProcIO(path)
        self.username = pwd.getpwuid(self.status.uid[0]).pw_name if not self.hasErrors() else 'nobody'
        self.pid = int(path.split('/')[-1])

    def hasErrors(self):
        return self.stat.error or self.status.error or self.io.error

def get_all_proc_data(include_threads=False, root=PROC_FS_ROOT):
    if include_threads:
        results = [ProcInfo(task_path) for task_path in glob.glob(root + '/[0-9]*/task/[0-9]*')]
    else:
        results = [ProcInfo(task_path) for task_path in glob.glob(root + '/[0-9]*')]
    return [info for info in results if not info.hasErrors()]

def get_proc_info(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcInfo(root + '/' + pid + ('task/' + threadid) if threadid else '')
def get_proc_stat(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcStat(root + '/' + pid + ('task/' + threadid) if threadid else '')
def get_proc_status(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcStatus(root + '/' + pid + ('task/' + threadid) if threadid else '')
def get_proc_io(pid, threadid=None, root=PROC_FS_ROOT):
    return ProcIO(root + '/' + pid + ('task/' + threadid) if threadid else '')

if __name__ == '__main__':

    usage = """usage: %prog [options] [pid...]
    Output CSV for procfs stat, status or io data for given thread/process pid's or
    for all processes and threads if no pid's are supplied."""
    parser = OptionParser(usage)
    parser.add_option('-s', '--stat', action='store_true', dest='do_stat', help='Output csv for pid/stat files.')
    parser.add_option('-S', '--status', action='store_true', dest='do_status', help='Output csv for pid/status files.')
    parser.add_option('-i', '--io', action='store_true', dest='do_io', help='Output csv for pid/io files.')
    parser.add_option('-t', '--titles', action='store_true', dest='output_titles', help='Output a title line.')
    parser.add_option('-r', '--repeat', action='store_true', dest='repeat', help='Repeat until interrupted.')
    parser.add_option('-w', '--sleep', type='int', dest='wait', default=5, help='Sleep seconds for each repetition.')
    parser.add_option('-p', '--processes', action='store_true', dest='processes_only', help='Show all processes, but not threads.')

    (options, args) = parser.parse_args()
    header = options.output_titles

    if len(args) == 0:
        args = [ '[0-9]*' ] # match all processes or threads
    elif options.processes_only:
        print >> sys.stderr, 'ignoring -p, showing requested processes and threads instead.'
        options.processes_only = False

    while True:
        for pid in args:
            for path in glob.glob(PROC_FS_ROOT + ('/' if options.processes_only else '/[0-9]*/task/') + pid):
                if options.do_stat or (not options.do_status and not options.do_io):
                    ProcStat(path).csv(sys.stdout, header=header)
                if options.do_status:
                    ProcStatus(path).csv(sys.stdout, header=header)
                if options.do_io:
                    ProcIO(path).csv(sys.stdout, header=header)
                header = False
        if not options.repeat:
            break
        time.sleep(options.wait)


Notes



Documentation for the Linux proc file-system can be found in the Linux proc (section 5) manual page (man 5 proc). The files I wanted to parse either contain several lines, with one value per line (status file and io file), or a single line, with multiple values per line (stat file). My linuxprocfs script contains some generalised code that should cope with basic parsing of both types of file and could be a basis for parsing other files in the procfs. The procfs is a prisoner of its history and suffers a bit from inconsistencies in it's syntax.

The linuxprocfs.py python script is coded for python 2.7 which includes all the dependent modules including the csv and options parsing modules.  I'm running an OpenSUSE 11.4 desktop, but I imagine the script will run on any of the modern Linux distributions.  The script may issue warnings to standard-error if processes disappear while it is traversing the procfs, these are normal and only diagnostic.

I have come across one other python interface to procfs, python-linux-procfs by Arnaldo Carvalho de Melo. It's source base is larger, and it decodes more details.  I will be looking into whether anything from my script is worth merging into this other version.

This post is dusting off some work I'd parked a couple of years back.  It's quite pleasant to return to python and its libraries - together they do more to close the gap between idea and implementation than any programming environment I've tired.

This blog page uses SyntaxHighlighter by Alex Gorbatchev.   You can easily copy and paste a line-number free version of  the code by selecting view-source icon in the mini-tool-bar that appears on the top right of the source listing (if javascript is enabled). If you click inside a source listing you will be able to use the arrow keys to scroll sideways.