Roll Your Own Performance Tools.. Real-time Graphing and Round Robin Data Storage

(Moving some of my older blog posts to this site for permanent archiving.. This entry was originally posted at testingReflections on 04/25/2006)

I have spent a lot of time playing around with graphics libraries and toolkits for integrating real-time graphs within my own performance testing and monitoring tools. It seems like there are many open source tools available in the world of performance testing and system monitoring. And lots of people roll their own tools in whatever programming language they are into... but many lack graphics capabilities.

Two of the toolkits/libraries I end up using often for my own homebrew test tools are: RRDTool, and JRobin.

from the RRDTool site:
"RRD is the Acronym for Round Robin Database. RRD is a system to store and display time-series data (i.e. network bandwidth, machine-room temperature, server load average). It stores the data in a very compact way that will not expand over time, and it can create beautiful graphs. It can be used via simple shell scripts or as a perl module."

So...
RRDTool is a really good back-end for storing time-series data; which is mostly what we care about in performance testing.  It has bindings for various scripting languages, or can be invoked from the command line. If you are developing tools that need a data repository and graphing capabilities, this provides you both. You create an RRD and then you begin inserting data values at regular intervals. You then call the graphing API to have a graph displayed. The cool thing about this data storage is its “round robin” nature. You define various time spans, and the granularity at which you want them stored. I fixed binary file is created, and this never grows in size over time. As you insert more data, it is inserted into each span. As results are collected, they are averaged and rolled into successive time spans. It makes a much more efficient system than using your own complex object structures, or a relational database, or file system storage.

You will probably recognize the graphs it creates, as RRDTool is integrated in many popular monitoring tools (it is Free/Open Source, GPL License). I have built many tools around RRDTool, and it is really a nice system.

If you are in the Java world, there is a very cool project named JRobin. JRobin is a clone of RRDTool in pure Java. So you can create RRD's directly from your Java code.. and all in memory if you want to!

Some days I pretend to be a Java programmer, so I had to build a tool using JRobin. As a proof of concept, I wrote a small network latency monitoring tool. It shows off some of JRobin's capabilities. It pings a host at a given interval and records the latency. A graph of the network latency is rendered in real-time onto a Swing panel.

Here is my network latency monitoring tool: NetPlot (includes Java source code, GPL Licensed)

The tool itself is just a trivial example, and really isn't the point. But you could easily adapt this code or create your own to develop real-time graphs of your own time-series data.