Monday, April 14, 2008

Pylot 1.1 - New Release With Test Case Recorder

New Pylot 1.1 release is available
Visit: www.pylot.org/download.html

It contains some minor code cleanup and a new test case recorder contributed by David Solomon. The recorder works with Windows and IE only.

It is a script that launches your web browser and records HTTP requests as you navigate. While it records, it prints Pylot's XML test cases. The test cases are printed to STDOUT, so just redirect it to a file and you will have a valid testcases.xml file to use as Pylot input.

The pylot_recorder script is included in the lib directory of Pylot 1.1.

View the recorder's source code from the SVN trunk:
http://code.google.com/p/pylt/source/browse/trunk/lib/pylot_recorder.py

(It can't handle some complex scenarios, but is useful for recording simple GET and POST requests from web applications)

Comments [4] |

Wednesday, March 19, 2008

Pylot Dev Update - Web Performance - Release 1.0

Finally did the version 1.0 release! visit www.pylot.org to download.

Pylot is still lacking some features I want to add for it to become a serious performance/load testing tool, but the current release delivers very usable functionality.

Current Features:

multi-threaded load generator
HTTP and HTTPS (SSL) support
response verification with regular expressions
execution/monitoring console
real-time stats
results reports with graphs
GUI mode
shell/console mode
cross-platform

Aside from the GUI, there is also a new shell/console interface mode with real-time output for quickly profiling performance your application/service under test from the command line. In this mode, Pylot can run cross-platform. (tested on Windows XP, Vista, Cygwin, Ubuntu, MacOS)

Note: Extra special thanks to Vasil Vangelovski for implementing the original console output and C++ extension

Screenshots of the GUI and new shell/console UI output:

Pylot | Python | Testing | Web Services

Comments [0] |

Wednesday, March 05, 2008

Pylot - Web Performance Tool - 0.2 Beta Release

I just quietly did a release of Pylot (my open source web performance tool). You can grab it here.

Pylot

Comments [2] |

Friday, August 24, 2007

Pylot - Dev Update #6 - Web Performance/Load Test Tool (Results Report and GUI)

(Pylot is the open source web performance/load test tool that I am developing)

When a test run is finished, a report is automatically generated to summarize the test results. It includes various statistics and graphs on response times and throughput from the run. A sample of the results report can be seen here:

http://www.pylot.org/samples/results/results.html

Pylot also writes results to CSV files so you can import them into your favorite spreadsheet to crunch numbers, generate statistics, and create graphs. I have been working

I have also been working on the GUI. Here is the latest:

http://www.pylot.org/samples/ui/pylot_ui_screenshot_2007_08_20.png

Comments [4] |

Wednesday, August 08, 2007

Pylot - Dev Update #5 - Web Performance/Load Test Tool (Graphs With MatPlotlib)

We performance practioners love our graphs! Visualizing data is helpful in analyzing performance results. Sometimes a quick glance at a graph provides better understanding than a mound of raw or summarized data. Pylot's Results Reporting feature creates graphs of response times (latency) and Throughput.

For the graphing toolkit, Pylot uses Matplotlib to produce fancy graphs like these:

Matplotlib allows you to graph data from Python. Here is a simple script that gives a glimpse of how a line/marker graph is created as a png image:

#!/usr/bin/env python

from pylab import *  # Matplotlib

def main():
    # sequence of data points to graph (x, y coordinates)
    points = [(1, 3), (2, 6), (3, 2), (4, 5)]
    graph(points)
   
   
def graph(points):
    fig = figure(figsize=(6, 2))  # image dimensions  
    ax = fig.add_subplot(111)
    ax.grid(True, color='#666666')
    xticks(size='x-small')
    yticks(size='x-small')
    x_seq = [item[0] for item in points] 
    y_seq = [item[1] for item in points] 
    ax.plot(x_seq, y_seq, 
        color='blue', linestyle='-', linewidth=1.0, marker='o', 
        markeredgecolor='blue', markerfacecolor='yellow', markersize=2.0)
    savefig('graph.png') 
    
    
if __name__ == '__main__':
    main()

The output looks like this:

Comments [0] |

Tuesday, July 24, 2007

Pylot - Dev Update #4 - Web Performance/Load Test Tool (New Name and Defining Test Cases)

PyLT has been renamed to Pylot (some sort of abbreviation for "Python Load Test")

I realized that having a pronounceable name for a piece of software is pretty important :)

So...
As I develop my load test tool, I need a way to define test cases. Here is my first attempt:

What Is A Pylot Test Case?

You must declare your test cases in an XML file. This is the format that the test engine understands. Editing XML may seem natural to some people, but awkward to others. The nice thing about this structure is that it will be very easy to create a more friendly user interface [sometime in the future] for defining test cases.

A test case is defined using the following syntax. Only the URL element is required.

<case>
  <url>URL</url>
  <method>HTTP METHOD</method>
  <body>REQUEST BODY CONTENT</body>
  <add_header>ADDITIONAL HTTP HEADER</add_header>
  <verify>STRING OR REGULAR EXPRESSION</verify>
  <verify_negative>STRING OR REGULAR EXPRESSION</verify_negative>
</case>

Below is an example of the simplest possible test case file. It contains a single test case which will be executed continuously during the test run. The test case contains a URL for the service under test. Since no method or body defined, it will default to sending an HTTP GET to this resource. Since no verifications are defined, it will pass/fail the test case based on the HTTP status code it receives (pass if status is < 400).

<testcases>
  <case>
    <url>http://www.example.com/foo</url>
  </case>
</testcases>

Comments [0] |

Friday, June 29, 2007

PyLT - Dev Update #3 - Web Performance/Load Test Tool

(Update: PyLT has been renamed to Pylot)

(PyLT is the open source web performance/load test tool that I am developing)

A quick update on PyLT development...

The load generating engine is looking pretty solid and seems to work really well so far. It uses threading for concurrency and seems to scale well (though I haven't put it through its paces enough yet).

The GUI is evolving more and starting to look like a real performance/load testing tool:

This is my first project using wxWidgets and wxPython. I am finding it to be very powerful and relatively straight forward to design nice user interfaces. However, this is a big jump for me. The past few years I have mostly done web programming and work with distributed systems. It took a bit to get my head back into traditional GUI application development and event-driven programming

More to come...

Comments [0] |

Friday, June 15, 2007

PyLT - Dev Update #2 - Web Performance/Load Test Tool

(Update: PyLT has been renamed to Pylot)

(PyLT is the web performance/load test tool that I am developing)

A quick update on PyLT development...

This week I rewrote the GUI using wxPython. It still needs a *lot* of work, but here is what it's starting to look like:

Comments [0] |

Monday, June 11, 2007

PyLT - Dev Update #1 - Web Performance/Load Test Tool

(Update: PyLT has been renamed to Pylot)

A quick update on PyLT development...

I have a working version of the guts of my tool (the multi-threaded load generator). I have now started working on the user interface. My initial idea was to use Tk for the GUI Toolkit. I started developing a minimal GUI and quickly realized I need a Toolkit more powerful than Tk.

My original justification for using Tkinter (from blog comments):

"I will probably eventually move to a richer toolkit (like wxPython) if I take this thing far. For right now, Tk works. It comes distributed with core python, it's super fast and light, it's easy to use, and I know it pretty well. Though it looks like crap and is limited in many ways."

As of today I am rewriting the GUI with wxPython, which uses the wxWidgets Toolkit. This should give me the ability to create a rich cross-platform UI for my tool.

[For posterity] Here is what the original prototype of the Tk UI looked like:

R.I.P. Tk... Hello wxWidgets

Comments [2] |

Friday, June 01, 2007

PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)

(Update: PyLT has been renamed to Pylot)

I have started development on a new web performance/load testing tool. It is targeted at testing Web Services.

Here is some Q&A with myself:

You know you are reinventing the wheel, right?

Yes, I know. There are already open source web load testing tools available (OpenSTA, JMeter, Grinder, WebLOAD, etc). I have used all of these as well as proprietary tools for years. I am a performance engineer and I feel like I need a tool set that I am intimately familiar with. I need the ability to easily alter and tweak the tool at will. I don't have the time, budget, or patience enough to wait on vendors when I need something. I also want a tool that is fun to hack and adapt. For this, I need to understand the code base deeply.

What language are you using?

Python. The initial GUI uses Tk, but this may be changed down the road. I use Python's threading module for concurrency. If this doesn't scale well enough, I will be exploring other models of concurrency (perhaps generator based coroutines).

Why do you think you can write a tool like this?

I have worked in performance testing for nearly 10 years. I have written many tools that work with various protocols to do distributed load generation and testing. Creating a simple HTTP load generator is sort of my Hello World 2.0 for each language I try (I have written these from scratch in Python, Perl, Java, and C#). This tool takes that basic concept and organizes it into a robust application.

Will it be Free and Open Source?

Of course! Licensed under GNU GPL.

For an early look, check out the source repository at: http://pylt.googlecode.com/svn/trunk

More details to come.

-Corey

Comments [6] |