This script uses your system's ping utility to send an ICMP ECHO_REQUEST to a list of hosts or devices. It uses a separate thread to ping each host/device. This can be useful for measuring network latency and verifying hosts are alive.

Check out more info here: http://www.goldb.org/python_pinger.html

---

#!/usr/bin/env python

import re
from subprocess import Popen, PIPE
from threading import Thread


class Pinger(object):
    def __init__(self, hosts):
        for host in hosts:
            pa = PingAgent(host)
            pa.start()
        
class PingAgent(Thread):
    def __init__(self, host):
        Thread.__init__(self)        
        self.host = host

    def run(self):
        p = Popen('ping -n 1 ' + self.host, stdout=PIPE)
        m = re.search('Average = (.*)ms', p.stdout.read())
        if m: print 'Round Trip Time: %s ms -' % m.group(1), self.host
        else: print 'Error: Invalid Response -', self.host
              
                             
if __name__ == '__main__':
    hosts = [
        'www.pylot.org',
        'www.goldb.org',
        'www.google.com',
        'www.this_one_wont_work.com'
       ]
    Pinger(hosts)

---

Output:

Round Trip Time: 14 ms - www.yahoo.com
Round Trip Time: 17 ms - www.goldb.org
Round Trip Time: 30 ms - www.google.com
Round Trip Time: 82 ms - www.pylot.org
Error: Invalid Response - www.this_one_wont_work.com

---

Note: I only tested this on Windows. To run on other systems, it would only require a one-line change.

In the various testing forums and mailing lists I frequent, I see lots of questions asked about automated test tools.  A certain question about performance/load testing tools is asked over and over.  I would like to answer it here.

The question goes something like:
"Can I test my application with this performance tool?".

If you are asking this question, you really don't understand the nature of these tools and how they work.

There are many performance/load testing tools in the proprietary and open source worlds (LoadRunner, SilkPerformer, QALoad, JMeter, OpenSTA, etc).  Conceptually, they all work the same way.  Performance tools do not work by driving a GUI like many functional test tools do.  Instead, they are used to generate load against servers by simulating concurrent client requests at the protocol level.

So the key question you should be asking is: "Which protocols do the client and server use to communicate, and does the tool support these protocols?"  HTTP?  ODBC?  DCOM?  JMS?  So next time you want to ask: "Will the tool be able to test my AJAX/ASP.NET/JSP/PHP/etc application?"; the real question should go something like: "My application uses <protocol x>, does the tool support that protocol?".

Since you are working at the protocol level, the language/platform your system uses has no relevance. All of the technologies I just mentioned use HTTP for transport (layer 7), so any tool that supports HTTP can be used to test that system.

---

Interested in Web performance testing?
Check out my open source tool: Pylot

A performance engineer is a little bit of a jack-of-all-trades.  Rather than focusing on small technological niche, performance testers must have a very wide range of technical skills to understand the inner working of a complex system under test.

As far as skill go, Scott Barber has said that you need to be a "mid-level everything":

"Become a "Mid-Level Everything" – Developer, DBA, Network Admin, Systems Admin, Architect, Business Analyst, etc."

If you want to become proficient in analyzing system performance and scalability, there are many technical areas you should study.  Here are some skills I have found to be invaluable in my success as a performance engineer:

Performance Concepts:

• Methodology
• Load Generation Tools
• User/Workload Modeling
• Results Analysis (Latency, Throughput, Metrics)
• Bottleneck Detection
• Code Profiling
• Scalability
• Concurrency
• Charting/Graphing
• Statistics

Operating Systems and Servers:

• Monitoring (CPU/Network/Mem/Disk/etc)
• System Tuning
• Web/Application/Middleware Server Tuning
• System Administration
• Virtualization
• OS Concepts (CPU Scheduling, Memory Management, etc)

Database:

• SQL
• Stored Procedures
• Monitoring
• Tuning

Network:

• Topology
• Monitoring
• Load Balancing
• TCP/IP
• HTTP
• Packet Sniffing and Protocol Analysis
• Caching
• OSI Model

Programming:

• Proficiency in at least on general programming language. Preferably a dynamic scripting language (Python/Perl/Ruby/etc)
• Code/Algorithm Analysis

(Note: There are lots of "soft skills" a performance tester would need to be successful. This post focuses only on technical skills)

Pretty interesting...

via BetaNews article:

"Twice each year, the rankings of 500 of the world's supercomputers are assessed by the University of Mannheim in association with Berkeley National Laboratory and the University of Tennessee, Knoxville. Their figures are then sorted by tested clusters' maximal observed peak performance, in gigaflops."

"Intel-based processors walked away with one, if not two, lions' shares worth of the Top 500 list, with a staggering 354 total systems."

"460 of the Top 500 systems were running one flavor of Linux or another, including all of the Top 10."

I just came across a great site with lots of papers related to performance monitoring and tuning for Linux:

http://www.ufsdump.org

One paper I especially liked:

Extreme Linux Performance Monitoring And Tuning

"The purpose of this document is to describe how to monitor Linux operating systems for performance.  This paper examines how to interpret common Linux performance tool output.  After collecting this output, the paper describes how to make conclusions about performance bottlenecks."

Lots of great info!

The OpenSTA team has announced the release of version 1.4.4

OpenSTA is a distributed software testing architecture designed around CORBA.  The applications that make up the current OpenSTA toolset were designed to be used by performance testing practitioners for web load testing.

Info:
http://portal.opensta.org/index.php?name=News&file=article&sid=51

Download:
http://opensta.org/download.html

Congrats and thanks to Bernie Velivis, Daniel Sutcliffe, Jerome Delemarche for making this release possible.

This is a module for generating concurrent requests to an HTTP server.  Each thread makes HTTP GET requests to a single URL at the specified interval.  Threads are added over a given rampup time if you want to generate increasing load.  Response times are printed to STDOUT.  Can be used for cursory performance benchmarking or load testing a web resource.

load_generator.py module

sample usage:

#!/usr/bin/env python

from load_generator import LoadManager

lm = LoadManager()
lm.msg = ('www.example.com', '/')
lm.start(threads=5, interval=2, rampup=2)

"Open Source"

In the words of Inigo Montoya [The Princess Bride]:  "You keep using that word.  I do not think it means, what you think it means."

A few months back, Radview Software announced that they are releasing an open source version of WebLOAD, their web performance and load testing tool.  I was very excited about this and thought it was a fantastic move that would have a big impact in the test tool market.  I am a performance engineer and a huge Free/Open Source Software advocate, so I love to see companies in the space that interests me most come around to embrace openness.

In their press release, Radview stated:

"WebLOAD Open Source, licensed under the GNU Public License (GPL) version 2, is based on WebLOAD, the company's flagship product that is already deployed at 1,600 sites. Immediately available for free download and use, WebLOAD is a commercial-grade open source project with more than 250 engineering years of product development."

Ok cool.. they used the GPL and opened up the whole shebang.  Wow, this company actually "gets it"... right?

Umm.. not quite.

If you look through the source code that is available for WebLOAD Open Source, you will notice that only code for a small subset of the product is available.  In actuality, WebLOAD Open Source is a partially proprietary tool which is marketed as Open Source Software.  The software has significant limitations in functionality and scalability.  The source code which needs to be modified to remove these restrictions is not distributed.  So what we are left with is a crippled version of the tool.

In a recent post to the WebLOAD OS Forum, someone asked to see the source code for "proxynator", which is the recording feature in WebLOAD.

The response from the Forum Admin (Amir Shoval, a Radview employee) was this:

"Currently the source code for the proxynator is not available as part of the open source code of WebLOAD."

This is in direct contradiction to what their website states:

"WebLOAD Open Source introduces a unified script authoring environment for recording, editing and debugging."

When further probed about this, he stated the following:

WebLOAD Open Source is dual licensed:
  1. the WebLOAD Load Engine is totally open sourced and hence is licensed under the GPL
  2. but the complete WebLOAD is still licensed under a proprietary license, which grants free usage in WebLOAD Open Source.

wait.. wait.. WHAT?
I thought the press release said "licensed under the GNU Public License", and "WebLOAD Open Source is a fully functional, commercial-grade performance testing product"?  Nowhere on their website or marketing materials to they talk about this dual licensing and limited availability of source code.

Now, if we look at the End User License Agreement (EULA) that applies to WebLOAD Open Source, it gets worse:

2. License Restrictions. This License does not permit you or any third party to:
(i) modify, translate, reverse engineer, decompile, disassemble (except to the extent that this restriction is expressly prohibited by law) or otherwise attempt to discover the source code of all or any portion of the Software;
(ii) modify, translate or create derivative works of all or any portion of the Software;
(iii) copy the Software (other than a single copy solely for back-up or archival purposes);
(iv) rent, lease, sell, offer to sell, distribute, or otherwise transfer rights to the Software;

OK... so we have an "open source" product that is actually dual licensed, where a large portion of the toolset is proprietary.  And furthermore, by accepting the EULA, you give up all of your rights that were granted under the GPL.  Huh?

So... to reiterate, WebLOAD Open Source is *not* open source.  A subset of it is open source: the [crippled] load engine.  Contrary to what their press release and website says, it contains proprietary components that are released in binary form with no source code.  It is rather disappointing to see a company jump on the bandwagon of open source without respecting the freedom that is supposed to come with it.

In conclusion: Is WebLOAD Open Source currently Open Source?  No
Will WebLOAD Open Source actually become Open Source?  Well.. that's up to Radview

Hey, I'm all for Freedom.  I applaud Radview for any of the code they released under the GPL.  But lets be fair, if you want to call your product open source and reap any benefits that come along with that... you gotta walk the walk.

(Pylot is the open source web performance/load test tool that I am developing)

When a test run is finished, a report is automatically generated to summarize the test results. It includes various statistics and graphs on response times and throughput from the run. A sample of the results report can be seen here:

http://www.pylot.org/samples/results/results.html

Pylot also writes results to CSV files so you can import them into your favorite spreadsheet to crunch numbers, generate statistics, and create graphs. I have been working

I have also been working on the GUI. Here is the latest:

http://www.pylot.org/samples/ui/pylot_ui_screenshot_2007_08_20.png

---

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool
• Pylot - Dev Update #4 - Web Performance/Load Test Tool (New Name and Defining Test Cases)
• Pylot - Dev Update #5 - Web Performance/Load Test Tool (Graphs With MatPlotlib)

highscalability.com pointed out an article from Pingdom titled "What the Web’s most popular sites are running on", which shows the results from an infrastructure survey of 7 popular web "super sites" (TechCrunch, FeedBurner, iStockPhoto, YouSendIt, Meebo, Vimeo, Alexaholic).

There is some intriguing information in the survey, but one thing stood out.  Apparently, TechCrunch runs on 1 web server and 1 database server, with no server clustering:

I almost find this hard to believe.  They serve > 1 million unique visitors per month off of this?

We performance practioners love our graphs!  Visualizing data is helpful in analyzing performance results.  Sometimes a quick glance at a graph provides better understanding than a mound of raw or summarized data.  Pylot's Results Reporting feature creates graphs of response times (latency) and Throughput.

For the graphing toolkit, Pylot uses Matplotlib to produce fancy graphs like these:

Matplotlib allows you to graph data from Python. Here is a simple script that gives a glimpse of how a line/marker graph is created as a png image:

---

#!/usr/bin/env python

from pylab import *  # Matplotlib

def main():
    # sequence of data points to graph (x, y coordinates)
    points = [(1, 3), (2, 6), (3, 2), (4, 5)]
    graph(points)
   
   
def graph(points):
    fig = figure(figsize=(6, 2))  # image dimensions  
    ax = fig.add_subplot(111)
    ax.grid(True, color='#666666')
    xticks(size='x-small')
    yticks(size='x-small')
    x_seq = [item[0] for item in points] 
    y_seq = [item[1] for item in points] 
    ax.plot(x_seq, y_seq, 
        color='blue', linestyle='-', linewidth=1.0, marker='o', 
        markeredgecolor='blue', markerfacecolor='yellow', markersize=2.0)
    savefig('graph.png') 
    
    
if __name__ == '__main__':
    main()

---

The output looks like this:

---

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool
• Pylot - Dev Update #4 - Web Performance/Load Test Tool (New Name and Defining Test Cases)

I stumbled across this site yesterday:  highscalability.com

"We started High Scalability to help you build successful scalable websites. This site tries to bring together all the lore, art, science, practice, and experience of building scalable websites into one place so you can learn how to build your system with confidence. Hopefully this site will move you further and faster along the learning curve of success."

This is really cool site/blog about building scalable architectures.  It contains lots of good info and overviews of some of the larger distributed systems on the net.  Definitely a regular stop for me.

Awesome technical overview of Google's inner workings:

Google Architecture

"Google is the King of scalability. Everyone knows Google for their large, sophisticated, and fast searching, but they don't just shine in search. Their platform approach to building scalable applications allows them to roll out internet scale applications at an alarmingly high competition crushing rate. Their goal is always to build a higher performing higher scaling infrastructure to support their products. How do they do that?"

... a breakdown of Google's massive highly scalable architecture.
This is modern distributed computing at its finest.

PyLT has been renamed to Pylot (some sort of abbreviation for "Python Load Test")

I realized that having a pronounceable name for a piece of software is pretty important :)

So...
As I develop my load test tool, I need a way to define test cases.  Here is my first attempt:

---

What Is A Pylot Test Case?

You must declare your test cases in an XML file. This is the format that the test engine understands. Editing XML may seem natural to some people, but awkward to others. The nice thing about this structure is that it will be very easy to create a more friendly user interface [sometime in the future] for defining test cases.

A test case is defined using the following syntax. Only the URL element is required.

<case>
  <url>URL</url>
  <method>HTTP METHOD</method>
  <body>REQUEST BODY CONTENT</body>
  <add_header>ADDITIONAL HTTP HEADER</add_header>
  <verify>STRING OR REGULAR EXPRESSION</verify>
  <verify_negative>STRING OR REGULAR EXPRESSION</verify_negative>
</case>

Below is an example of the simplest possible test case file. It contains a single test case which will be executed continuously during the test run. The test case contains a URL for the service under test. Since no method or body defined, it will default to sending an HTTP GET to this resource. Since no verifications are defined, it will pass/fail the test case based on the HTTP status code it receives (pass if status is < 400).

<testcases>
  <case>
    <url>http://www.example.com/foo</url>
  </case>
</testcases>

---

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool

(Update: PyLT has been renamed to Pylot)

(PyLT is the open source web performance/load test tool that I am developing)

A quick update on PyLT development...

The load generating engine is looking pretty solid and seems to work really well so far. It uses threading for concurrency and seems to scale well (though I haven't put it through its paces enough yet).

The GUI is evolving more and starting to look like a real performance/load testing tool:

This is my first project using wxWidgets and wxPython.  I am finding it to be very powerful and relatively straight forward to design nice user interfaces.  However, this is a big jump for me.  The past few years I have mostly done web programming and work with distributed systems.  It took a bit to get my head back into traditional GUI application development and event-driven programming

More to come...

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool

(Update: PyLT has been renamed to Pylot)

(PyLT is the web performance/load test tool that I am developing)

A quick update on PyLT development...

This week I rewrote the GUI using wxPython.  It still needs a *lot* of work, but here is what it's starting to look like:

Related:
PyLT - Dev Update #1 - Web Performance/Load Test Tool
PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)

(Update: PyLT has been renamed to Pylot)

A quick update on PyLT development...

I have a working version of the guts of my tool (the multi-threaded load generator).  I have now started working on the user interface.  My initial idea was to use Tk for the GUI Toolkit.  I started developing a minimal GUI and quickly realized I need a Toolkit more powerful than Tk.

My original justification for using Tkinter (from blog comments):

"I will probably eventually move to a richer toolkit (like wxPython) if I take this thing far. For right now, Tk works. It comes distributed with core python, it's super fast and light, it's easy to use, and I know it pretty well. Though it looks like crap and is limited in many ways."

As of today I am rewriting the GUI with wxPython, which uses the wxWidgets Toolkit.  This should give me the ability to create a rich cross-platform UI for my tool.

[For posterity] Here is what the original prototype of the Tk UI looked like:

R.I.P. Tk... Hello wxWidgets

Related:
PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source) 

(Update: PyLT has been renamed to Pylot)

I have started development on a new web performance/load testing tool.  It is targeted at testing Web Services.

Here is some Q&A with myself:

---

You know you are reinventing the wheel, right?

Yes, I know.  There are already open source web load testing tools available (OpenSTA, JMeter, Grinder, WebLOAD, etc).  I have used all of these as well as proprietary tools for years.  I am a performance engineer and I feel like I need a tool set that I am intimately familiar with.  I need the ability to easily alter and tweak the tool at will.  I don't have the time, budget, or patience enough to wait on vendors when I need something.  I also want a tool that is fun to hack and adapt.  For this, I need to understand the code base deeply.

What language are you using?

Python.  The initial GUI uses Tk, but this may be changed down the road. I use Python's threading module for concurrency. If this doesn't scale well enough, I will be exploring other models of concurrency (perhaps generator based coroutines).

Why do you think you can write a tool like this?

I have worked in performance testing for nearly 10 years.  I have written many tools that work with various protocols to do distributed load generation and testing.  Creating a simple HTTP load generator is sort of my Hello World 2.0 for each language I try (I have written these from scratch in Python, Perl, Java, and C#).  This tool takes that basic concept and organizes it into a robust application.

Will it be Free and Open Source?

Of course!  Licensed under GNU GPL.

---

For an early look, check out the source repository at:  http://pylt.googlecode.com/svn/trunk

More details to come.

-Corey

This looks cool:  http://o2s.csail.mit.edu/o2s-wiki/multitask

"multitask allows Python programs to use generators (aka coroutines) to perform cooperative multitasking and asynchronous I/O. Applications written using multitask consist of a set of cooperating tasks that yield to a shared task manager whenever they perform a (potentially) blocking operation, such as I/O on a socket or getting data from a queue. The task manager temporarily suspends the task (allowing other tasks to run in the meantime) and then restarts it when the blocking operation is complete. Such an approach is suitable for applications that would otherwise have to use select() and/or multiple threads to achieve concurrency."

It is built on some of the new generator features in Python 2.5.  I wrote about this a few months back and actually tried to implement a version of a coroutine scheduler myself.  Glad to see someone packaged up a nice version I can use :)

A few months ago I deployed my StockQuote Google Gadget, which is used for retrieving stock quotes and daily price graphs.

Behind the gadget is a remote .NET/C# service I created which scrapes stock quotes and charts from Google Finance.

You can see it and play with the demo: cgoldberg.googlepages.com

- Add my gadget to your Google Personalized Homepage
- Add my gadget to your own web page

I have been logging usage stats; just to see how many people are using it and how many transactions it is doing. Stats have been collected for about 4 months:

12000 transactions per day and growing fast.. yikes.

Update: My StockQuote gadget is no longer in service.  I Received a takedown notice from Google Finance on 05/23/2007.   umm...  sorta saw that comin' :)

Programmers Need To Learn Statistics Or I Will Kill Them All

Zed Shaw:

"I have a major pet peeve that I need to confess. I go insane when I hear programmers talking about statistics like they know sh-t when it’s clearly obvious they do not. I’ve been studying it for years and years and still don’t think I know anything. This article is my call for all programmers to finally learn enough about statistics to at least know they don’t know sh-t."

I just got the RESTful Web Services book (Leonard Richardson & Sam Ruby, O'Reilly, 2007) in the mail today.  I've only read the beginning, but so far it is great.  In fact, it brings me back to when I first started working with the "programmable web".  I got into the programmable web back when the web was only a few years old.  I spent years doing performance/scalability testing and tuning for large Web 1.0 applications and bizarre custom Web API's (think huge financial services rushing to get online).  Building tools to run realistic workloads through a system involves writing custom clients to simulate real user/browser interaction.  This is pretty ugly stuff when you are dealing with an application that was designed with only humans in mind (AKA all).  It involves lots of HTTP protocol level work.. screen scraping.. protocol sniffing and analyzing.. requests.. header mangling.. cookie handling.. redirects.. authentication.. session information parsing.. etc, etc.

Application simulation is pretty messy work.  There is no simple API to hide behind; you had to figure out what the API was for yourself.  See.. *every* web application has an API.  Though it might have been designed by accident.  This allowed me to see first hand how developers and frameworks butchered the use of the "Web" as a platform.  Staring at naked HTTP let me see every little bit of the hairball underneath.  Alas, any standardization around web services (or the concept to be officially named) was far off.

A friend (bearded Perl hacker) let me borrow a book to show me how Perl can do this cool web stuff:  Web Client Programming with Perl (Clinton Wong, O'Reilly, 1997).  This book helped me build my first web clients to do application simulation and testing.  There wasn't a ton of documentation at the time to do this sort of thing, so i relied heavily on this book.

So now.. 10 years later..  the Web has changed..  it has morphed into *the* distributed platform..  it is becoming organized.

As I flip through Restful Web Services, it all just looks right..  REST looks right..   It is simple..  it is HTTP..  it is all the guts I already know.  It almost feels like a sequel to my old favorite:

I have traded Perl for Python as my preferred scripting language the past few years, but I am still building simulators, web clients, and virtual users. I am excited to work on some new stuff in this area.

InfoWorld article:

Three open source Web service testing tools get high marks

Rick Grehan of InfoWorld reviewed 3 popular open source tools for testing web services.  Rick is a contributing editor of the InfoWorld Test Center.  One of the tools he reviewed was WebInject (which I wrote).

"In this roundup, I examined three tools that purport to verify that your Web services do what they are supposed to do, that they resist graceless failure, and (in some cases) that they conduct themselves with efficiency. The tools are soapUI, TestMaker, and WebInject. All are open source, and are available for free download and incorporation into your next Web services project."

My tool (WebInject) scored pretty well in the comparison.

---

From the article:

WebInject

WebInject is a super-lightweight testing tool that can automate the testing of both Web services and Web applications. In fact, WebInject's ability to test XML/SOAP Web services appears to be a recent addition to the tool, as earlier versions could not readily handle the SOAP protocol.

Written in Perl, WebInject is primarily a command-line tool, though its author provides a thin Perl/Tk user interface that at least simplifies the execution of tests for those unwilling to spend too much time at the command prompt. If you're not familiar with Perl, don't panic. WebInject is built so that you can construct your tests without having to touch so much as a byte of Perl code.

WebInject is really an execution and reporting engine. Unlike the other tools, it has no IDE-style user interface, so tests must be written in an editor outside of the WebInject UI. This gives WebInject a less professional feel, but doesn't hamper the tool. I envision users of WebInject having directories filled with text files of various test “templates.” To add a new test case, the user just pops open his or her favorite editor, does some cutting, some pasting, and a bit of tweaking to alter the template to fit the specific circumstance, and ba-ding!, you've got a new test case.

...

In essence, a WebInject “project” is nothing more than an XML file filled with a set of elements strung one after the other. WebInject's simple structure lets you build tests with amazing rapidity. You must, however, have a moderately good understanding of the mechanics of SOAP protocols as well as a tool that lets you generate and capture HTTP/SOAP requests and responses. You'll need the requests to build the POST body and the responses so that you can create proper “verifypositive” and “verifynegative” regular expressions to check for success or failure. I used the Web Service Toolkit add-on for Eclipse to grab requests and responses for WebInject; once I had gotten the hang of it, I fell easily into the groove of building test cases.

---

Criteria	Score	Weight
Documentation	8	20%
Features	8	20%
Scalability	8	20%
Ease-of-use	8	15%
Portability	9	15%
Value	9	10%

Review Score:
Very Good 8.3

Cost:
Free download - open source

Platforms
Any platform that runs Perl or has a Perl interpreter installed

Bottom Line:
Much less feature-rich than the other tools, the lightweight WebInject nonetheless bolts out of the starting gate. If you need testing that will be off the ground and flying in minutes, reach for WebInject. On the other hand, it has far fewer capabilities than the other two products in this test, and unless you want to hack the Perl code, WebInject's feature set is pretty much what you install.

---

visit www.webInject.org
for more of my tools, visit: www.goldb.org

SlideAway - There is a world outside of Ruby on Rails:

"Who needs Oracle/Mysql when you have Mnesia, a free, distributed, in memory database ? The ability to store native Erlang structures out of the box is so liberating: suddenly the need for your object-database mapping layer almost vanishes (well, not 100% to be fairly honest, but a big chunk of it: no need to create a 1-to-n relationship or a n-to-n relationship and a mapping table in many simple cases)

Not to mention that Mnesia supports table replication and is fully distributed, with the ability to add new 'nodes' on the fly. All of this out of the box ! (did I mention it was free too ?) This makes scaling up almost a joke. Compare this to the usual nightmares (and cost) of trying to implement a distributed Mysql/Oracle."

Awesome.

I was recently interview by Joseph McAllister for his Sticky ToolLook newsletter.  Sticky ToolLook is an extension of StickyMinds.com and Better Software magazine.  I mostly talk about Performance testing and tools.

The article can be found here: http://www.stickyminds.com/stickytoollook/index.asp?cd=5/10/2007

Transcript:

---

A Word with the Wise:
Tools and System Performance with Corey Goldberg
by Joseph McAllister

Corey Goldberg is a Boston-based software engineer who focuses on performance engineering and tool development. He also contributes to open source projects and has developed some of his own, such as WebInject. I spoke with him earlier this year about his passion for the craft of software tools.

Joseph McAllister:
What makes you passionate about performance test tools?

Corey Goldberg:
I am passionate about system performance, so tools are an integral part of that. Performance is an interesting and diverse space. It touches technology in so many ways, and the skill set it requires allows me to be close to several different technical areas at once: development, testing, analysis, design, operations, etc.

The thinking becomes pervasive, though. For example, last night I was standing in line for movie tickets, and all I could think about was how they could improve the queuing system to get better sales throughput. I regularly have conversations with my colleagues about service performance and input/output contention at our local burrito joint.

JM:
Is there a particular type of performance tool that is more "fun" to use? Is there a type that tends to offer more results?

CG:
The various types of tools all work together to form your full tool set or test suite. They all have their own fun parts. Load generation can be complex, as it involves software development alongside workload modeling. But it is also the fun part where you get to slam load through a system and watch it react.

The most satisfying tools to build are analysis and monitoring tools, especially tools with real-time monitoring and graphing. This enables you to look inside your test runs or production system and actually see what is happening in real time. Complex data sets and metrics collected from deep within a system are transformed into informative graphs as things happen. That is pretty exciting to work with.

JM:
Is there a clear division between the commercial tools you've used and the tools you've written? Do you prefer one or the other?

CG:
Yes and no. First, for terminology, I like to think of things in terms of proprietary vs. free tools. Proprietary tools tend to force you into a certain pattern of use and often don't provide the flexibility to change or extend them in ways you might want.

Lately I have been building a lot of my own tools that work alongside some commercial tools. I recently developed a reporting and analysis suite that replaced the analytics in a commercial tool we were using.

Commercial tools also offer some rich features that are sometimes not feasible to re-create in a reasonable amount of time. So you have to remember that building your own tools is only worthwhile if it is cost effective.

JM:
Describe your open source test tool, WebInject.

CG: WebInject is a test tool that I developed in Perl that is used for functional testing of Web application/services and ad-hoc monitoring of HTTP response times. It can run as its own GUI application with real-time graphing capabilities or can be integrated as a plug-in with other tools.

I was doing this type of stuff in various scripts for years, so I packaged a lot of it together and made a more generic interface that can be used across a variety of projects. I thought others might be interested in it, so I setup a SourceForge project and released it in January 2004.

The basic concept is that you define test cases in XML files that are fed through WebInject and executed against your system under test. WebInject provides a basic harness/framework that includes HTTP transport, parsing, cookie handling, authentication, SSL, etc. It gives you real-time response timing as well as functional verification using regex-based content verification and HTTP status codes.

I spend all of my time on newer tools these days, but I still keep on top of WebInject enough to facilitate others' using it and posting patches/updates to it. Other test tools have progressed a lot in the past few years, so I am sure there are lots of new options for doing this type of testing. Oddly, WebInject has become somewhat entrenched in monitoring systems. Most of the users lately seem to be people running Nagios (an open source monitoring system) that need an intelligent Web plug-in/agent.

JM:
What is your favorite element of creating and distributing an open source software tool?

CG:
Community feedback feels really good. I like sharing and collaborating. The feedback is also tremendously useful in terms of discovering bugs and offering suggestions, advice, or even patches of working code. I care about my craft, and I realize the only way to advance is through open collaboration.

I am also pretty influenced by the free software movement and do some volunteer work with the GNU Project. I have some core beliefs about the ethics of software freedom. Creating and distributing my own GPL-licensed software is my own little way to help that cause.

I wrote a small tool that I have found useful.  It is a Python script that parses and analyzes web log files (in W3C Extended Log File Format).  It creates and HTML report with data and PNG images showing graphs of things like: request throughput, error rates, HTTP method distribution, content type distribution, time-series, etc.

Many log parsing/analysis tools exist, but I was looking for something more specific to Performance than something a webmaster would want to look at.

The script is pretty basic. It was very useful for my own needs, but others might want to modify it.  If anyone has good suggestions to add to it, I am willing to enhance it at some point (or just grab my code and hack it yourself if you know Python).

---

Project Home

• www.goldb.org/perflog

Features

• Produces metrics and graphs from web logs (W3C Extended Log File Format)
• Useful during performance testing and analysis
• Output is created in XHTML/CSS with embedded PNG images
• PerfLog is written in Python and uses Matplotlib for graphs and plotting

License

• GNU GPLv2 (always free)

Project Info

• Google Code - Project Home
• Subversion Trunk (Source Code)

Requirements

• Python 2.4+
• Matplotlib (requires Numeric or Numpy)

Platforms

• Cross-Platform.  PerfLog will run on any system that supports Python and Matplotlib.

OK, this is huge news: www.webload.org

The commercial performance/load test tool market is dominated by large proprietary commercial vendors (HP/Mercury, Borland/Segue, etc). Radview has a nice product called WebLOAD that competes in the space.

As of this morning, Radview announced they have released WebLOAD OS, an open source version of WebLOAD. It is full-on GPL licensed (no fake open source). I already browsed their source tree. They have a Subversion repository.. code is in C and C++,

The Open Source performance/load test tool market doesn't offer many choices. Currently the most popular tools are JMeter and OpenSTA. 

This will be exciting. I wonder how well Radview will deal with the community on this. Though if it's not good, GNU GPL certainly allows forking :)

more to come...

Wow..
CGI will run well on IIS
Rails will run well on IIS.

Rob Conery on running Ruby on Rails (or other CGI based platforms) on IIS:

"Rails works using CGI - basically an executable that gets run each time a request comes into a web site. Most of the frameworks out there do NOT support multi-threading, so each time a request comes in that requires anything dynamic, CGI is "instanced" and executed. If you have a lot of requests at once, this isn't really a good thing. Now some servers are built to mitigate this (Apache, Lighttpd, etc); IIS is not.

... I would imagine that in the next 6 months we'll see a great addition to IIS 6 and 7 for all the CGI-enabled platforms out there."

hmm.. good to hear. (seriously)
but damn... weren't we doing this 10 years ago with Perl/Apache? :)

(via reddit programming)

Very lengthy overview of performance and scalability issues for web systems by Ask Bjorne Hansen.  This presentation covers a vast range of information:

Real World Web Scalability  (warning large PDF)

The takeaway?
Create horizontally scalable distributed systems..  always.

A report about scalability of virtualization techniques:

SCALABILITY COMPARISON OF 4 HOST VIRTUALIZATION TOOLS (QUETIER B / NERI V / CAPPELLO F)

"Virtualization tools are becoming popular in the context of Grid Computing because they allow running multiple operating systems on a single host and provide a confined execution environment.  In several Grid projects, virtualization tools are envisioned to run many virtual machines per host.  This immediately raises the issue of virtualization scalability."

4 types of virtualization tools are discussed in the context of scalability:

• Processor Virtualization
• Kernel Replication
• Operating System Virtualization
• Resource Virtualization

(Note to self: buy this book when it comes out)

Steve Souders (Chief Performance Yahoo! at Yahoo) is writing a book for O'Reilly about web performance:

High Performance Web Sites

It's great to see Performance continue to gain exposure.

-Corey

I am in the process of tuning a large distributed system; using an F5 BIG-IP Load Balancer to distribute traffic.

Willy Tarreau has a very good overview of load balancing options:

Making applications scalable with Load Balancing

Tim Jones gives a brief overview of SMP and discusses working with the Linux kernel:

Linux and symmetric multiprocessing

Tim Jones:

"As processor frequencies reach their limits, a popular way to increase performance is simply to add more processors. In the early days, this meant adding more processors to the motherboard or clustering multiple independent computers together. Today, chip-level multiprocessing provides more CPUs on a single chip, permitting even greater performance due to reduced memory latency.

You'll find SMP systems not only in servers, but also desktops, particularly with the introduction of virtualization. Like most cutting-edge technologies, Linux provides support for SMP. The kernel does its part to optimize the load across the available CPUs (from threads to virtualized operating systems). All that's left is to ensure that the application can be sufficiently multi-threaded to exploit the power in SMP."

Dan Pritchett posted his excellent "How eBay Scales" presentation a few months back.

It is a great look into a real-world massive distributed system and the evolution of its scalable architecture.  One interesting thing to notice is that eBay is a transactionless environment (meaning it doesn't use Database Transactions).

I have always seen the data layer as the difficult part to scale.  Separating logic from data and working in a purely transactionless environment can mitigate this issue.

Martin fowler commented on this today:

"The rationale for not using transactions was that they harm performance at the sort of scale that eBay deals with. This effect is exacerbated by the fact that eBay heavily partitions its data into many, many physical databases. As a result using transactions would mean using distributed transactions, which is a common thing to be wary of.

This heavy partitioning, and the database's central role in performance issues, means that eBay doesn't use many other database facilities. Referential integrity and sorting are done in application code. There's hardly any triggers or stored procedures."

OpenSTA is a distributed performance testing architecture and tool.  Recently, on the opensta-users mailing list, Bernie Velivis (from Performax Inc) posted an excellent article about performance testing strategies.  I thought this information would be useful for all performance testers who are just learning the craft.  Rather than letting it sit in some arcane mailing list archive, I asked Bernie if I could re-publish it here.

enjoy.

-Corey

Bernie Velivis on Performance Testing Strategies:

---

PERFORMANCE TESTING STRATEGIES

The terminology used to discuss Performance testing in technical publications and support forums can be ambiguous or inconsistent. Hopefully this article will help participants in the OpenSTA user support forum by providing a common frame of reference for discussing tools, testing, and test results. It may also be helpful to those new to performance testing.

CAPACITY TESTING

If your goal is to determine the CAPACITY of the system under test, start by creating a "realistic" workload consisting of a mix of the most popular transactions plus those deemed critical or known to cause problems even when executed infrequently. Pick a manageable set of transactions to emulate (considering time, budget, and goals), determine the probability of executing each transaction, the work rate for the emulated users, and the "success criteria for performance metrics (i.e. response time limits, concurrent users, and throughput).

One way to implement this approach is to create a master script, assign it to each VU, and have it generate random numbers and then call other scripts which model the individual workload transactions based on a table of probabilities. The scripts should be modeled with think times consistent with the way your users interact with the system. This varies greatly from one application to another and unless you are mining log files from an application already in use, this is a somewhat subjective process. The best advice I can give in defining the workload is to get input from people who know how the application is (or will be) used, make conservative assumptions (but no so much so that the sum of all your conservative decisions is pathological), and balance the scope of the workload vs. time to complete the project. Another important consideration is the data demographics of the transactions and the size and contents of the database.

When its time to test, increase the number of emulated users and monitor how response times, server resource utilization (CPU, disk IO, network, and memory), and throughput (the rate of tasks completed system wide) vary with the increased load. You might construct a test that ramps up to a specific number of users, lets them run for a while, and then repeats as necessary. This way, you can observe the behavior of the system in various steady states under increasing load. Workloads containing transactions having a low probability of being executed and/or a disproportionately large impact on the performance of other transactions usually need to run longer to reach a steady state. If you can't get repeatable results, your steady state interval might be too small. As a rule of thumb I would suggest a minimum ramp up time equal to the duration of the longest running script and the steady state observation period at least twice as long as the ramp up period. I also tend to ignore response times and performance statistics gathered during the ramp up periods and focus instead on the data collected during the steady state periods.

That's a rough outline of one approach to capacity testing which in summary is an attempt to load up the system with VUs in a way that is indistinguishable from a "real users" in order to find the capacity limit. Pick the wrong workload however and you might miss something very important or end up solving problems that won’t exist in the real world.

The end game here is to increase load until response times become excessive at which point you have found the system’s capacity limit. This limit will be due to either a hardware or software bottleneck. If time and goals allow, analyze the performance metrics captured, do some tuning, improve code efficiency or concurrency, or add some hardware resources. Make one change at a time and repeat as necessary until you meet capacity goals, find the limits to the architecture, or run out of time (which happens more then most performance engineers would like).

SOAK TESTING

The same scripts created for capacity testing can also be used for SOAK TESTING where you load up the system close to its maximum capacity and let it run for hours, days, etc. This is a great way to spot stability problems that only occur after the system has been running a long time (memory leaks are a good example of things you might find).

FAILOVER TESTING

Get the system under test into a steady state and start failing components (servers, routers, etc) and observe how response times are effected during and after the failover and how long the system takes to transition back to steady state and you are on your way towards FAILOVER TESTING. (A gross simplification and again there is lots of good reading material out there on failover and high availability testing).

STRESS TESTING

If your goal is to determine where or how (not if) the system will fail under load, then you are doing STRESS TESTING. One way to do this is to comment out the think times and increase VUs until something (hopefully not your emulator!) breaks. This is one form of stress testing, a valuable aspect of performance testing, but not the same as capacity testing. How the VUs compare to "real users" may be irrelevant as you are trying to determine how the system behaves when pushed past its limits.

A report illustrating how these concepts were used to performance test a SOAP application using OpenSTA can be downloaded here:
SamplePerformaxPerformanceReport.pdf

Bernie Velivis
Principle Consultant and President, Performax Inc
www.iPerformax.com

---

I have used Nagios for several years, and it has served me well as an open source distributed monitoring system.

I just read about Zabbix, and I'm posting here so I won't forget to check it out.  Zabbix is GPL (v2) licensed and looks interesting.  I will post more once I get a chance to play with it.

Joe Duffy (from Microsoft) talking about the .NET 2.0 CLR:

"It's also worth noting that the threadpool throttles its creation of threads to 2/second once the count has exceeded the # of CPUs."

Yuck...  I don't like throttling like that behind the scenes.  It can make performance problems *very* hard to diagnose.

The many hats of a performance engineer:

• tester
• developer
• toolsmith
• dba
• sysadmin
• network/ops/tech
• system integrator
• architect
• statistician
• data visualizer
• sadist

---

Scott Barber just posted some info about: Patterns & Practices: Performance Testing Guidance, a new guide to Performance Testing over at Microsoft's Codeplex.

from Scott:

"I am involved in Microsoft's Patterns & Practices Performance Testing Guidance project. We have reached a critical mass with regards to our "mostly final" content and have made that content publicly available"

"We're tackling various flavors of performance testing (stress, load, capacity) as well as how to bake performance testing into your life cycle."

from the Patterns & Practices site:

"The purpose of this project is to build some insightful and practical guidance around doing performance testing and using Visual Studio 2005. It's a collaborative effort between industry experts, Microsoft ACE, patterns & practices, Premier, and VSTS team members."

I have always had ambivalent feelings towards MS, but it is great to see all of the effort they are putting into the Performance field. Performance has always been a sort of niche domain. It straddles the disciplines of development, testing, and operations, while also involving the integration of software, hardware, and networks. The past few years have provided much more thought leadership that is pushing the state of Performance (load, stress, scalability, capacity, availability) into more mature territory.

For those that don't know Scott Barber, he certainly gets my vote for the most prolific writer in Performance over the past several years. His body of writing is second to none: http://www.perftestplus.com/pubs.htm

Psyco is a Python extension module which can speed up the execution of any Python code.

from the Psyco site:

"Think of Psyco as a kind of just-in-time (JIT) compiler, a little bit like what exists for other languages, that emit machine code on the fly instead of interpreting your Python program step by step. The difference with the traditional approach to JIT compilers is that Psyco writes several version of the same blocks (a block is a bit of a function), which are optimized by being specialized to some kinds of variables (a "kind" can mean a type, but it is more general). The result is that your unmodified Python programs run faster"

I have been working on some Python projects recently where Pysco has given me a a really substantial performance increase. The type of work I am doing mostly involves statistcal analysis of large numerical data sets.. array math.. percentiles.. time-series.. etc, etc.

To use it, all I do is copy Psyco to my system (to Python's Lib/site-packages), and add the following to the top of my python source file:

Thats all ...

... or even better; wrap it in a try/except so your program still runs on systems without Psyco installed.

Alex from Dojo just linked to a fascinating article about by Russ Cox about Regular Expressions:  Regular Expression Matching Can Be Simple And Fast (but is slow in Java, Perl, PHP, Python, Ruby, ...)

from the article:  

"This article reviews the good theory: regular expressions, finite automata, and a regular expression search algorithm invented by Ken Thompson in the mid-1960s. It also puts the theory into practice, describing a simple implementation of Thompson's algorithm. That implementation, less than 400 lines of C, is the one that went head to head with Perl above. It outperforms the more complex real-world implementations used by Perl, Python, PCRE, and others. The article concludes with a discussion of how theory might yet be converted into practice in the real-world implementations."

so.. there is a 40 year old technique that improves performance of regexes dramatically?

The following graph plots time required to check whether a?^na^n matches a^n:



wow... so awk and grep use the Thomson NFA implementation of regexes, while most programming languages don't.  

... and here I thought Perl was the regex king.

(Moving some of my older blog posts to this site for permanent archiving.. This entry was originally posted at testingReflections on 04/25/2006)

I have spent a lot of time playing around with graphics libraries and toolkits for integrating real-time graphs within my own performance testing and monitoring tools. It seems like there are many open source tools available in the world of performance testing and system monitoring. And lots of people roll their own tools in whatever programming language they are into... but many lack graphics capabilities.

Two of the toolkits/libraries I end up using often for my own homebrew test tools are: RRDTool, and JRobin.

from the RRDTool site:
"RRD is the Acronym for Round Robin Database. RRD is a system to store and display time-series data (i.e. network bandwidth, machine-room temperature, server load average). It stores the data in a very compact way that will not expand over time, and it can create beautiful graphs. It can be used via simple shell scripts or as a perl module."

So...
RRDTool is a really good back-end for storing time-series data; which is mostly what we care about in performance testing.  It has bindings for various scripting languages, or can be invoked from the command line. If you are developing tools that need a data repository and graphing capabilities, this provides you both. You create an RRD and then you begin inserting data values at regular intervals. You then call the graphing API to have a graph displayed. The cool thing about this data storage is its “round robin” nature. You define various time spans, and the granularity at which you want them stored. I fixed binary file is created, and this never grows in size over time. As you insert more data, it is inserted into each span. As results are collected, they are averaged and rolled into successive time spans. It makes a much more efficient system than using your own complex object structures, or a relational database, or file system storage.

You will probably recognize the graphs it creates, as RRDTool is integrated in many popular monitoring tools (it is Free/Open Source, GPL License). I have built many tools around RRDTool, and it is really a nice system.

If you are in the Java world, there is a very cool project named JRobin. JRobin is a clone of RRDTool in pure Java. So you can create RRD's directly from your Java code.. and all in memory if you want to!

Some days I pretend to be a Java programmer, so I had to build a tool using JRobin. As a proof of concept, I wrote a small network latency monitoring tool. It shows off some of JRobin's capabilities. It pings a host at a given interval and records the latency. A graph of the network latency is rendered in real-time onto a Swing panel.

Here is my network latency monitoring tool: NetPlot (includes Java source code, GPL Licensed)

The tool itself is just a trivial example, and really isn't the point. But you could easily adapt this code or create your own to develop real-time graphs of your own time-series data.

RASP:

Reliability. Availability. Scalability. Performance.

System performance has been one of my main interests since I got into computing.  My first memories of this go back to my DOS 5.0 days ('92?).  I would obsessively optimize my beige-box x86 desktop... staying up all night tweaking my config.sys and running fractals.  (anyone remember all the fiddling with device drivers and Terminate and Stay Resident programs to load them into Upper Memory Blocks, so you can free up conventional memory?)  That obsession grew as I gained more knowledge and eventually led me into testing/tuning large distributed systems, which I have essentially focused on professionally for the past 9 years  (hey, my job title even includes "Performance Engineer").

anyway,

I just wanted to re-introduce the acronym RASP.  In the performance world, there is a lack of standard vocabulary.  There is enough shared terminology to have intelligent conversations and get engineering problems solved, but the terms used vary pretty widely from company to company and region to region.. much more so than pure development or admin language.  So sharing vocabulary is a good thing, to further push performance as its own discipline.  

I first heard the acronym "RASP" from Goranka Bjedov, a performance engineer at Google (shown here in her excellent tech-talk Using Open Source Tools for Performance Testing), during discussions at WOPR6.  She said it was an old Bell Labs term used in telco.  

RASP encapsulates all things related to system performance into a nice logical taxonomy.

Reliability
Availability
Scalability
Performance

For some reason RASP has really stuck in my head... just passing the word.