Jesse Noller just blogged "Finding Python people is hard":

Here is a good quote regarding the difficulty in finding skilled Test Engineers with Python experience:

"Either you teach QA people automation/test engineering, or you try to find a programmer who wants to learn/do test engineering and teaching them python. It's a hard sell either way. I technically view QA as one discipline, Development as another, but Test Engineering as the Hybrid of the two - and you need a strong background in both."

I have seen lots of QA Engineers and Testers with little to no development/programming experience. This seems to be such a valuable skill; why not learn some? The bar is set really low with today's dynamic languages. Getting into some quick scripting for data manipulation and building test harnesses is not a huge task. If a QA engineer can't learn some simple programming in a week, would you trust his efficiency and technical skills?

I agree with Jesse on this one. We need to see more Test Engineers and Developers In Test. Unfortunately, this hybrid roll often falls through cracks as many people view quality/testing vs. developing as a binary choice.

Over the last several years, we have seen many big software players like IBM, HP, and Borland, acquire most of the larger automation tool vendors (Rational, Mercury, Segue, respectively).

Oracle just followed suit with their announcement to buy acquire the e-TEST suite products from Empirix:

http://www.oracle.com/empirix/index.html

I guess it was just a matter of time.  It will be interesting to see where all of this is going.

In the various testing forums and mailing lists I frequent, I see lots of questions asked about automated test tools.  A certain question about performance/load testing tools is asked over and over.  I would like to answer it here.

The question goes something like:
"Can I test my application with this performance tool?".

If you are asking this question, you really don't understand the nature of these tools and how they work.

There are many performance/load testing tools in the proprietary and open source worlds (LoadRunner, SilkPerformer, QALoad, JMeter, OpenSTA, etc).  Conceptually, they all work the same way.  Performance tools do not work by driving a GUI like many functional test tools do.  Instead, they are used to generate load against servers by simulating concurrent client requests at the protocol level.

So the key question you should be asking is: "Which protocols do the client and server use to communicate, and does the tool support these protocols?"  HTTP?  ODBC?  DCOM?  JMS?  So next time you want to ask: "Will the tool be able to test my AJAX/ASP.NET/JSP/PHP/etc application?"; the real question should go something like: "My application uses <protocol x>, does the tool support that protocol?".

Since you are working at the protocol level, the language/platform your system uses has no relevance. All of the technologies I just mentioned use HTTP for transport (layer 7), so any tool that supports HTTP can be used to test that system.

---

Interested in Web performance testing?
Check out my open source tool: Pylot

Nice testing article from Titus Brown.  He makes some good points about automated tests:

The (Lack of) Testing Death Spiral

"So, automated tests are important for maintenance, and they are critical for making sure that your old code still works while you focus on new code. Without automated tests, you will be doomed to releasing increasingly buggy software as your body of code increases and the average level of testing decreases."

Finally did the version 1.0 release! visit www.pylot.org to download.

Pylot is still lacking some features I want to add for it to become a serious performance/load testing tool, but the current release delivers very usable functionality.

Current Features:

• multi-threaded load generator
• HTTP and HTTPS (SSL) support
• response verification with regular expressions
• execution/monitoring console
• real-time stats
• results reports with graphs
• GUI mode
• shell/console mode
• cross-platform

Aside from the GUI, there is also a new shell/console interface mode with real-time output for quickly profiling performance your application/service under test from the command line. In this mode, Pylot can run cross-platform. (tested on Windows XP, Vista, Cygwin, Ubuntu, MacOS)

Note: Extra special thanks to Vasil Vangelovski for implementing the original console output and C++ extension

---

Screenshots of the GUI and new shell/console UI output:

Programmatic Testing:  A style of software testing where tests are executed without human interaction.

When I say "this test can be automated", I don't mean a computer can magically replace a sapient process to create a useful test.  All test design and methodology is devised using a sapient processes (obviously).  However, testcase generation, data generation, test execution, results analysis, etc, can often be done without human interaction.  This has been described using terms like: "Automated Testing, "Test Automation", etc.  Among some of the more vocal people in the testing community, these terms are considered confusing,  inaccurate, harmful, or just a cover up for what is actually a non-scripted test

Automated and Manual Testing are not mutually exclusive.  Rather, they are at either end of a continuum.  A test isn't "automated" or "manual".  It falls somewhere on that continuum.  Each step of the process has some mixture of the elements.

I have also heard the term: "Computer Assisted Testing".  I don't like this term because it tries to place a name on the entire testing process.  I am not referring to the entire process, so this becomes confusing talk to me.  I am referring to a style of test execution.

Therefore, to clear up ambiguity of terminology, I now refer to "Automation" as "Programmatic Testing".  This refers to a style of test execution where tests are executed in a non-interactive manner.

I you have a tool, a test framework, a software library, a program/script, functions for checking conditions, or a mechanism for reporting errors and stats, then you are doing a form of programmatic testing.

A performance engineer is a little bit of a jack-of-all-trades.  Rather than focusing on small technological niche, performance testers must have a very wide range of technical skills to understand the inner working of a complex system under test.

As far as skill go, Scott Barber has said that you need to be a "mid-level everything":

"Become a "Mid-Level Everything" – Developer, DBA, Network Admin, Systems Admin, Architect, Business Analyst, etc."

If you want to become proficient in analyzing system performance and scalability, there are many technical areas you should study.  Here are some skills I have found to be invaluable in my success as a performance engineer:

Performance Concepts:

• Methodology
• Load Generation Tools
• User/Workload Modeling
• Results Analysis (Latency, Throughput, Metrics)
• Bottleneck Detection
• Code Profiling
• Scalability
• Concurrency
• Charting/Graphing
• Statistics

Operating Systems and Servers:

• Monitoring (CPU/Network/Mem/Disk/etc)
• System Tuning
• Web/Application/Middleware Server Tuning
• System Administration
• Virtualization
• OS Concepts (CPU Scheduling, Memory Management, etc)

Database:

• SQL
• Stored Procedures
• Monitoring
• Tuning

Network:

• Topology
• Monitoring
• Load Balancing
• TCP/IP
• HTTP
• Packet Sniffing and Protocol Analysis
• Caching
• OSI Model

Programming:

• Proficiency in at least on general programming language. Preferably a dynamic scripting language (Python/Perl/Ruby/etc)
• Code/Algorithm Analysis

(Note: There are lots of "soft skills" a performance tester would need to be successful. This post focuses only on technical skills)

Open Source Testing is a great resource that lists most of the open source tools available to testers.  The site is run by Mark Aberdour and has been around since 2003.

Mark's contributions to the testing and open source communities have been very valuable.

Well... he just stepped it up a notch by posting details of his new Community Donations Program:

"during 2007 Open Source Testing has begun to generate fairly regular revenue. It has always been my aim, should the site become commercially viable, to put some profits back into the open source community. I will be aiming to make bi-monthly donations (funds providing) to open source testing projects and open source organisations of my own choosing. The donations will not be earth shattering, but whether they cover hosting and hardware costs, contractor costs, publicity, trips to events or just some extra motivation, they will certainly make a difference."

Great work Mark!

The OpenSTA team has announced the release of version 1.4.4

OpenSTA is a distributed software testing architecture designed around CORBA.  The applications that make up the current OpenSTA toolset were designed to be used by performance testing practitioners for web load testing.

Info:
http://portal.opensta.org/index.php?name=News&file=article&sid=51

Download:
http://opensta.org/download.html

Congrats and thanks to Bernie Velivis, Daniel Sutcliffe, Jerome Delemarche for making this release possible.

This is a module for generating concurrent requests to an HTTP server.  Each thread makes HTTP GET requests to a single URL at the specified interval.  Threads are added over a given rampup time if you want to generate increasing load.  Response times are printed to STDOUT.  Can be used for cursory performance benchmarking or load testing a web resource.

load_generator.py module

sample usage:

#!/usr/bin/env python

from load_generator import LoadManager

lm = LoadManager()
lm.msg = ('www.example.com', '/')
lm.start(threads=5, interval=2, rampup=2)

I was tipped off about this report in a recent forum discussion.

Report from Forrester Research:
    Evaluating Functional Testing Solutions (Powerpoint)

It is a nice overview of automated test execution and popular functional testing tools. It also gives an overview of the top tool vendors.

My first programming course was as an undergrad freshman in 1993. It was the basic introductory programming class for CS majors. The course was pretty difficult and was a great filter to separate the real CS students from the wannabes. About one-third of the students dropped the class, and out of the remaining two-thirds, many changed majors after this course was complete.

The course was taught using Scheme, with SICP as the text book. We programmed on a VAX cluster with Ultrix (DEC's Unix flavor) as the Operating System. We had to learn the Unix shells, VI, and all sorts of fun stuff to get us up and running.

The computer lab ("the cluster") was a large sterile room with rows of green-screen dumb terminals. I remember our professor told us that the VAX had 128 MB of memory and I was blown away by how huge that was (my rippin' fast PC had 4 MB at the time).

Spending hours in the computer lab was no fun at all. But I was one of the lucky ones. I owned a brand new 486-DX33 PC running Windows 3.1. I had a blazing fast 14.4 bps Zoom modem and could use Procomm Plus to dial into the VAX and program from the comfort of my own dorm room. I also found a Scheme interpreter that ran on DOS, giving me further options to do my work offline.

The programming assignments were brutal. All-nighters were the norm. Collaboration on the assignments was encouraged, but we were all expected to turn in our own original work. I found a fellow student that I got along with well and we decided to work together (unfortunately, I don't even remember his name... all I remember is that he was a lot smarter than me).

So the basic workflow was that we would get together, work out the basics of the assignment, get most of the algorithms working, then each take the code and finish it on our own. Since I had the bad-ass PC, we would work in my room. Two things quickly became apparent: He was a much better programmer than me, but I had a better eye for subtle details and debugging. Eventually we settled into a pattern where he would do the programming and I would look over his shoulder to give advice and input. Every few minutes, he would shoot a copy of the code to my dot-matrix printer. I would grab it, go through it line by line, and mark errors with my red pen and hand-write parts of the code that weren't correct. I would then hand the printout back to him and let him enter the changes. We iterated like this until we had all of the core code working.

For some reason, that instinct for attention to detail and debugging has always stuck with me. Because of that, my career has always been influenced by testing. I am a develop/tester, rather than just a developer. Most of the impact I have had in all of my jobs is from creating test tools and bringing in new ways to test software.

Just an interesting observation. I wonder how many others were naturally drawn to testing as soon as they started writing code?

"Open Source"

In the words of Inigo Montoya [The Princess Bride]:  "You keep using that word.  I do not think it means, what you think it means."

A few months back, Radview Software announced that they are releasing an open source version of WebLOAD, their web performance and load testing tool.  I was very excited about this and thought it was a fantastic move that would have a big impact in the test tool market.  I am a performance engineer and a huge Free/Open Source Software advocate, so I love to see companies in the space that interests me most come around to embrace openness.

In their press release, Radview stated:

"WebLOAD Open Source, licensed under the GNU Public License (GPL) version 2, is based on WebLOAD, the company's flagship product that is already deployed at 1,600 sites. Immediately available for free download and use, WebLOAD is a commercial-grade open source project with more than 250 engineering years of product development."

Ok cool.. they used the GPL and opened up the whole shebang.  Wow, this company actually "gets it"... right?

Umm.. not quite.

If you look through the source code that is available for WebLOAD Open Source, you will notice that only code for a small subset of the product is available.  In actuality, WebLOAD Open Source is a partially proprietary tool which is marketed as Open Source Software.  The software has significant limitations in functionality and scalability.  The source code which needs to be modified to remove these restrictions is not distributed.  So what we are left with is a crippled version of the tool.

In a recent post to the WebLOAD OS Forum, someone asked to see the source code for "proxynator", which is the recording feature in WebLOAD.

The response from the Forum Admin (Amir Shoval, a Radview employee) was this:

"Currently the source code for the proxynator is not available as part of the open source code of WebLOAD."

This is in direct contradiction to what their website states:

"WebLOAD Open Source introduces a unified script authoring environment for recording, editing and debugging."

When further probed about this, he stated the following:

WebLOAD Open Source is dual licensed:
  1. the WebLOAD Load Engine is totally open sourced and hence is licensed under the GPL
  2. but the complete WebLOAD is still licensed under a proprietary license, which grants free usage in WebLOAD Open Source.

wait.. wait.. WHAT?
I thought the press release said "licensed under the GNU Public License", and "WebLOAD Open Source is a fully functional, commercial-grade performance testing product"?  Nowhere on their website or marketing materials to they talk about this dual licensing and limited availability of source code.

Now, if we look at the End User License Agreement (EULA) that applies to WebLOAD Open Source, it gets worse:

2. License Restrictions. This License does not permit you or any third party to:
(i) modify, translate, reverse engineer, decompile, disassemble (except to the extent that this restriction is expressly prohibited by law) or otherwise attempt to discover the source code of all or any portion of the Software;
(ii) modify, translate or create derivative works of all or any portion of the Software;
(iii) copy the Software (other than a single copy solely for back-up or archival purposes);
(iv) rent, lease, sell, offer to sell, distribute, or otherwise transfer rights to the Software;

OK... so we have an "open source" product that is actually dual licensed, where a large portion of the toolset is proprietary.  And furthermore, by accepting the EULA, you give up all of your rights that were granted under the GPL.  Huh?

So... to reiterate, WebLOAD Open Source is *not* open source.  A subset of it is open source: the [crippled] load engine.  Contrary to what their press release and website says, it contains proprietary components that are released in binary form with no source code.  It is rather disappointing to see a company jump on the bandwagon of open source without respecting the freedom that is supposed to come with it.

In conclusion: Is WebLOAD Open Source currently Open Source?  No
Will WebLOAD Open Source actually become Open Source?  Well.. that's up to Radview

Hey, I'm all for Freedom.  I applaud Radview for any of the code they released under the GPL.  But lets be fair, if you want to call your product open source and reap any benefits that come along with that... you gotta walk the walk.

(Pylot is the open source web performance/load test tool that I am developing)

When a test run is finished, a report is automatically generated to summarize the test results. It includes various statistics and graphs on response times and throughput from the run. A sample of the results report can be seen here:

http://www.pylot.org/samples/results/results.html

Pylot also writes results to CSV files so you can import them into your favorite spreadsheet to crunch numbers, generate statistics, and create graphs. I have been working

I have also been working on the GUI. Here is the latest:

http://www.pylot.org/samples/ui/pylot_ui_screenshot_2007_08_20.png

---

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool
• Pylot - Dev Update #4 - Web Performance/Load Test Tool (New Name and Defining Test Cases)
• Pylot - Dev Update #5 - Web Performance/Load Test Tool (Graphs With MatPlotlib)

We performance practioners love our graphs!  Visualizing data is helpful in analyzing performance results.  Sometimes a quick glance at a graph provides better understanding than a mound of raw or summarized data.  Pylot's Results Reporting feature creates graphs of response times (latency) and Throughput.

For the graphing toolkit, Pylot uses Matplotlib to produce fancy graphs like these:

Matplotlib allows you to graph data from Python. Here is a simple script that gives a glimpse of how a line/marker graph is created as a png image:

---

#!/usr/bin/env python

from pylab import *  # Matplotlib

def main():
    # sequence of data points to graph (x, y coordinates)
    points = [(1, 3), (2, 6), (3, 2), (4, 5)]
    graph(points)
   
   
def graph(points):
    fig = figure(figsize=(6, 2))  # image dimensions  
    ax = fig.add_subplot(111)
    ax.grid(True, color='#666666')
    xticks(size='x-small')
    yticks(size='x-small')
    x_seq = [item[0] for item in points] 
    y_seq = [item[1] for item in points] 
    ax.plot(x_seq, y_seq, 
        color='blue', linestyle='-', linewidth=1.0, marker='o', 
        markeredgecolor='blue', markerfacecolor='yellow', markersize=2.0)
    savefig('graph.png') 
    
    
if __name__ == '__main__':
    main()

---

The output looks like this:

---

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool
• Pylot - Dev Update #4 - Web Performance/Load Test Tool (New Name and Defining Test Cases)

PyLT has been renamed to Pylot (some sort of abbreviation for "Python Load Test")

I realized that having a pronounceable name for a piece of software is pretty important :)

So...
As I develop my load test tool, I need a way to define test cases.  Here is my first attempt:

---

What Is A Pylot Test Case?

You must declare your test cases in an XML file. This is the format that the test engine understands. Editing XML may seem natural to some people, but awkward to others. The nice thing about this structure is that it will be very easy to create a more friendly user interface [sometime in the future] for defining test cases.

A test case is defined using the following syntax. Only the URL element is required.

<case>
  <url>URL</url>
  <method>HTTP METHOD</method>
  <body>REQUEST BODY CONTENT</body>
  <add_header>ADDITIONAL HTTP HEADER</add_header>
  <verify>STRING OR REGULAR EXPRESSION</verify>
  <verify_negative>STRING OR REGULAR EXPRESSION</verify_negative>
</case>

Below is an example of the simplest possible test case file. It contains a single test case which will be executed continuously during the test run. The test case contains a URL for the service under test. Since no method or body defined, it will default to sending an HTTP GET to this resource. Since no verifications are defined, it will pass/fail the test case based on the HTTP status code it receives (pass if status is < 400).

<testcases>
  <case>
    <url>http://www.example.com/foo</url>
  </case>
</testcases>

---

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool

(Update: PyLT has been renamed to Pylot)

(PyLT is the open source web performance/load test tool that I am developing)

A quick update on PyLT development...

The load generating engine is looking pretty solid and seems to work really well so far. It uses threading for concurrency and seems to scale well (though I haven't put it through its paces enough yet).

The GUI is evolving more and starting to look like a real performance/load testing tool:

This is my first project using wxWidgets and wxPython.  I am finding it to be very powerful and relatively straight forward to design nice user interfaces.  However, this is a big jump for me.  The past few years I have mostly done web programming and work with distributed systems.  It took a bit to get my head back into traditional GUI application development and event-driven programming

More to come...

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool

(Update: PyLT has been renamed to Pylot)

(PyLT is the web performance/load test tool that I am developing)

A quick update on PyLT development...

This week I rewrote the GUI using wxPython.  It still needs a *lot* of work, but here is what it's starting to look like:

Related:
PyLT - Dev Update #1 - Web Performance/Load Test Tool
PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)

(Update: PyLT has been renamed to Pylot)

A quick update on PyLT development...

I have a working version of the guts of my tool (the multi-threaded load generator).  I have now started working on the user interface.  My initial idea was to use Tk for the GUI Toolkit.  I started developing a minimal GUI and quickly realized I need a Toolkit more powerful than Tk.

My original justification for using Tkinter (from blog comments):

"I will probably eventually move to a richer toolkit (like wxPython) if I take this thing far. For right now, Tk works. It comes distributed with core python, it's super fast and light, it's easy to use, and I know it pretty well. Though it looks like crap and is limited in many ways."

As of today I am rewriting the GUI with wxPython, which uses the wxWidgets Toolkit.  This should give me the ability to create a rich cross-platform UI for my tool.

[For posterity] Here is what the original prototype of the Tk UI looked like:

R.I.P. Tk... Hello wxWidgets

Related:
PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source) 

(Update: PyLT has been renamed to Pylot)

I have started development on a new web performance/load testing tool.  It is targeted at testing Web Services.

Here is some Q&A with myself:

---

You know you are reinventing the wheel, right?

Yes, I know.  There are already open source web load testing tools available (OpenSTA, JMeter, Grinder, WebLOAD, etc).  I have used all of these as well as proprietary tools for years.  I am a performance engineer and I feel like I need a tool set that I am intimately familiar with.  I need the ability to easily alter and tweak the tool at will.  I don't have the time, budget, or patience enough to wait on vendors when I need something.  I also want a tool that is fun to hack and adapt.  For this, I need to understand the code base deeply.

What language are you using?

Python.  The initial GUI uses Tk, but this may be changed down the road. I use Python's threading module for concurrency. If this doesn't scale well enough, I will be exploring other models of concurrency (perhaps generator based coroutines).

Why do you think you can write a tool like this?

I have worked in performance testing for nearly 10 years.  I have written many tools that work with various protocols to do distributed load generation and testing.  Creating a simple HTTP load generator is sort of my Hello World 2.0 for each language I try (I have written these from scratch in Python, Perl, Java, and C#).  This tool takes that basic concept and organizes it into a robust application.

Will it be Free and Open Source?

Of course!  Licensed under GNU GPL.

---

For an early look, check out the source repository at:  http://pylt.googlecode.com/svn/trunk

More details to come.

-Corey

One cool thing about developing Open Source software is seeing where people end up using your software. I have seen my WebInject test tool show up in various places for various uses.

The most recent example of this come from op5 AB:

"op5 is a leading product developer of systems and network monitoring and management software. Our aim is to give our customers an increased and measurable availability to the IT system – both in terms of quality and quantity. Our products are op5 Monitor, op5 Statistics and op5 LogServer."

op5 Monitor (Linux Open Source Awards: Best Open Source Application 2006) is their network and application monitoring solution:

"op5 Monitor is a system that monitors the whole network. op5 Monitor is unique in its flexibility and it can monitor all net connected components from servers, routers and printers to individual processors, for example mail services, web servers and virus programmes. All these functions are handled by a web – browser. The system can handle a net with a several thousand units."

Carl Ekman wrote a nice paper explaining how to use WebInject as an intelligent agent/plugin for use in web application monitoring:
WebInject and op5 Monitor - Setting up advanced website monitoring with WebInject (PDF)

The paper serves a nice example and tutorial for using WebInject as a monitoring plugin (It can be used in a similar fashion with Nagios also):

"Most op5 Monitor users have detailed monitoring of all inhouse applications and servers, but sometimes not even that is enough. What if something unforeseen happens that makes dynamically generated content spout jibberish to thousands of visitors without you even knowing about it?

With WebInject you can monitor the actual content of the web pages, and you can perform simulated user actions such as logging in and checking an account balance. If a search string is not present, an error message occurs or a link is broken, you can get an alert with a customized, descriptive message."

Programmers Need To Learn Statistics Or I Will Kill Them All

Zed Shaw:

"I have a major pet peeve that I need to confess. I go insane when I hear programmers talking about statistics like they know sh-t when it’s clearly obvious they do not. I’ve been studying it for years and years and still don’t think I know anything. This article is my call for all programmers to finally learn enough about statistics to at least know they don’t know sh-t."

I just got the RESTful Web Services book (Leonard Richardson & Sam Ruby, O'Reilly, 2007) in the mail today.  I've only read the beginning, but so far it is great.  In fact, it brings me back to when I first started working with the "programmable web".  I got into the programmable web back when the web was only a few years old.  I spent years doing performance/scalability testing and tuning for large Web 1.0 applications and bizarre custom Web API's (think huge financial services rushing to get online).  Building tools to run realistic workloads through a system involves writing custom clients to simulate real user/browser interaction.  This is pretty ugly stuff when you are dealing with an application that was designed with only humans in mind (AKA all).  It involves lots of HTTP protocol level work.. screen scraping.. protocol sniffing and analyzing.. requests.. header mangling.. cookie handling.. redirects.. authentication.. session information parsing.. etc, etc.

Application simulation is pretty messy work.  There is no simple API to hide behind; you had to figure out what the API was for yourself.  See.. *every* web application has an API.  Though it might have been designed by accident.  This allowed me to see first hand how developers and frameworks butchered the use of the "Web" as a platform.  Staring at naked HTTP let me see every little bit of the hairball underneath.  Alas, any standardization around web services (or the concept to be officially named) was far off.

A friend (bearded Perl hacker) let me borrow a book to show me how Perl can do this cool web stuff:  Web Client Programming with Perl (Clinton Wong, O'Reilly, 1997).  This book helped me build my first web clients to do application simulation and testing.  There wasn't a ton of documentation at the time to do this sort of thing, so i relied heavily on this book.

So now.. 10 years later..  the Web has changed..  it has morphed into *the* distributed platform..  it is becoming organized.

As I flip through Restful Web Services, it all just looks right..  REST looks right..   It is simple..  it is HTTP..  it is all the guts I already know.  It almost feels like a sequel to my old favorite:

I have traded Perl for Python as my preferred scripting language the past few years, but I am still building simulators, web clients, and virtual users. I am excited to work on some new stuff in this area.

InfoWorld article:

Three open source Web service testing tools get high marks

Rick Grehan of InfoWorld reviewed 3 popular open source tools for testing web services.  Rick is a contributing editor of the InfoWorld Test Center.  One of the tools he reviewed was WebInject (which I wrote).

"In this roundup, I examined three tools that purport to verify that your Web services do what they are supposed to do, that they resist graceless failure, and (in some cases) that they conduct themselves with efficiency. The tools are soapUI, TestMaker, and WebInject. All are open source, and are available for free download and incorporation into your next Web services project."

My tool (WebInject) scored pretty well in the comparison.

---

From the article:

WebInject

WebInject is a super-lightweight testing tool that can automate the testing of both Web services and Web applications. In fact, WebInject's ability to test XML/SOAP Web services appears to be a recent addition to the tool, as earlier versions could not readily handle the SOAP protocol.

Written in Perl, WebInject is primarily a command-line tool, though its author provides a thin Perl/Tk user interface that at least simplifies the execution of tests for those unwilling to spend too much time at the command prompt. If you're not familiar with Perl, don't panic. WebInject is built so that you can construct your tests without having to touch so much as a byte of Perl code.

WebInject is really an execution and reporting engine. Unlike the other tools, it has no IDE-style user interface, so tests must be written in an editor outside of the WebInject UI. This gives WebInject a less professional feel, but doesn't hamper the tool. I envision users of WebInject having directories filled with text files of various test “templates.” To add a new test case, the user just pops open his or her favorite editor, does some cutting, some pasting, and a bit of tweaking to alter the template to fit the specific circumstance, and ba-ding!, you've got a new test case.

...

In essence, a WebInject “project” is nothing more than an XML file filled with a set of elements strung one after the other. WebInject's simple structure lets you build tests with amazing rapidity. You must, however, have a moderately good understanding of the mechanics of SOAP protocols as well as a tool that lets you generate and capture HTTP/SOAP requests and responses. You'll need the requests to build the POST body and the responses so that you can create proper “verifypositive” and “verifynegative” regular expressions to check for success or failure. I used the Web Service Toolkit add-on for Eclipse to grab requests and responses for WebInject; once I had gotten the hang of it, I fell easily into the groove of building test cases.

---

Criteria	Score	Weight
Documentation	8	20%
Features	8	20%
Scalability	8	20%
Ease-of-use	8	15%
Portability	9	15%
Value	9	10%

Review Score:
Very Good 8.3

Cost:
Free download - open source

Platforms
Any platform that runs Perl or has a Perl interpreter installed

Bottom Line:
Much less feature-rich than the other tools, the lightweight WebInject nonetheless bolts out of the starting gate. If you need testing that will be off the ground and flying in minutes, reach for WebInject. On the other hand, it has far fewer capabilities than the other two products in this test, and unless you want to hack the Perl code, WebInject's feature set is pretty much what you install.

---

visit www.webInject.org
for more of my tools, visit: www.goldb.org

I was recently interview by Joseph McAllister for his Sticky ToolLook newsletter.  Sticky ToolLook is an extension of StickyMinds.com and Better Software magazine.  I mostly talk about Performance testing and tools.

The article can be found here: http://www.stickyminds.com/stickytoollook/index.asp?cd=5/10/2007

Transcript:

---

A Word with the Wise:
Tools and System Performance with Corey Goldberg
by Joseph McAllister

Corey Goldberg is a Boston-based software engineer who focuses on performance engineering and tool development. He also contributes to open source projects and has developed some of his own, such as WebInject. I spoke with him earlier this year about his passion for the craft of software tools.

Joseph McAllister:
What makes you passionate about performance test tools?

Corey Goldberg:
I am passionate about system performance, so tools are an integral part of that. Performance is an interesting and diverse space. It touches technology in so many ways, and the skill set it requires allows me to be close to several different technical areas at once: development, testing, analysis, design, operations, etc.

The thinking becomes pervasive, though. For example, last night I was standing in line for movie tickets, and all I could think about was how they could improve the queuing system to get better sales throughput. I regularly have conversations with my colleagues about service performance and input/output contention at our local burrito joint.

JM:
Is there a particular type of performance tool that is more "fun" to use? Is there a type that tends to offer more results?

CG:
The various types of tools all work together to form your full tool set or test suite. They all have their own fun parts. Load generation can be complex, as it involves software development alongside workload modeling. But it is also the fun part where you get to slam load through a system and watch it react.

The most satisfying tools to build are analysis and monitoring tools, especially tools with real-time monitoring and graphing. This enables you to look inside your test runs or production system and actually see what is happening in real time. Complex data sets and metrics collected from deep within a system are transformed into informative graphs as things happen. That is pretty exciting to work with.

JM:
Is there a clear division between the commercial tools you've used and the tools you've written? Do you prefer one or the other?

CG:
Yes and no. First, for terminology, I like to think of things in terms of proprietary vs. free tools. Proprietary tools tend to force you into a certain pattern of use and often don't provide the flexibility to change or extend them in ways you might want.

Lately I have been building a lot of my own tools that work alongside some commercial tools. I recently developed a reporting and analysis suite that replaced the analytics in a commercial tool we were using.

Commercial tools also offer some rich features that are sometimes not feasible to re-create in a reasonable amount of time. So you have to remember that building your own tools is only worthwhile if it is cost effective.

JM:
Describe your open source test tool, WebInject.

CG: WebInject is a test tool that I developed in Perl that is used for functional testing of Web application/services and ad-hoc monitoring of HTTP response times. It can run as its own GUI application with real-time graphing capabilities or can be integrated as a plug-in with other tools.

I was doing this type of stuff in various scripts for years, so I packaged a lot of it together and made a more generic interface that can be used across a variety of projects. I thought others might be interested in it, so I setup a SourceForge project and released it in January 2004.

The basic concept is that you define test cases in XML files that are fed through WebInject and executed against your system under test. WebInject provides a basic harness/framework that includes HTTP transport, parsing, cookie handling, authentication, SSL, etc. It gives you real-time response timing as well as functional verification using regex-based content verification and HTTP status codes.

I spend all of my time on newer tools these days, but I still keep on top of WebInject enough to facilitate others' using it and posting patches/updates to it. Other test tools have progressed a lot in the past few years, so I am sure there are lots of new options for doing this type of testing. Oddly, WebInject has become somewhat entrenched in monitoring systems. Most of the users lately seem to be people running Nagios (an open source monitoring system) that need an intelligent Web plug-in/agent.

JM:
What is your favorite element of creating and distributing an open source software tool?

CG:
Community feedback feels really good. I like sharing and collaborating. The feedback is also tremendously useful in terms of discovering bugs and offering suggestions, advice, or even patches of working code. I care about my craft, and I realize the only way to advance is through open collaboration.

I am also pretty influenced by the free software movement and do some volunteer work with the GNU Project. I have some core beliefs about the ethics of software freedom. Creating and distributing my own GPL-licensed software is my own little way to help that cause.

I wrote a small tool that I have found useful.  It is a Python script that parses and analyzes web log files (in W3C Extended Log File Format).  It creates and HTML report with data and PNG images showing graphs of things like: request throughput, error rates, HTTP method distribution, content type distribution, time-series, etc.

Many log parsing/analysis tools exist, but I was looking for something more specific to Performance than something a webmaster would want to look at.

The script is pretty basic. It was very useful for my own needs, but others might want to modify it.  If anyone has good suggestions to add to it, I am willing to enhance it at some point (or just grab my code and hack it yourself if you know Python).

---

Project Home

• www.goldb.org/perflog

Features

• Produces metrics and graphs from web logs (W3C Extended Log File Format)
• Useful during performance testing and analysis
• Output is created in XHTML/CSS with embedded PNG images
• PerfLog is written in Python and uses Matplotlib for graphs and plotting

License

• GNU GPLv2 (always free)

Project Info

• Google Code - Project Home
• Subversion Trunk (Source Code)

Requirements

• Python 2.4+
• Matplotlib (requires Numeric or Numpy)

Platforms

• Cross-Platform.  PerfLog will run on any system that supports Python and Matplotlib.

OK, this is huge news: www.webload.org

The commercial performance/load test tool market is dominated by large proprietary commercial vendors (HP/Mercury, Borland/Segue, etc). Radview has a nice product called WebLOAD that competes in the space.

As of this morning, Radview announced they have released WebLOAD OS, an open source version of WebLOAD. It is full-on GPL licensed (no fake open source). I already browsed their source tree. They have a Subversion repository.. code is in C and C++,

The Open Source performance/load test tool market doesn't offer many choices. Currently the most popular tools are JMeter and OpenSTA. 

This will be exciting. I wonder how well Radview will deal with the community on this. Though if it's not good, GNU GPL certainly allows forking :)

more to come...

I have been using IEC lately for automating repetitive administrative tasks within my company:

IEC.py - Automating Internet Explorer with Python

IEC is a simple library with a nice API for automating an IE browser. I found it simple to work with for basic automation needs. I have also used it as the core of a small UI testing framework.

From Mayukh Bose:

IEC is a python library designed to help you automate and control an Internet Explorer window. You can use this library to navigate to web pages, read the values of various HTML elements, set the values of checkboxes, text boxes, radio buttons etc., click on buttons and submit forms.

Yeah I know.. pretty lame it only works with IE, but in the environment I was working in, the applications ran on *IE Only*.

A personal story:

My company is very analytical and detail oriented when it comes to tracking/planning project resource allocation. We track all sorts of projections, budgets, resources, etc. The workflow is basically: some business guys (no idea what they actually do) take data from some reports and enter them into some arcane hosted tracking software. This is done by entering copious amounts of data into web form after web form. Then they submit the form to run a report. Once that is finished, they cut & paste the data into MS Excel. Then they take the Excel spreadsheet and follow some wild sequence of copying, cutting, pasting, converting, running macros, graphing, etc. At the end of this, a few images are produced so some wizz-bang graphs can go into a monthly Powerpoint... wow.

So... I wrote a Python script that takes their input data, drives a web browser to do the report, screen scrapes the result, processes it, generates some fancy graphs with Matplotlib, and presents a web page with the results.  End result: Converted a multi-hour manual process into the click of an icon and 20 seconds of processing.

I could have done this with HTTP directly, but this UI automation technique made it very quick to develop; and it looked impressive ("whoa it's like.. making my browser move on its own").

To use IEC, you need the Python for Windows Extensions. If you use the ActiveState Python distribution, these are already included.

I used to use ActiveState Python for Windows programming (because I was a big fan of ActiveState Perl, where the installer and PPM package manager rocked). I recently spent close to an hour getting SSL (HTTP) to work with ActiveState.  I couldn't get it to work so I ditched it for the standard Python distro.

--
Happy Hacking.

Impressive...

JakeBrake is a true geek:

"I am so technical that:

• I routinely do unit-level performance/timing tests on Cialis to see if will time-out at 4 hours.
• For meals I eat only donuts and hotdogs; arranging them on my plate as "bites" in patterns of ones and zeros.  I use a burnt hotdog as a signed bit."

If you are into testing and performance, Sounds of Jake Braking blog is a great read.

OpenSTA is a distributed performance testing architecture and tool.  Recently, on the opensta-users mailing list, Bernie Velivis (from Performax Inc) posted an excellent article about performance testing strategies.  I thought this information would be useful for all performance testers who are just learning the craft.  Rather than letting it sit in some arcane mailing list archive, I asked Bernie if I could re-publish it here.

enjoy.

-Corey

Bernie Velivis on Performance Testing Strategies:

---

PERFORMANCE TESTING STRATEGIES

The terminology used to discuss Performance testing in technical publications and support forums can be ambiguous or inconsistent. Hopefully this article will help participants in the OpenSTA user support forum by providing a common frame of reference for discussing tools, testing, and test results. It may also be helpful to those new to performance testing.

CAPACITY TESTING

If your goal is to determine the CAPACITY of the system under test, start by creating a "realistic" workload consisting of a mix of the most popular transactions plus those deemed critical or known to cause problems even when executed infrequently. Pick a manageable set of transactions to emulate (considering time, budget, and goals), determine the probability of executing each transaction, the work rate for the emulated users, and the "success criteria for performance metrics (i.e. response time limits, concurrent users, and throughput).

One way to implement this approach is to create a master script, assign it to each VU, and have it generate random numbers and then call other scripts which model the individual workload transactions based on a table of probabilities. The scripts should be modeled with think times consistent with the way your users interact with the system. This varies greatly from one application to another and unless you are mining log files from an application already in use, this is a somewhat subjective process. The best advice I can give in defining the workload is to get input from people who know how the application is (or will be) used, make conservative assumptions (but no so much so that the sum of all your conservative decisions is pathological), and balance the scope of the workload vs. time to complete the project. Another important consideration is the data demographics of the transactions and the size and contents of the database.

When its time to test, increase the number of emulated users and monitor how response times, server resource utilization (CPU, disk IO, network, and memory), and throughput (the rate of tasks completed system wide) vary with the increased load. You might construct a test that ramps up to a specific number of users, lets them run for a while, and then repeats as necessary. This way, you can observe the behavior of the system in various steady states under increasing load. Workloads containing transactions having a low probability of being executed and/or a disproportionately large impact on the performance of other transactions usually need to run longer to reach a steady state. If you can't get repeatable results, your steady state interval might be too small. As a rule of thumb I would suggest a minimum ramp up time equal to the duration of the longest running script and the steady state observation period at least twice as long as the ramp up period. I also tend to ignore response times and performance statistics gathered during the ramp up periods and focus instead on the data collected during the steady state periods.

That's a rough outline of one approach to capacity testing which in summary is an attempt to load up the system with VUs in a way that is indistinguishable from a "real users" in order to find the capacity limit. Pick the wrong workload however and you might miss something very important or end up solving problems that won’t exist in the real world.

The end game here is to increase load until response times become excessive at which point you have found the system’s capacity limit. This limit will be due to either a hardware or software bottleneck. If time and goals allow, analyze the performance metrics captured, do some tuning, improve code efficiency or concurrency, or add some hardware resources. Make one change at a time and repeat as necessary until you meet capacity goals, find the limits to the architecture, or run out of time (which happens more then most performance engineers would like).

SOAK TESTING

The same scripts created for capacity testing can also be used for SOAK TESTING where you load up the system close to its maximum capacity and let it run for hours, days, etc. This is a great way to spot stability problems that only occur after the system has been running a long time (memory leaks are a good example of things you might find).

FAILOVER TESTING

Get the system under test into a steady state and start failing components (servers, routers, etc) and observe how response times are effected during and after the failover and how long the system takes to transition back to steady state and you are on your way towards FAILOVER TESTING. (A gross simplification and again there is lots of good reading material out there on failover and high availability testing).

STRESS TESTING

If your goal is to determine where or how (not if) the system will fail under load, then you are doing STRESS TESTING. One way to do this is to comment out the think times and increase VUs until something (hopefully not your emulator!) breaks. This is one form of stress testing, a valuable aspect of performance testing, but not the same as capacity testing. How the VUs compare to "real users" may be irrelevant as you are trying to determine how the system behaves when pushed past its limits.

A report illustrating how these concepts were used to performance test a SOAP application using OpenSTA can be downloaded here:
SamplePerformaxPerformanceReport.pdf

Bernie Velivis
Principle Consultant and President, Performax Inc
www.iPerformax.com

---

I posted a lengthy entry to the "getting feedback on tools" thread going on at the toolsmith-guild group. 

The thread is discussing how to get feedback and participation going in test tools projects.  This experience report sheds some light on this, but more generally can be applied to getting your open source project noticed and used.  I thought other open source developers might find this useful.

Here is my post:

---

Hi.. new to the list.

I have some experience in this area I want to comment on:

I had a specific need for a tool to test web services a few years ago ('02-'03). I had been rolling my own tools in various languages for quite a while and had some stuff that I thought others might be interested in; so I setup a SourceForge project and released it in Jan'04. The tool is called WebInject. Not many people used it at first. I just kept adding features and releasing. I was scratching my own itches and using my tool internally at a company I was working for. Eventually I got a few bites and some people started to download it and try it out.

As of today, the project site has served over a quarter million pageviews and the tool has been downloaded over 30,000 times. I've had feedback from all corners of the world.

See the Download Stats

In the early days, *any* bug that was reported, I would dive into immediately.. I was turning around fixes and new releases in crazy time. This personal attention sparked a lot of lengthy email conversations where I was starting to get real feedback. It wasn't the case where I could just sit back and expect quality feedback. I really engaged the users and asked questions. Pretty soon I was getting more feedback than I could reasonably handle. Around this time I started getting some patches sent to me. This accelerated development as you start to get various ideas from other people. A few silly bugs were also cleaned up.. stuff I never would have caught on my own. Encouraging patches is huge.. once someone gets a piece of their own code into your code base, they have an intimate connection.

I was also contacted by someone writing a book that wanted to use my tool as an example. I was really excited about this and put in some serious time getting the tool cleaned up and more useful. I never saw a published copy, but apparently it exists (I saw the PDF drafts :) It was also mentioned in a few testing magazines and articles which was cool for me to see.

I don't maintain WebInject much any more. I think it is cool and useful, but it's a rats nest of Perl code that needs some serious love. There is some stuff that bothers me so much about it that I barely want to touch it sometimes :) I spend all my time on newer tools these days, but I still keep on top of it enough to facilitate others using it and posting patches/updates to it (though I haven't released in over a year I think). Oddly, it has become somewhat entrenched in monitoring systems. Most of the users lately seem to be people running Nagios (open source monitoring system) that need an intelligent web plugin/agent.

Some basic thoughts on open source tool adoption and getting feedback:

• A tool without documentation doesn't exist.. without quality documentation, don't even bother.. nobody will touch it.
  
• If it takes someone more than 10 minutes to install, configure, run, and see the results from a simple test case.. nobody will use it. Once someone sees something work, it is tangible and they become engaged. I achieved this with my tool.. if you download and run it, you are presented with a GUI. Pressing "run" executes a preconfigured sample test case that hits my site. I wanted the most clueless of people to be able to immediately see a green "PASS", and say "hey, cool it actually did something". This satisfaction must come immediately or it will be abandoned. (unless of course you already have a user base that has made it past this hurdle and can encourage others to put in the effort to get it working)
  
• Make your tool easy to get. The online presence of your tool should allow someone to visit the website and immediately download it. I have seen tools released that take 5 levels of navigation before you get to the download options, or don't even have a website or project page somewhere.
  
• Announce your tools.. release early, release often.. who cares.. blog it, forum it, email it, digg it, usenet it, comment about it.. see if anyone has interest. Don't annoyingly spam it, bet let people know it exists. If you announce your tool once only, buried deep in a technical mailing list, don't expect people to come rushing.
  
• Encourage participation. Go out of your way to let people know that you encourage and appreciate feedback, bug reports, offers for collaboration, patches, etc.
  
• You need a forum or a mailing list with a web accessible archive. It has to be super simple to talk to you in public and let others see answers to questions and begin to participate on their own and to offer advice. Using a many-to-one scheme where you just tell people to email you isn't gonna work. It is too private.. people want to know what is going on and hear from each other and offer tips. I have had several thousand posts to my forums for this little tool.
  

- Corey Goldberg

---

I saw this posted in the toolsmith-guild group (Danny Faught):

"The developers were very surprised to find a whole room full of testers who didn't cringe at the thought of reading and writing code. The rather odd terms Developer-Tester/Tester-Developer emerged from AWTA." (Austin Workshop on Test Automation)

ahh, the elusive "develester"

... and.. the develester algorithm in Python :)

Scott Barber just posted some info about: Patterns & Practices: Performance Testing Guidance, a new guide to Performance Testing over at Microsoft's Codeplex.

from Scott:

"I am involved in Microsoft's Patterns & Practices Performance Testing Guidance project. We have reached a critical mass with regards to our "mostly final" content and have made that content publicly available"

"We're tackling various flavors of performance testing (stress, load, capacity) as well as how to bake performance testing into your life cycle."

from the Patterns & Practices site:

"The purpose of this project is to build some insightful and practical guidance around doing performance testing and using Visual Studio 2005. It's a collaborative effort between industry experts, Microsoft ACE, patterns & practices, Premier, and VSTS team members."

I have always had ambivalent feelings towards MS, but it is great to see all of the effort they are putting into the Performance field. Performance has always been a sort of niche domain. It straddles the disciplines of development, testing, and operations, while also involving the integration of software, hardware, and networks. The past few years have provided much more thought leadership that is pushing the state of Performance (load, stress, scalability, capacity, availability) into more mature territory.

For those that don't know Scott Barber, he certainly gets my vote for the most prolific writer in Performance over the past several years. His body of writing is second to none: http://www.perftestplus.com/pubs.htm

(Moving some of my older blog posts to this site for permanent archiving.. This entry was originally posted at testingReflections on 04/25/2006)

I have spent a lot of time playing around with graphics libraries and toolkits for integrating real-time graphs within my own performance testing and monitoring tools. It seems like there are many open source tools available in the world of performance testing and system monitoring. And lots of people roll their own tools in whatever programming language they are into... but many lack graphics capabilities.

Two of the toolkits/libraries I end up using often for my own homebrew test tools are: RRDTool, and JRobin.

from the RRDTool site:
"RRD is the Acronym for Round Robin Database. RRD is a system to store and display time-series data (i.e. network bandwidth, machine-room temperature, server load average). It stores the data in a very compact way that will not expand over time, and it can create beautiful graphs. It can be used via simple shell scripts or as a perl module."

So...
RRDTool is a really good back-end for storing time-series data; which is mostly what we care about in performance testing.  It has bindings for various scripting languages, or can be invoked from the command line. If you are developing tools that need a data repository and graphing capabilities, this provides you both. You create an RRD and then you begin inserting data values at regular intervals. You then call the graphing API to have a graph displayed. The cool thing about this data storage is its “round robin” nature. You define various time spans, and the granularity at which you want them stored. I fixed binary file is created, and this never grows in size over time. As you insert more data, it is inserted into each span. As results are collected, they are averaged and rolled into successive time spans. It makes a much more efficient system than using your own complex object structures, or a relational database, or file system storage.

You will probably recognize the graphs it creates, as RRDTool is integrated in many popular monitoring tools (it is Free/Open Source, GPL License). I have built many tools around RRDTool, and it is really a nice system.

If you are in the Java world, there is a very cool project named JRobin. JRobin is a clone of RRDTool in pure Java. So you can create RRD's directly from your Java code.. and all in memory if you want to!

Some days I pretend to be a Java programmer, so I had to build a tool using JRobin. As a proof of concept, I wrote a small network latency monitoring tool. It shows off some of JRobin's capabilities. It pings a host at a given interval and records the latency. A graph of the network latency is rendered in real-time onto a Swing panel.

Here is my network latency monitoring tool: NetPlot (includes Java source code, GPL Licensed)

The tool itself is just a trivial example, and really isn't the point. But you could easily adapt this code or create your own to develop real-time graphs of your own time-series data.

There currently aren't many commercial UI test tools for GNU/Linux applications.  GNU/Linux has come a long way towards becoming more popular on the desktop, but it is still somewhat niche in the business world.   There is a large contingent of Windows software testers and QA engineers that make their living using commercial UI test tools (WinRunner, QTP, SilkTest, Robot, etc) from the big tool vendors (HP/Mercury, IBM/Rational, Borland/Segue, Compuware, etc).  I am not talking about small test utilities; I am talking about large UI layer test suites that people build extensive customized test frameworks on top of.  These are used most often in large business applications for automating functional and regression tests.

Good test tooling is a prerequisite for any large deployment of a business application.  As GNU/Linux becomes more popular on the desktop, I think this will become a more important factor and tool vendors will begin to beef up their GNU/Linux UI test tool offerings.  It would be great if there were viable open source tools as an alternative.  On Windows, this never happened.  There are currently no high quality open source UI test tools available.


I just took a look at the GNU/Linux Desktop Testing Project (GNU/LDTP):

"GNU/Linux Desktop Testing Project (GNU/LDTP) is aimed at producing high quality test automation framework and cutting-edge tools that can be used to test GNU/Linux Desktop and improve it."

wow.. I had never heard of that until now.

The description looks good.. it is a UI layer test tool that works in both GNOME and KDE environments.

.. and it is Free/Open Source.

.. and it is written in Python (which completely rules)

I will be keeping an eye on this and any other open source test tools in that space.

Cem Kaner just posted an incredible piece of writing about the Schools of Software Testing.

Especially intriguing was his explanations of "paradigms" in a generic sense, and references to Thomas Kuhn.

good stuff.