goldb.org home

AS OF MAY 2008, THIS BLOG IS NO LONGER BEING UPDATED.
Visit the new blog at: http://coreygoldberg.blogspot.com


 Friday, March 28, 2008

Can I Test My Application With This Performance Tool?

In the various testing forums and mailing lists I frequent, I see lots of questions asked about automated test tools.  A certain question about performance/load testing tools is asked over and over.  I would like to answer it here.

The question goes something like:
"Can I test my application with this performance tool?".

If you are asking this question, you really don't understand the nature of these tools and how they work.

There are many performance/load testing tools in the proprietary and open source worlds (LoadRunner, SilkPerformer, QALoad, JMeter, OpenSTA, etc).  Conceptually, they all work the same way.  Performance tools do not work by driving a GUI like many functional test tools do.  Instead, they are used to generate load against servers by simulating concurrent client requests at the protocol level.

So the key question you should be asking is: "Which protocols do the client and server use to communicate, and does the tool support these protocols?"  HTTP?  ODBC?  DCOM?  JMS?  So next time you want to ask: "Will the tool be able to test my AJAX/ASP.NET/JSP/PHP/etc application?"; the real question should go something like: "My application uses <protocol x>, does the tool support that protocol?".

Since you are working at the protocol level, the language/platform your system uses has no relevance. All of the technologies I just mentioned use HTTP for transport (layer 7), so any tool that supports HTTP can be used to test that system.



Interested in Web performance testing?
Check out my open source tool: Pylot

 | 
#    Comments [4] |
Friday, March 28, 2008 1:05:03 PM (Eastern Standard Time, UTC-05:00)
you are right with one exception: AJAX - while all of those tools you mention are able to handle http, not all can simulate the correct user behaviour caused by asynchronous (non-blocking) http responses.

Christian Kreutzer | herr dot kreutzerAT NOSPAMgmail dot com
Friday, March 28, 2008 2:21:50 PM (Eastern Standard Time, UTC-05:00)
@ Christian:

Why is AJAX any different?

With these tools you can model HTTP transactions in any way you want. You have complete control down to the socket in many cases. If a browser or client can do something, the tool can also. If you want to pipeline requests over HTTP 1.1 or whatever, it is all available.
Corey
Friday, March 28, 2008 5:53:20 PM (Eastern Standard Time, UTC-05:00)
@ Corey:
while you can simulate all of the AJAX http requests with a performance test tool, the simulated user experience might differ from that of a real user. Lets take SilkPerformer for example: SilkPerformer treats requests synchronous (blocking) and waits for the response in order to make the next request (unless Borland added some support for asynchronous reqs since the last time i used SP...). This behaviour can skew the response times of your tests. More important: the generated load can differ as well. While the tool waits for a response which was meant to be asynchronous a real user could make 2nd request in the meantime...

You are right: as long as the tool can handle the protocol of the application under test, the tool can be used to test the app. But in my experience not every tool was up to the task to deliver accurate results when dealing with AJAX apps.

-Chris
Christian Kreutzer | herr dot kreutzerAT NOSPAMgmail dot com
Friday, March 28, 2008 7:26:03 PM (Eastern Standard Time, UTC-05:00)
@Chris:

I see your point and guess thatmay be one more factor. But all tools don't work like that. For example, OpenSTA can do nonblocking requests from virtual users.

you may be right about SilkPeformer, but last time I used it, I remember there being a setting for threads per virtual user. So you could therefor make multiple concurrent requests and simulate AJAX.
Corey
Comments are closed.