Impressive...

JakeBrake is a true geek:

"I am so technical that:

• I routinely do unit-level performance/timing tests on Cialis to see if will time-out at 4 hours.
• For meals I eat only donuts and hotdogs; arranging them on my plate as "bites" in patterns of ones and zeros.  I use a burnt hotdog as a signed bit."

If you are into testing and performance, Sounds of Jake Braking blog is a great read.

I have been writing a lot of code in both C# and Python lately... flipping back and forth between both languages.  One thing I keep getting tripped up on is the terminology used in regular expression syntax, and what a "match" is.

So for my own disambiguation:

• Python's re.match() is different than C#'s Regex.IsMatch()
• Python's re.search() is similar to C#'s Regex.IsMatch()

Better explained in code:

Using Regex.IsMatch() in C# to match a pattern with some text:

this prints 'Match'

Same thing, using re.match() in Python:

this prints 'No Match'

oops.. didn't get a match. What happened?

match() only checks if the regex matches at the beginning of the string, while search() will scan forward through the string for a match.

If you were expecting the pattern to match anywhere in the string, you need to use re.search() instead:

this prints 'Match'

... or else you must supply a pattern that will match from the beginning of the string:

this prints 'Match'

OpenSTA is a distributed performance testing architecture and tool.  Recently, on the opensta-users mailing list, Bernie Velivis (from Performax Inc) posted an excellent article about performance testing strategies.  I thought this information would be useful for all performance testers who are just learning the craft.  Rather than letting it sit in some arcane mailing list archive, I asked Bernie if I could re-publish it here.

enjoy.

-Corey

Bernie Velivis on Performance Testing Strategies:

PERFORMANCE TESTING STRATEGIES

The terminology used to discuss Performance testing in technical publications and support forums can be ambiguous or inconsistent. Hopefully this article will help participants in the OpenSTA user support forum by providing a common frame of reference for discussing tools, testing, and test results. It may also be helpful to those new to performance testing.

CAPACITY TESTING

If your goal is to determine the CAPACITY of the system under test, start by creating a "realistic" workload consisting of a mix of the most popular transactions plus those deemed critical or known to cause problems even when executed infrequently. Pick a manageable set of transactions to emulate (considering time, budget, and goals), determine the probability of executing each transaction, the work rate for the emulated users, and the "success criteria for performance metrics (i.e. response time limits, concurrent users, and throughput).

One way to implement this approach is to create a master script, assign it to each VU, and have it generate random numbers and then call other scripts which model the individual workload transactions based on a table of probabilities. The scripts should be modeled with think times consistent with the way your users interact with the system. This varies greatly from one application to another and unless you are mining log files from an application already in use, this is a somewhat subjective process. The best advice I can give in defining the workload is to get input from people who know how the application is (or will be) used, make conservative assumptions (but no so much so that the sum of all your conservative decisions is pathological), and balance the scope of the workload vs. time to complete the project. Another important consideration is the data demographics of the transactions and the size and contents of the database.

When its time to test, increase the number of emulated users and monitor how response times, server resource utilization (CPU, disk IO, network, and memory), and throughput (the rate of tasks completed system wide) vary with the increased load. You might construct a test that ramps up to a specific number of users, lets them run for a while, and then repeats as necessary. This way, you can observe the behavior of the system in various steady states under increasing load. Workloads containing transactions having a low probability of being executed and/or a disproportionately large impact on the performance of other transactions usually need to run longer to reach a steady state. If you can't get repeatable results, your steady state interval might be too small. As a rule of thumb I would suggest a minimum ramp up time equal to the duration of the longest running script and the steady state observation period at least twice as long as the ramp up period. I also tend to ignore response times and performance statistics gathered during the ramp up periods and focus instead on the data collected during the steady state periods.

That's a rough outline of one approach to capacity testing which in summary is an attempt to load up the system with VUs in a way that is indistinguishable from a "real users" in order to find the capacity limit. Pick the wrong workload however and you might miss something very important or end up solving problems that won’t exist in the real world.

The end game here is to increase load until response times become excessive at which point you have found the system’s capacity limit. This limit will be due to either a hardware or software bottleneck. If time and goals allow, analyze the performance metrics captured, do some tuning, improve code efficiency or concurrency, or add some hardware resources. Make one change at a time and repeat as necessary until you meet capacity goals, find the limits to the architecture, or run out of time (which happens more then most performance engineers would like).

SOAK TESTING

The same scripts created for capacity testing can also be used for SOAK TESTING where you load up the system close to its maximum capacity and let it run for hours, days, etc. This is a great way to spot stability problems that only occur after the system has been running a long time (memory leaks are a good example of things you might find).

FAILOVER TESTING

Get the system under test into a steady state and start failing components (servers, routers, etc) and observe how response times are effected during and after the failover and how long the system takes to transition back to steady state and you are on your way towards FAILOVER TESTING. (A gross simplification and again there is lots of good reading material out there on failover and high availability testing).

STRESS TESTING

If your goal is to determine where or how (not if) the system will fail under load, then you are doing STRESS TESTING. One way to do this is to comment out the think times and increase VUs until something (hopefully not your emulator!) breaks. This is one form of stress testing, a valuable aspect of performance testing, but not the same as capacity testing. How the VUs compare to "real users" may be irrelevant as you are trying to determine how the system behaves when pushed past its limits.

Bernie Velivis
Principle Consultant and President, Performax Inc
www.iPerformax.com

I have used Nagios for several years, and it has served me well as an open source distributed monitoring system.

I just read about Zabbix, and I'm posting here so I won't forget to check it out.  Zabbix is GPL (v2) licensed and looks interesting.  I will post more once I get a chance to play with it.

Here are some examples of iterating through multiple sequences simultaneously in Python:

I start with 2 lists of numbers:

I want to create a new list that is made up of the sum of the items at each position in the original lists.  So I will end up with this:

Starting with an unpythonic way...
Here I use a counter to iterate through the indexes of each sequence and build a new list:

Getting more pythonic...
Here I use zip. Zip allows me to iterate each sequence simultaneously, assigning the current sequence values each time through the loop:

The older pythonic way to do this was with map:

Getting even more pythonic and more concise...
I can combine zip with a list comprehension and do it in a one-liner like this:

*note:  zip will not be part of Python 3000.  It will be replaced by izip and iterators to achieve similar results.

In his article: "Joe Barr rips proprietary software vendor a new one", Joe does exactly what the title states  :)

His article was a response to an earlier piece by Roger Greene (CEO of Ipswitch), where Roger says some very confused/uninformed things about Open Source software.

One thing Joe didn't rip was this excerpt from Roger Greene:

"The open source community claims bugs can be fixed faster for open source software than commercial software because hundreds, if not thousands, of people are looking at the code daily and can help with fixes. [ ... ] Even when those individuals generously offer their time for free, can you really afford to wait for one to agree with you on the urgency of action if your network is down?"

Huh?

That is a very odd and misleading way to look at it.  Open Source gives you the ability to modify the code yourself.  You don't have to wait for anyone.  You can hire a freelance developer or consultancy to fix it on the spot.  If you find a problem in a proprietary vendor's software, can you do the same?

No.. proprietary software puts you at the mercy of your vendor.

I just stumbled across the PLEAC Project (Programming Language Examples Alike Cookbook).

Project Description:

"Following the great Perl Cookbook (by Tom Christiansen & Nathan Torkington, published by O'Reilly; you can freely browse an excerpt of the book here) which presents a suite of common programming problems solved in the Perl language, this project aims to gather fans of programming, in order to implement the solutions in other programming languages."

There is sample code in many popular languages.  The Python examples are really good.  They would serve as an excellent primer for someone moving from Perl to Python, or as a general Python reference with cookbook-style examples.

It is hosted at SourceForge and licensed under the GNU Free Documentation License (GFDL).

I got hooked on Last.fm last summer.  Since then, I've scrobbled over 6,800 tracks.. not bad!  (I've been a music junkie my entire life).

I submit my played tracks from all of my music players (Foobar2000, Winamp, iTunes, Squeezebox/SlimServer).  It populates fantastic statistics and charts of my listening habits and lets me listen to streams from people with similar tastes.

Here are my top 10 artists since I started using it:

Once you scrobble enough tracks, you can start streaming custom stations based on your listening habits.  Something new they just added is the ability to embed a Last.fm player widget into your own site. I just knocked up a quick web page with the embedded player so I can listen to my own station from anywhere that has a browser (requires Flash).  Check it out and have a listen.

Reverend Ted just started the "Show Us Your Rack" blog campaign ;)

Well.. I've seen a few nice racks in my day.  I took these pics last year.  Forget the tiny colo cage, this data center gives you some room to spread out:

plenty of room:

looking down the aisle:

got load balancing?:

the big dogs:

Joe Duffy (from Microsoft) talking about the .NET 2.0 CLR:

"It's also worth noting that the threadpool throttles its creation of threads to 2/second once the count has exceeded the # of CPUs."

Yuck...  I don't like throttling like that behind the scenes.  It can make performance problems *very* hard to diagnose.

Passing parameters to functions and methods.  Pass-by-value?  Pass-by-reference?  Which does your language use?

You probably learned this in your first CS class... so did I.

Then why did it take me a frakin' month to understand what Python does? :)

Well... if you look online, you will find some very ambiguous answers about Python being pass-by-reference or pass-by-value.  (which ends up boiling down to semantics and how you use certain terminology, but forget that for now)

To review, how do other languages handle this concept?

C is pass-by-value

Straightforward. You can simulate pass-by-reference with pointers.  Not much else to say here.

Java is pass-by-value

Primitive Types (non-object built-in types) are simply passed by value.  Passing Object References feels like pass-by-reference, but it isn't.  What you are really doing is passing references-to-objects by value.

OK, so what about Python?

Python passes references to objects by value (like Java), and everything in Python is an object. This sounds simple, but then you will notice that some data types seem to exhibit pass-by-value characteristics, while others seem to act like pass-by-reference... what's the deal?

It is important to understand mutable and immutable objects. Some objects, like strings, tuples, and numbers, are immutable.  Altering them inside a function/method will create a new instance and the original instance outside the function/method is not changed.  Other objects, like lists and dictionaries are mutable, which means you can change the object in-place.  Therefore, altering an object inside a function/method will also change the original object outside.

For entirely too much information about this topic in Python and across many other languages (Java, Scheme, C#, C, C++, Python), read the thread where these quotes come from:

Is Python By Value Or By Reference?

Alex Martelli:

The terminology problem may be due to the fact that, in python, the value of a name is a reference to an object. So, you always pass the value (no implicit copying), and that value is always a reference.
[...]
Now if you want to coin a name for that, such as "by object reference", "by uncopied value", or whatever, be my guest. Trying to reuse terminology that is more generally applied to languages where "variables are boxes" to a language where "variables are post-it tags" is, IMHO, more likely to confuse than to help.

Michael Hoffman:

Alex is right that trying to shoehorn Python into a "pass-by-reference" or "pass-by-value" paradigm is misleading and probably not very helpful. In Python every variable assignment (even an assignment of a small integer) is an assignment of a reference. Every function call involves passing the values of those references.

word.

The many hats of a performance engineer:

• tester
• developer
• toolsmith
• dba
• sysadmin
• network/ops/tech
• system integrator
• architect
• statistician
• data visualizer
• sadist

Wow. I just read something interesting about the One Laptop Per Child (OLPC) project in Guido's PyCon writeup:

"The software is far from finished.  An early version of the GUI and window manager are available, and a few small demo applications: chat, video, two games, and a web browser, and that's about it!  The plan is to write all applications in Python (except for the web browser), and a "view source" button should show the Python source for the currently running application.  In the tradition of Smalltalk (Alan Kay is on the OLPC board, and has endorsed the project's use of Python) the user should be able to edit any part of a "live" application and see the effects of the change immediately in the application's behavior."

So... they are going to be running a GNU/Linux OS (a stripped down version of Fedora), with essentially all applications in Python.

This is very cool on many levels. It is the ultimate endorsement of Python.  It also makes me think about the future...  If OLPC is successful, a few years down the road we might be looking at several million young new Open Source/Python hackers.  Nice!

With Python's Windows Extensions, you can talk via COM to an Exchange Server and read/process your email.  You must have the Outlook Client installed on the box you are running this from.

Here is a sample script that will:

• connect to your mailbox
• print the inbox name
• print the message count
• print the subjects for all your email messages

Guido van Rossum just published his slides from the PyCon 2007 Keynote where he discusses Python 3000.

His talk is also available on Google Video.

I'm psyched to watch this!

New England weather is crazy. In Boston, we get pounded with snow some winters.  Other winters we hardly get any of the white stuff.  This year has had really minimal snowfall, but it was coming down this morning.

I snapped a few pics on my way to work (I live in Back Bay).

Looking out my apartment window:

Looking out from my front stoop:

The view down Comm. Ave:

Copley Square, Old Trinity Church, John Hancock Building, Boston Public Library:

Types Of Confusion

He tackles a lot of misconceptions and myths about Typing.

To frame his points, he starts with some fantastic definitions.  I am all for clarifying vocabulary and I really like this:

Based on the apparent confusion, I think it is best to clarify what I mean by each of the following terms:

Type Checking - Verifying that code respects type constraints.

Statically Typed - Type checking occurs at compile time.

Dynamically Typed - Type checking occurs at run time.

Type Safe Language - A language which protects its own abstractions.

Type Unsafe Language - A language which is not type safe.

Strongly Typed and Weakly Typed - Depends on the author; The definitions are so many and so varied that the terms are practically useless. It seems that anyone can claim that language X is either strongly typed or weakly typed based on sound reasoning derived from one of the various definitions.

Dynamic Language - A language which enables runtime inspection or modification of a program; most languages can do this but dynamic languages make it easy. It is common for people to refer to "dynamic languages" and mean "dynamically typed languages" as the term is defined here.

good read.

Good to know..

See all the exceptions you are swallowing in .NET (Visual Studio debugger):

Turn on 'all exceptions' and watch the fireworks fly

One of the most glaring differences for me between Java and .NET is the difference between Checked/Unchecked Exceptions, so stuff like this has been helpful to me in figuring out how exception handling in C#/.NET really works.

I personally use 4-space indents when I write code (in any language, period).

Oddly... in Python, where whitespace matters, there is no single common practice (which would fit in nicely with Python's TOOWTDI ideology).

Some observations of source code indentations in Python:

• I mostly see 4-space indents in code I read (colleagues, libraries, 3rd party code)
• Python's own source (the Python parts) use 4-space indents
• Google uses 2-space indents in their Python
• I hate tabs (if I randomly sucker-punch you, this is why)
  

Guido van Rossum (creator, lead developer, and BDFL of Python) writes: 

"If it uses two-space indents, it's corporate code; if it uses four-space indents, it's open source. (If it uses tabs, I didn't write it! :)"

IronPython is the Python implementation that runs on the .NET platform... originally created by Jim Hugginin, but later backed (overtaken?) by Microsoft.

Microsoft's ambivalence towards Free software is a bit hard to follow sometimes and it really makes me question their entire approach to the software community  (wait.. have I ever *not* questioned that? :).

As a Python advocate and *nix geek trying to make my way working in a .NET shop, I am really excited about IronPython.  I was also initially impressed with Microsoft's embracement of the Python community and toe dipping into Open Source.  But then I hear Microsoft will not take patches from non-Microsoft developers and will not bundle IronPython with other applications which have certain Free licenses(LGPL, BSD). To me, this is really a shame. That is not how you approach a community.

Well... at least somebody has stepped up and is maintaining IronPython Community Edition (IPCE).

props.

So check out IPCE and the FePy Project!

This is what I was talking about when I wrote:

"one of the most attractive things about the Web is the ability to use HTTP as a simple transport protocol abstraction.  [...]  with this additional transport abstraction in place, you can build another application layer protocol on top of this and use that as your API for distributed operations.  That is where the rubber meets the road in modern large scale systems, and that is where the action is taking place in the current debate about SOA, REST, Web Services, and distributed architectures.  Furthermore, the foundation for this style is built directly into HTTP 1.1."

Bill de hÓra states it better in his "Confederacy" post:

"the Web is not just the presentation tier anymore; it's becoming a data integration and machine-oriented publishing layer.  The presentation layer is being pushed down to the client machine in the form of AJAX, XUL and Flex."

A new web/middleware layer has been forming, and this is the engine that is driving Web 2.0 and creating a new level of integration and interoperability.

Spread Awareness.

Awesome new video (via Lessig): 

“Humanity Lobotomy” - Net Neutrality Open Source Documentary

What a great month for videos!  I feel inspired to be working in technology again.  We control the future.

Check out more here

A little while back I wrote about finishing my Master's Degree at Boston Universtiy. 

Well... this bad boy arrived in the mail today:

I guess I am now officially educated... or something.

O'Toole's Corollary of Finagle's Law: “The perversity of the Universe tends towards a maximum”

I posted a lengthy entry to the "getting feedback on tools" thread going on at the toolsmith-guild group. 

The thread is discussing how to get feedback and participation going in test tools projects.  This experience report sheds some light on this, but more generally can be applied to getting your open source project noticed and used.  I thought other open source developers might find this useful.

Here is my post:

Hi.. new to the list.

I have some experience in this area I want to comment on:

I had a specific need for a tool to test web services a few years ago ('02-'03). I had been rolling my own tools in various languages for quite a while and had some stuff that I thought others might be interested in; so I setup a SourceForge project and released it in Jan'04. The tool is called WebInject. Not many people used it at first. I just kept adding features and releasing. I was scratching my own itches and using my tool internally at a company I was working for. Eventually I got a few bites and some people started to download it and try it out.

As of today, the project site has served over a quarter million pageviews and the tool has been downloaded over 30,000 times. I've had feedback from all corners of the world.

See the Download Stats

In the early days, *any* bug that was reported, I would dive into immediately.. I was turning around fixes and new releases in crazy time. This personal attention sparked a lot of lengthy email conversations where I was starting to get real feedback. It wasn't the case where I could just sit back and expect quality feedback. I really engaged the users and asked questions. Pretty soon I was getting more feedback than I could reasonably handle. Around this time I started getting some patches sent to me. This accelerated development as you start to get various ideas from other people. A few silly bugs were also cleaned up.. stuff I never would have caught on my own. Encouraging patches is huge.. once someone gets a piece of their own code into your code base, they have an intimate connection.

I was also contacted by someone writing a book that wanted to use my tool as an example. I was really excited about this and put in some serious time getting the tool cleaned up and more useful. I never saw a published copy, but apparently it exists (I saw the PDF drafts :) It was also mentioned in a few testing magazines and articles which was cool for me to see.

I don't maintain WebInject much any more. I think it is cool and useful, but it's a rats nest of Perl code that needs some serious love. There is some stuff that bothers me so much about it that I barely want to touch it sometimes :) I spend all my time on newer tools these days, but I still keep on top of it enough to facilitate others using it and posting patches/updates to it (though I haven't released in over a year I think). Oddly, it has become somewhat entrenched in monitoring systems. Most of the users lately seem to be people running Nagios (open source monitoring system) that need an intelligent web plugin/agent.

Some basic thoughts on open source tool adoption and getting feedback:

• A tool without documentation doesn't exist.. without quality documentation, don't even bother.. nobody will touch it.
  
• If it takes someone more than 10 minutes to install, configure, run, and see the results from a simple test case.. nobody will use it. Once someone sees something work, it is tangible and they become engaged. I achieved this with my tool.. if you download and run it, you are presented with a GUI. Pressing "run" executes a preconfigured sample test case that hits my site. I wanted the most clueless of people to be able to immediately see a green "PASS", and say "hey, cool it actually did something". This satisfaction must come immediately or it will be abandoned. (unless of course you already have a user base that has made it past this hurdle and can encourage others to put in the effort to get it working)
  
• Make your tool easy to get. The online presence of your tool should allow someone to visit the website and immediately download it. I have seen tools released that take 5 levels of navigation before you get to the download options, or don't even have a website or project page somewhere.
  
• Announce your tools.. release early, release often.. who cares.. blog it, forum it, email it, digg it, usenet it, comment about it.. see if anyone has interest. Don't annoyingly spam it, bet let people know it exists. If you announce your tool once only, buried deep in a technical mailing list, don't expect people to come rushing.
  
• Encourage participation. Go out of your way to let people know that you encourage and appreciate feedback, bug reports, offers for collaboration, patches, etc.
  
• You need a forum or a mailing list with a web accessible archive. It has to be super simple to talk to you in public and let others see answers to questions and begin to participate on their own and to offer advice. Using a many-to-one scheme where you just tell people to email you isn't gonna work. It is too private.. people want to know what is going on and hear from each other and offer tips. I have had several thousand posts to my forums for this little tool.
  

- Corey Goldberg

I saw this posted in the toolsmith-guild group (Danny Faught):

"The developers were very surprised to find a whole room full of testers who didn't cringe at the thought of reading and writing code. The rather odd terms Developer-Tester/Tester-Developer emerged from AWTA." (Austin Workshop on Test Automation)

ahh, the elusive "develester"

... and.. the develester algorithm in Python :)