The OpenSTA team has announced the release of version 1.4.4

OpenSTA is a distributed software testing architecture designed around CORBA.  The applications that make up the current OpenSTA toolset were designed to be used by performance testing practitioners for web load testing.

Info:
http://portal.opensta.org/index.php?name=News&file=article&sid=51

Download:
http://opensta.org/download.html

Congrats and thanks to Bernie Velivis, Daniel Sutcliffe, Jerome Delemarche for making this release possible.

To all the chart/graph/plot/visualization weenies out there...
Here is a great overview of some modern charting and graphing technologies.

Some options I will be exploring:

• Dojo Charting Engine (AJAX/JavaScript)
• Open Flash Chart (Flash)

This is a module for generating concurrent requests to an HTTP server.  Each thread makes HTTP GET requests to a single URL at the specified interval.  Threads are added over a given rampup time if you want to generate increasing load.  Response times are printed to STDOUT.  Can be used for cursory performance benchmarking or load testing a web resource.

load_generator.py module

sample usage:

#!/usr/bin/env python

from load_generator import LoadManager

lm = LoadManager()
lm.msg = ('www.example.com', '/')
lm.start(threads=5, interval=2, rampup=2)

I love reading old RFC's.

One of my favorites is RFC 1925 - The Twelve Networking Truths:

The Fundamental Truths

1. It Has To Work.
2. No matter how hard you push and no matter what the priority, you can't increase the speed of light.
   (corollary) No matter how hard you try, you can't make a baby in much less than 9 months. Trying to speed this up *might* make it slower, but it won't make it happen any quicker.
3. With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead.
4. Some things in life can never be fully appreciated nor understood unless experienced firsthand. Some things in networking can never be fully understood by someone who neither builds commercial networking equipment nor runs an operational network.
5. It is always possible to agglutinate multiple separate problems into a single complex interdependent solution. In most cases this is a bad idea.
6. It is easier to move a problem around (for example, by moving the problem to a different part of the overall network architecture) than it is to solve it.
   (corollary) It is always possible to add another level of indirection.
7. It is always something.
   (corollary) Good, Fast, Cheap: Pick any two (you can't have all three).
8. It is more complicated than you think.
9. For all resources, whatever it is, you need more.
   (corollary) Every networking problem always takes longer to solve than it seems like it should.
10. One size never fits all.
11. Every old idea will be proposed again with a different name and a different presentation, regardless of whether it works.
12. In protocol design, perfection has been reached not when there is nothing left to add, but when there is nothing left to take away.

I was tipped off about this report in a recent forum discussion.

Report from Forrester Research:
    Evaluating Functional Testing Solutions (Powerpoint)

It is a nice overview of automated test execution and popular functional testing tools. It also gives an overview of the top tool vendors.

A while back, I created a Google Gadget for displaying stock quotes and daily charts.  It was very popular and I was getting more than 12,000 page views per day.  I was scraping data from Google Finance and then received a takedown notice from Google.  So.. I took down the service that I was running.

Well... it seems the gadget was really popular and I have received lots of emails asking to revive the service.

And now... it's baaaack (using data from Yahoo)

- Add my gadget to your Google Personalized Homepage
- Add my gadget to your own web page

Have at it!

This is embarrassing.

After a long day of hacking, I fell asleep at my desk.  This has *never* happened to me at work.

Of course, somebody got a picture of it on their camera phone:

I found a snippet to draw bar graphs in Python using Tk:
http://www.daniweb.com/code/snippet583.html

The output looks like this:

Here is a modified version that creates a bar graph in a Tk panel:

import Tkinter as tk

def graph_points(seq, width=375, height=325):
    root = tk.Tk()
    c = tk.Canvas(root, width=width, height=height, bg='white')
    c.pack()
    y_stretch = 15
    y_gap = 20
    x_stretch = 10
    x_width = 20
    x_gap = 20
    for x, y in enumerate(data):
        x0 = x * x_stretch + x * x_width + x_gap
        y0 = height - (y * y_stretch + y_gap)
        x1 = x * x_stretch + x * x_width + x_width + x_gap
        y1 = height - y_gap
        c.create_rectangle(x0, y0, x1, y1, fill="red")
        c.create_text(x0+2, y0, anchor=tk.SW, text=str(y))
    root.mainloop()

data = (18, 15, 10, 7, 5, 4, 2, 5, 8, 10, 13)
graph_points(data)

OK.. looks like I need to plunk down $400 to get an XO.  When you buy one, they will give one away to a child in a developing nation.  Do some good, and get my own OLPC?  Too good to pass up.

For 2 weeks only, starting Nov. 12:  www.xogiving.org

My first programming course was as an undergrad freshman in 1993. It was the basic introductory programming class for CS majors. The course was pretty difficult and was a great filter to separate the real CS students from the wannabes. About one-third of the students dropped the class, and out of the remaining two-thirds, many changed majors after this course was complete.

The course was taught using Scheme, with SICP as the text book. We programmed on a VAX cluster with Ultrix (DEC's Unix flavor) as the Operating System. We had to learn the Unix shells, VI, and all sorts of fun stuff to get us up and running.

The computer lab ("the cluster") was a large sterile room with rows of green-screen dumb terminals. I remember our professor told us that the VAX had 128 MB of memory and I was blown away by how huge that was (my rippin' fast PC had 4 MB at the time).

Spending hours in the computer lab was no fun at all. But I was one of the lucky ones. I owned a brand new 486-DX33 PC running Windows 3.1. I had a blazing fast 14.4 bps Zoom modem and could use Procomm Plus to dial into the VAX and program from the comfort of my own dorm room. I also found a Scheme interpreter that ran on DOS, giving me further options to do my work offline.

The programming assignments were brutal. All-nighters were the norm. Collaboration on the assignments was encouraged, but we were all expected to turn in our own original work. I found a fellow student that I got along with well and we decided to work together (unfortunately, I don't even remember his name... all I remember is that he was a lot smarter than me).

So the basic workflow was that we would get together, work out the basics of the assignment, get most of the algorithms working, then each take the code and finish it on our own. Since I had the bad-ass PC, we would work in my room. Two things quickly became apparent: He was a much better programmer than me, but I had a better eye for subtle details and debugging. Eventually we settled into a pattern where he would do the programming and I would look over his shoulder to give advice and input. Every few minutes, he would shoot a copy of the code to my dot-matrix printer. I would grab it, go through it line by line, and mark errors with my red pen and hand-write parts of the code that weren't correct. I would then hand the printout back to him and let him enter the changes. We iterated like this until we had all of the core code working.

For some reason, that instinct for attention to detail and debugging has always stuck with me. Because of that, my career has always been influenced by testing. I am a develop/tester, rather than just a developer. Most of the impact I have had in all of my jobs is from creating test tools and bringing in new ways to test software.

Just an interesting observation. I wonder how many others were naturally drawn to testing as soon as they started writing code?

Last week I wrote a small Python module for retrieving stock prices.

It used screen scraping to get data from Google Finance.  Yahoo offers stock data in a much more digestible form which allowed me to get values without screen scraping and regular expressions.  So, I wrote a module based around this.

This new module is much more comprehensive and exposes a Python API for retrieving all sorts of stock data from Yahoo Finance.

My ystockquote module provides a Python API for retrieving stock data from Yahoo Finance.  This module contains the following functions:

• get_all(symbol)
• get_price(symbol)
• get_change(symbol)
• get_volume(symbol)
• get_avg_daily_volume(symbol)
• get_stock_exchange(symbol)
• get_market_cap(symbol)
• get_book_value(symbol)
• get_ebitda(symbol)
• get_dividend_per_share(symbol)
• get_dividend_yield(symbol)
• get_earnings_per_share(symbol)
• get_52_week_high(symbol)
• get_52_week_low(symbol)
• get_50day_moving_avg(symbol)
• get_200day_moving_avg(symbol)
• get_price_earnings_ratio(symbol)
• get_price_earnings_growth_ratio(symbol)
• get_price_sales_ratio(symbol)
• get_price_book_ratio(symbol)
• get_short_ratio(symbol)

Sample Usage:

>>> import ystockquote
>>> print ystockquote.get_price('GOOG')
529.46
>>> print ystockquote.get_all('MSFT')
{'stock_exchange': '"NasdaqNM"', 'market_cap': '268.6B', 
'200day_moving_avg': '29.2879', '52_week_high': '31.84', 
'price_earnings_growth_ratio': '1.45', 'price_sales_ratio': '5.33',
'price': '28.65', 'earnings_per_share': '1.423', 
'50day_moving_avg': '28.7981', 'avg_daily_volume': '55579700',
'volume': '25330856', '52_week_low': '26.48', 'short_ratio': '1.60', 
'price_earnings_ratio': '28.65', 'dividend_yield': '1.38', 
'dividend_per_share': '0.40', 'price_book_ratio': '8.76', 
'ebitda': '20.441B', 'change': '-0.39', 'book_value': '3.315'}

The module is available here:  http://www.goldb.org/ystockquote.html

I just wrote a tiny Python module for programmatically retrieving stock quotes from Google Finance:

The module:

import urllib
import re

def get_quote(symbol):
    base_url = 'http://finance.google.com/finance?q='
    content = urllib.urlopen(base_url + symbol).read()
    m = re.search('class="pr".*?>(.*?)<', content)
    if m:
        quote = m.group(1)
    else:
        quote = 'no quote available for: ' + symbol
    return quote

Sample usage:

#!/usr/bin/env python

import stockquote

print stockquote.get_quote('goog')

Output:

>> 529.56

"Open Source"

In the words of Inigo Montoya [The Princess Bride]:  "You keep using that word.  I do not think it means, what you think it means."

A few months back, Radview Software announced that they are releasing an open source version of WebLOAD, their web performance and load testing tool.  I was very excited about this and thought it was a fantastic move that would have a big impact in the test tool market.  I am a performance engineer and a huge Free/Open Source Software advocate, so I love to see companies in the space that interests me most come around to embrace openness.

In their press release, Radview stated:

"WebLOAD Open Source, licensed under the GNU Public License (GPL) version 2, is based on WebLOAD, the company's flagship product that is already deployed at 1,600 sites. Immediately available for free download and use, WebLOAD is a commercial-grade open source project with more than 250 engineering years of product development."

Ok cool.. they used the GPL and opened up the whole shebang.  Wow, this company actually "gets it"... right?

Umm.. not quite.

If you look through the source code that is available for WebLOAD Open Source, you will notice that only code for a small subset of the product is available.  In actuality, WebLOAD Open Source is a partially proprietary tool which is marketed as Open Source Software.  The software has significant limitations in functionality and scalability.  The source code which needs to be modified to remove these restrictions is not distributed.  So what we are left with is a crippled version of the tool.

In a recent post to the WebLOAD OS Forum, someone asked to see the source code for "proxynator", which is the recording feature in WebLOAD.

The response from the Forum Admin (Amir Shoval, a Radview employee) was this:

"Currently the source code for the proxynator is not available as part of the open source code of WebLOAD."

This is in direct contradiction to what their website states:

"WebLOAD Open Source introduces a unified script authoring environment for recording, editing and debugging."

When further probed about this, he stated the following:

WebLOAD Open Source is dual licensed:
  1. the WebLOAD Load Engine is totally open sourced and hence is licensed under the GPL
  2. but the complete WebLOAD is still licensed under a proprietary license, which grants free usage in WebLOAD Open Source.

wait.. wait.. WHAT?
I thought the press release said "licensed under the GNU Public License", and "WebLOAD Open Source is a fully functional, commercial-grade performance testing product"?  Nowhere on their website or marketing materials to they talk about this dual licensing and limited availability of source code.

Now, if we look at the End User License Agreement (EULA) that applies to WebLOAD Open Source, it gets worse:

2. License Restrictions. This License does not permit you or any third party to:
(i) modify, translate, reverse engineer, decompile, disassemble (except to the extent that this restriction is expressly prohibited by law) or otherwise attempt to discover the source code of all or any portion of the Software;
(ii) modify, translate or create derivative works of all or any portion of the Software;
(iii) copy the Software (other than a single copy solely for back-up or archival purposes);
(iv) rent, lease, sell, offer to sell, distribute, or otherwise transfer rights to the Software;

OK... so we have an "open source" product that is actually dual licensed, where a large portion of the toolset is proprietary.  And furthermore, by accepting the EULA, you give up all of your rights that were granted under the GPL.  Huh?

So... to reiterate, WebLOAD Open Source is *not* open source.  A subset of it is open source: the [crippled] load engine.  Contrary to what their press release and website says, it contains proprietary components that are released in binary form with no source code.  It is rather disappointing to see a company jump on the bandwagon of open source without respecting the freedom that is supposed to come with it.

In conclusion: Is WebLOAD Open Source currently Open Source?  No
Will WebLOAD Open Source actually become Open Source?  Well.. that's up to Radview

Hey, I'm all for Freedom.  I applaud Radview for any of the code they released under the GPL.  But lets be fair, if you want to call your product open source and reap any benefits that come along with that... you gotta walk the walk.

I often need to automate tasks in web based applications.  I like to do this at the protocol level by simulating a real user's interactions via HTTP.  Python comes with two built-in modules for this: urllib (higher level Web interface) and httplib (lower level HTTP interface).

However, I usually don't use either of these.  I prefer to use Joe Gregario's excellent httplib2 module (btw, I really wish this could make its way into Python's Standard Library).  It is a much richer library and has a lot of nice features for dealing with HTTP.  

When automating something, you often need to "login" to maintain some sort of session/state with the server.  This is usually achieved with form-based authentication. You post a form to the server, and it responds with a cookie in the incoming HTTP header.  You need to pass this cookie back to the server in subsequent requests to maintain state or to keep a session alive.

Here is an example of how to deal with cookies when doing your HTTP Post.

First, lets import the modules we will use:

import urllib
import httplib2

Now, lets define the data we will need: In this case, we are doing a form post with 2 fields representing a username and a password.

url = 'http://www.example.com/login'   
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}

Now we can send the HTTP request:

http = httplib2.Http()
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))

At this point, our "response" variable contains a dictionary of HTTP header fields that were returned by the server. If a cookie was returned, you would see a "set-cookie" field containing the cookie value. We want to take this value and put it into the outgoing HTTP header for our subsequent requests:

headers['Cookie'] = response['set-cookie']

Now we can send a request using this header and it will contain the cookie, so the server can recognize us.

So... here is the whole thing in a script. We login to a site and then make another request using the cookie we received:

#!/usr/bin/env python

import urllib
import httplib2

http = httplib2.Http()

url = 'http://www.example.com/login'   
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))

headers = {'Cookie': response['set-cookie']}

url = 'http://www.example.com/home'   
response, content = http.request(url, 'GET', headers=headers)

wow... congrats to Guido and everyone else involved.

get it here:

Posting a link to your email address on your website is inviting spiders to grab it and spam you.  I get around this by using a JavaScript snippet so my email address link renders on the client but not in the actual HTML source.

While I'm sure there are much better ways to do this, this has been successful for me so far.

This script creates a "mailto" link for my email address:

<script type="text/javascript">
    <!--
    var name = "corey"
    var emailHost = "goldb.org"
    document.write("<a href=" + "mail" + "to:" + name + "@" + emailHost + ">"
        + name + "@" + emailHost + "</a>")
    //-->
</script>

(this post is completely off-topic. stay tuned for regularly scheduled software and technology stuff)

With the rough weeks in US markets (subprime fiasco, credit crunch.. ouch), I thought it would be a good time to talk about the only investment strategy that really works.  Yes, it is easy. Yes, it is boring.  Yes, it works.

Ever since I finished my Bachelor's in Economics (10 years ago), I have been consumed with following financial markets and investing any shred of money I come upon.  I've had ups and downs, but I like to consider myself a pretty savvy investor.  My career as a software engineer has also kept me close to finance.  I worked in institutional financial services for 8 years as a developer/tester.  I helped develop trading platforms, market research databases, quantitative analysis tools, buy-side/sell-side communication platforms, pre-trade negotiation messaging, etc.  I like to think that I have a pretty good idea of how markets work.

So in this huge multi-billion dollar game, how is the personal investor supposed to navigate his way to profit?

Well.. actually it's quite easy.

There are 2 simple questions you must first answer:
1. What is the time-frame in which you need access to your money? (next week? next year? 10 years? retirement?)
2. How much risk and volatility are you comfortable with?

The Answer:
Asset Allocation

... not fundamental analysis, not technical analysis, not market trending, not tips from brokers and analysts ... but straight up asset allocation.

There are numerous ways to allocation your assets:  Equities (Large Cap, Small Cap, Value/Growth, Domestic/International), Fixed Income, REITs, Money Instruments, etc.  How you choose to allocate depends on your answers to the questions above.

All The Reading You Need:

I've read dozens of investment books.  The majority of them are garbage.  If you want a good recommendation, pick one of the following books to read and study it thoroughly... it's as simple as that.

• A Random Walk Down Wall Street (Burton G. Malkiel)
• The Four Pillars of Investing (William J. Bernstein)

They are essentially the same book. They cover nearly identical material, history, commentary, and strategy.  They both highly favor the random walk theory and passive index investing.  Both authors are excellent, so just pick one or both.  Study these closely and you have all the information you will ever need.

and one parting tip:
If you are constantly worrying about the stock market, you are over-invested. Sell until you can sleep at night.

(Pylot is the open source web performance/load test tool that I am developing)

When a test run is finished, a report is automatically generated to summarize the test results. It includes various statistics and graphs on response times and throughput from the run. A sample of the results report can be seen here:

http://www.pylot.org/samples/results/results.html

Pylot also writes results to CSV files so you can import them into your favorite spreadsheet to crunch numbers, generate statistics, and create graphs. I have been working

I have also been working on the GUI. Here is the latest:

http://www.pylot.org/samples/ui/pylot_ui_screenshot_2007_08_20.png

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool
• Pylot - Dev Update #4 - Web Performance/Load Test Tool (New Name and Defining Test Cases)
• Pylot - Dev Update #5 - Web Performance/Load Test Tool (Graphs With MatPlotlib)

highscalability.com pointed out an article from Pingdom titled "What the Web’s most popular sites are running on", which shows the results from an infrastructure survey of 7 popular web "super sites" (TechCrunch, FeedBurner, iStockPhoto, YouSendIt, Meebo, Vimeo, Alexaholic).

There is some intriguing information in the survey, but one thing stood out.  Apparently, TechCrunch runs on 1 web server and 1 database server, with no server clustering:

I almost find this hard to believe.  They serve > 1 million unique visitors per month off of this?

In a recent post: "What does your favorite text editor say about you", the author lists popular text editors and what they say about their users.  Here is the Editor or IDE I use with various programming languages:

Python:  SciTE
Perl:  SciTE
C#:  Visual Studio
Java:  Eclipse

I do all of my writing and a large portion of my programming in a plain old text editor.  Most of the code I write is in Python.  I love using a lightweight text editor instead of a big bloated IDE.  So... I pretty much live inside a text editor.

... and I love SciTE.  It rocks equally on Windows and GNU/Linux.  So what does this say about me?

SciTE:
"Your text editor is lightweight, full featured, extensible and cross platform. In addition, it can work as a stand-alone executable which requires no installation. Fits perfectly with all your other portable tools on your USB thumb drive. You also love how SciTE let’s you write Lua scripts to extend it’s functionality. You take your text editor choice very seriously. You like tinkering, and minimalistic, portable applications."

(via Matt Asay)

I found this pretty interesting.

Lines Of Code (LOC) in some popular Open Source code bases:

• Linux Kernel: 6 million
• Sun Java Development Kit: 6.5 million
• Sun StarOffice: 9 million
• Eclipse: 17 million

Keeping up to date with what's going on in Linux kernel development and what to expect in the future isn't always easy.  The Linux Kernel Mailing List has very high traffic and is very technical.

Luckily, the Linux Foundation has provided the Linux Weather Forecast, which provides a high level roadmap of what's going on in kernel development and where it's all going.

Pretty handy for people depending on future features as well as casual observers that want to keep up to date with major kernel development activities.

Earlier this week, SCO finally lost the infamous Linux copyright infringement case against IBM.  The judge ruled Unix copyrights belong to Novell, not SCO.  A lot of people have forgotten about this case; but when it was originally filed, it really spooked a lot of Free software developers and Linux advocates.

After the ruling, Wall Street punished SCO's stock price accordingly.

5-Day stock price chart for SCOX:

Ouch.  Decimated.  I'm actually surprised it didn't get hammered more... Time to short this rag?

We performance practioners love our graphs!  Visualizing data is helpful in analyzing performance results.  Sometimes a quick glance at a graph provides better understanding than a mound of raw or summarized data.  Pylot's Results Reporting feature creates graphs of response times (latency) and Throughput.

For the graphing toolkit, Pylot uses Matplotlib to produce fancy graphs like these:

Matplotlib allows you to graph data from Python. Here is a simple script that gives a glimpse of how a line/marker graph is created as a png image:

#!/usr/bin/env python

from pylab import *  # Matplotlib

def main():
    # sequence of data points to graph (x, y coordinates)
    points = [(1, 3), (2, 6), (3, 2), (4, 5)]
    graph(points)
   
   
def graph(points):
    fig = figure(figsize=(6, 2))  # image dimensions  
    ax = fig.add_subplot(111)
    ax.grid(True, color='#666666')
    xticks(size='x-small')
    yticks(size='x-small')
    x_seq = [item[0] for item in points] 
    y_seq = [item[1] for item in points] 
    ax.plot(x_seq, y_seq, 
        color='blue', linestyle='-', linewidth=1.0, marker='o', 
        markeredgecolor='blue', markerfacecolor='yellow', markersize=2.0)
    savefig('graph.png') 
    
    
if __name__ == '__main__':
    main()

The output looks like this:

Related:

• PyLT - Scratching My Itch - New Web Performance/Load Test Tool (Open Source)
• PyLT - Dev Update #1 - Web Performance/Load Test Tool
• PyLT - Dev Update #2 - Web Performance/Load Test Tool
• PyLT - Dev Update #3 - Web Performance/Load Test Tool
• Pylot - Dev Update #4 - Web Performance/Load Test Tool (New Name and Defining Test Cases)

I have the opportunity to spread Python to some junior/newbie programmers. In doing so, I wanted to compile a concise list of reccomended learning materials. The intended audience is someone who has a basic familiarity with programming but no specific Python experience.

There are a ton of books and online materials available, but where should you start? Here is my very brief list:

First Book:

• Learning Python (O'Reilly - by Mark Lutz, David Ascher)

Python Tutorials Online:

• Official Python Tutuorial (by Guido van Rossum)
• Dive Into Python (by Mark Pilgrim)
• Instant Python (by Magnus Lie Hetland)