When working with data sets, a common task I need to do is slurp a csv file into a nested data structure that contains a sequence of lists correlating to the rows and values in the csv file.

For example...

File contents (foo.csv):

          10,20,30,40

19,29,39,55

16,21,31,59
        

Result:

          [['10', '20', '30', '40'],

['19', '29', '39', '55'],

['16', '21', '31', '59']]
        

To accomplish this. you could parse it inside a big honkin' list comprehension and build our structure in one step:

          csv_file = 'foo.csv'

value_lists = [line.split(',') for line in 


[line.strip() for line in open(csv_file, 'r').readlines()]]
        

You could also use the csv module from Python's standard library:

          import csv

csv_file = 'foo.csv'

value_lists = list(csv.reader(open(csv_file, 'r')))
        

The csv module has some useful tools for reading/writing csv files.  Check it out.

Jesse Noller just blogged "Finding Python people is hard":

Here is a good quote regarding the difficulty in finding skilled Test Engineers with Python experience:

"Either you teach QA people automation/test engineering, or you try to find a programmer who wants to learn/do test engineering and teaching them python. It's a hard sell either way. I technically view QA as one discipline, Development as another, but Test Engineering as the Hybrid of the two - and you need a strong background in both."

I have seen lots of QA Engineers and Testers with little to no development/programming experience. This seems to be such a valuable skill; why not learn some? The bar is set really low with today's dynamic languages. Getting into some quick scripting for data manipulation and building test harnesses is not a huge task. If a QA engineer can't learn some simple programming in a week, would you trust his efficiency and technical skills?

I agree with Jesse on this one. We need to see more Test Engineers and Developers In Test. Unfortunately, this hybrid roll often falls through cracks as many people view quality/testing vs. developing as a binary choice.

New Pylot 1.1 release is available
Visit: www.pylot.org/download.html

It contains some minor code cleanup and a new test case recorder contributed by David Solomon. The recorder works with Windows and IE only.

It is a script that launches your web browser and records HTTP requests as you navigate. While it records, it prints Pylot's XML test cases. The test cases are printed to STDOUT, so just redirect it to a file and you will have a valid testcases.xml file to use as Pylot input.

The pylot_recorder script is included in the lib directory of Pylot 1.1.

View the recorder's source code from the SVN trunk:
http://code.google.com/p/pylt/source/browse/trunk/lib/pylot_recorder.py

(It can't handle some complex scenarios, but is useful for recording simple GET and POST requests from web applications)

The Python Cookbook has a recipe for splitting a list into roughly equal-sized pieces:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/425397

In the comments, there are several alterate implementations. Sebastian Hempel has an interesting take on it using slicing for the calculation of the list lengths. It basically looks like this:

          def split_seq(seq, num_pieces): start = 0 for i in xrange(num_pieces):
stop = start + len(seq[i::num_pieces]) yield seq[start:stop] start = stop 
        

This version of the function distributes the remaindered items evenly over the first few splits.

Example Usage:

          seq = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] for seq in split_seq(seq,
3): print seq 
        

This script uses your system's ping utility to send an ICMP ECHO_REQUEST to a list of hosts or devices. It uses a separate thread to ping each host/device. This can be useful for measuring network latency and verifying hosts are alive.

Check out more info here: http://www.goldb.org/python_pinger.html

          #!/usr/bin/env python import re from subprocess import
Popen, PIPE from threading import Thread class Pinger(object): def __init__(self,
hosts): for host in hosts: pa = PingAgent(host) pa.start() class PingAgent(Thread):
def __init__(self, host): Thread.__init__(self) self.host = host def run(self): p
= Popen('ping -n 1 ' + self.host, stdout=PIPE) m = re.search('Average = (.*)ms', p.stdout.read())
if m: print 'Round Trip Time: %s ms -' % m.group(1), self.host else: print 'Error:
Invalid Response -', self.host if __name__ == '__main__': hosts = [ 'www.pylot.org',
'www.goldb.org', 'www.google.com', 'www.this_one_wont_work.com' ] Pinger(hosts) 
        

Output:

          Round Trip Time: 14 ms - www.yahoo.com Round Trip Time:
17 ms - www.goldb.org Round Trip Time: 30 ms - www.google.com Round Trip Time: 82
ms - www.pylot.org Error: Invalid Response - www.this_one_wont_work.com 
        

Note: I only tested this on Windows. To run on other systems, it would only require a one-line change.

"compared to Java code, XML is agile and flexible. Compared to Python code, XML is a boat anchor, a ball and chain."
    - Phillip J. Eby

If you are new to Python and coming from Java (or C#, or other similar statically typed OO language), these classic articles from PJE and Ryan Tomayko are necessary reading:

• Python Is Not Java (PJE)
• Java is not Python, either (PJE)
• Python Interfaces are not Java Interfaces (PJE)
• The Static Method Thing (Tomayko)
• Getters/Setters/Fuxors (Tomayko)

I love this sarcastic rant by Ryan Tomayko from March 2006.  In the article, he rubutts some of James Gosling's comments regarding Java vs. dynamic languages.  Ryan pokes some fun at comments about dynamic languages and why they should be taken seriously; rather than thrown into the "scripting language ghetto".

My favorite part:

"Dealing with questions on dynamic languages:

First, call anything not statically compiled a “scripting language”. Attempt to insinuate that all languages without an explicit compilation step are not to be taken seriously and that they are all equivalently shitty. Best results are achieved when you provide no further explanation of the pros and cons of static and dynamic compilation and/or typing and instead allow the reader to simply assume that there are a wealth of benefits and very few, if any, disadvantages to static compilation. While the benefits of dynamic languages–first realized millions of years ago in LISP and Smalltalk–are well understood in academia, IT managers and Sun certified developers are perfectly accepting of our static = professional / dynamic = amateurish labeling scheme.

This technique is also known to result in dynamic language advocates going absolute bat-shit crazy and making complete fools of themselves. There have been no fewer than three head explosions recorded as a result of this technique.

Also, avoid the term “dynamic language” at all cost. It’s important that the reader not be exposed to the concepts separating scripting languages like bash, MS-DOS batch, and perl-in-the-eighties from general purpose dynamic languages like Ruby, Python, Smalltalk, and Perl present day."

This script will output bytes received and transmitted for a local Windows machine since the last reboot:

           import re from subprocess import Popen, PIPE p = Popen('net
statistics workstation', stdout=PIPE) for line in p.stdout: m = re.search('Bytes received\W+(.*)',
line) if m: print 'Bytes received: %s' % (m.group(1)) m = re.search('Bytes transmitted\W+(.*)',
line) if m: print 'Bytes transmitted: %s' % (m.group(1)) 
        

This script will output the last reboot date/time for a local Windows machine:

           import re from subprocess import Popen, PIPE p = Popen('net
statistics workstation', stdout=PIPE) m = re.search('(\d+/\d+/\d{4}.*[A|P]M)', p.stdout.read())
if m: print 'Last Reboot: %s' % (m.group(1)) 
        

Output:

          >> Last Reboot: 3/1/2008 1:51:41 PM
        

* updated the original script thanks to Ian's comment below

Here is how to zero-pad single digit days or months in a date string:

           date = '3/2/2008' padded_date = time.strftime('%m/%d/%Y',
time.strptime(date,'%m/%d/%Y')) print padded_date 
        

          >> 03/02/2008

        

A palindrome is a sequence that reads the same in either direction.

Here is function I wrote to check if a phrase is a palindrome:

           import re def is_palindrome(txt): txt = re.sub('\W+',
'', txt).lower() return txt == txt[::-1] 
        

           phrase = "Go hang a salami, I'm a lasagna hog" print
is_palindrome(phrase) >> True 
        

I write a lot of command line tools and scripts in Python. Sometimes I need to kick them off remotely. A simple way to do this is to launch a tiny web server that listens for a specific request to start the script.

I add a "WebRequestHandler" class to my script and call it from my main method. There is a "do_something()" method in the class. You call your code from this method.

All you have to do is launch your script and it will sit there and wait for requests. If the request is bad, it spits back a 404 error. If the request path matches what we are looking for (in this case "/foo"), the code is launched.

Now you have an easy way to call your script remotely. Just open a browser and type in the URL: http://your_server/foo, or call it with a tool like 'wget' or 'curl'.

           import BaseHTTPServer class WebRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def do_GET(self): if self.path == '/foo': self.send_response(200) self.do_something()
else: self.send_error(404) def do_something(self): print 'hello world' server = BaseHTTPServer.HTTPServer(('',80),
WebRequestHandler) server.serve_forever() 
        

(this was adapted from a code sample in "Python In A Nutshell" by Alex Martelli)

In many programming languages you can't terminate a thread directly.  Python is no different.  Rather than termintaing a thread from the code that spawned it, you just a pass a flag to the thread that tells it to terminate itself.  Typically a thread will run in a loop, periodically checking this flag so it knows if it should continue or not.  To terminate the thread from the outside, you just set its flag to die.

I was using this idiom in Python by setting a boolean flag in my spawned thread.

So a simplified thread class would look something like this:

           class MyThread(threading.Thread): def __init__(self,
num): threading.Thread.__init__(self) self.running = True self.num = num def stop(self):
self.running = False def run(self): while self.running: print 'hello from thread %d'
% self.num time.sleep(1) 
        

I just read an old post in comp.lang.python that pointed to a recipe in the Python Cookbook that suggests using threading.Event() rather than a simple boolean flag.

So the thread class would look something like this:

           class MyThread(threading.Thread): def __init__(self,
num): threading.Thread.__init__(self) self.stop_event = threading.Event() self.num
= num def stop(self): self.stop_event.set() def run(self): while not self.stop_event.isSet():
print 'hello from thread %d' % self.num time.sleep(1) 
        

They work exactly the same.

I am just wondering what other flexibility threading.Event() gives you, and if there is anything bad about using simple boolean checks to kill threads. I guess I will have to look it up and play around a bit.

I just installed the latest Alpha of Python 3000.

So far so good...

Sending HTTP Requests from a C# program seems unnecessarily hard.  I wrote a small helper class to deal with sending and timing GET requests:
http://www.goldb.org/httpgetcsharp.html

You use it like this:

           public class Program { static void Main(string[] args)
{ HTTPGet req = new HTTPGet(); req.Request("http://www.google.com"); Console.WriteLine(req.StatusLine);
Console.WriteLine(req.ResponseTime); } } 
        

Convert a number of seconds into a human readable time string HH:MM:SS

7046 seconds is: 1 hour 57 mins 26 secs, or 01:57:26

The Function:

          def humanize_time(secs):

mins, secs = divmod(secs, 60)

hours, mins = divmod(mins, 60)

return '%02d:%02d:%02d' % (hours, mins, secs)

        

The Output:

          print humanize_time(7046)

>> 01:57:26

        

The new issue of the Python Papers is out.  It includes a small article I wrote called: Screen Scraping Web Pages

The issue can be downloaded here:  The Python Papers, Volume 2, Issue 4 (pdf)

This tutorial shows how to programmatically retrieve a stock quote from Google Finance.  It uses Python's high level Web API and screen scraping with regular expressions.

I was recently interviewed for the article:
Python Experts - Why They Do Python

I don't think I am even close to an "expert", but it was nice being asked to participate.

Here is a way to unzip files in Python.  If you have a zip containing multiple files, you can unzip it like this:

          import zipfile


fh = open('foo.zip', 'rb')

z = zipfile.ZipFile(fh)

for name in z.namelist():

outfile = open(name, 'wb')

outfile.write(z.read(name))

outfile.close()

fh.close()