I often need to automate tasks in web based applications. I like to do this at the protocol level by simulating a real user's interactions via HTTP. Python comes with two built-in modules for this:
urllib (higher level Web interface) and httplib (lower level HTTP interface).
However, I usually don't use either of these. I prefer to use Joe Gregario's excellent httplib2 module (btw, I really wish this could make its way into Python's Standard Library). It is a much richer library and has a lot of nice features for dealing with HTTP.
When automating something, you often need to "login" to maintain some sort of session/state with the server. This is usually achieved with form-based authentication. You post a form to the server, and it responds with a cookie in the incoming HTTP header. You need to pass this cookie back to the server in subsequent requests to maintain state or to keep a session alive.
Here is an example of how to deal with cookies when doing your HTTP Post.
First, lets import the modules we will use:
import urllib
import httplib2
Now, lets define the data we will need: In this case, we are doing a form post with 2 fields representing a username and a password.
url = 'http://www.example.com/login'
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
Now we can send the HTTP request:
http = httplib2.Http()
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))
At this point, our "response" variable contains a dictionary of HTTP header fields that were returned by the server. If a cookie was returned, you would see a "set-cookie" field containing the cookie value. We want to take this value and put it into the outgoing HTTP header for our subsequent requests:
headers['Cookie'] = response['set-cookie']
Now we can send a request using this header and it will contain the cookie, so the server can recognize us.
So... here is the whole thing in a script. We login to a site and then make another request using the cookie we received:
#!/usr/bin/env python
import urllib
import httplib2
http = httplib2.Http()
url = 'http://www.example.com/login'
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))
headers = {'Cookie': response['set-cookie']}
url = 'http://www.example.com/home'
response, content = http.request(url, 'GET', headers=headers)