Dive Into Python

Example 11.17. The openanything function

This function is defined in openanything.py.

def openAnything(source, etag=None, lastmodified=None, agent=USER_AGENT):
    # non-HTTP code omitted for brevity
    if urlparse.urlparse(source)[0] == 'http':                                       
        # open URL with urllib2                                                     
        request = urllib2.Request(source)                                           
        request.add_header('User-Agent', agent)                                      
        if etag:                                                                    
            request.add_header('If-None-Match', etag)                                
        if lastmodified:                                                            
            request.add_header('If-Modified-Since', lastmodified)                    
        request.add_header('Accept-encoding', 'gzip')                                
        opener = urllib2.build_opener(SmartRedirectHandler(), DefaultErrorHandler()) 
        return opener.open(request)                                                  

	urlparse is a handy utility module for, you guessed it, parsing URLs. It's primary function, also called urlparse, takes a URL and splits it into a tuple of (scheme, domain, path, params, query string parameters, and fragment identifier). Of these, the only thing you care about is the scheme, to make sure that you're dealing with an HTTP URL (which urllib2 can handle).
	You identify yourself to the HTTP server with the User-Agent passed in by the calling function. If no User-Agent was specified, you use a default one defined earlier in the openanything.py module. You never use the default one defined by urllib2.
	If an ETag hash was given, send it in the If-None-Match header.
	If a last-modified date was given, send it in the If-Modified-Since header.
	Tell the server you would like compressed data if possible.
	Build a URL opener that uses both of the custom URL handlers: SmartRedirectHandler for handling 301 and 302 redirects, and DefaultErrorHandler for handling 304, 404, and other error conditions gracefully.
	That's it! Open the URL and return a file-like object to the caller.

Example 11.18. The fetch function

This function is defined in openanything.py.

def fetch(source, etag=None, last_modified=None, agent=USER_AGENT):  
    '''Fetch data and metadata from a URL, file, stream, or string'''
    result = {}                                                      
    f = openAnything(source, etag, last_modified, agent)              
    result['data'] = f.read()                                         
    if hasattr(f, 'headers'):                                        
        # save ETag, if the server sent one                          
        result['etag'] = f.headers.get('ETag')                        
        # save Last-Modified header, if the server sent one          
        result['lastmodified'] = f.headers.get('Last-Modified')       
        if f.headers.get('content-encoding', '') == 'gzip':           
            # data came back gzip-compressed, decompress it          
            result['data'] = gzip.GzipFile(fileobj=StringIO(result['data']])).read()
    if hasattr(f, 'url'):                                             
        result['url'] = f.url                                        
        result['status'] = 200                                       
    if hasattr(f, 'status'):                                          
        result['status'] = f.status                                  
    f.close()                                                        
    return result                                                    

	First, you call the openAnything function with a URL, ETag hash, Last-Modified date, and User-Agent.
	Read the actual data returned from the server. This may be compressed; if so, you'll decompress it later.
	Save the ETag hash returned from the server, so the calling application can pass it back to you next time, and you can pass it on to openAnything, which can stick it in the If-None-Match header and send it to the remote server.
	Save the Last-Modified date too.
	If the server says that it sent compressed data, decompress it.
	If you got a URL back from the server, save it, and assume that the status code is 200 until you find out otherwise.
	If one of the custom URL handlers captured a status code, then save that too.

Example 11.19. Using openanything.py

>>> import openanything
>>> useragent = 'MyHTTPWebServicesApp/1.0'
>>> url = 'https://book.diveintopython.org/redir/example301.xml'
>>> params = openanything.fetch(url, agent=useragent)              
>>> params                                                         
{'url': 'http://diveintomark.org/xml/atom.xml', 
'lastmodified': 'Thu, 15 Apr 2004 19:45:21 GMT', 
'etag': '"e842a-3e53-55d97640"', 
'status': 301,
'data': '<?xml version="1.0" encoding="iso-8859-1"?>
<feed version="0.3"
<-- rest of data omitted for brevity -->'}
>>> if params['status'] == 301:                                    
...     url = params['url']
>>> newparams = openanything.fetch(
...     url, params['etag'], params['lastmodified'], useragent)    
>>> newparams
{'url': 'http://diveintomark.org/xml/atom.xml', 
'lastmodified': None, 
'etag': '"e842a-3e53-55d97640"', 
'status': 304,
'data': ''}                                                        

	The very first time you fetch a resource, you don't have an ETag hash or Last-Modified date, so you'll leave those out. (They're optional parameters.)
	What you get back is a dictionary of several useful headers, the HTTP status code, and the actual data returned from the server. openanything handles the gzip compression internally; you don't care about that at this level.
	If you ever get a 301 status code, that's a permanent redirect, and you need to update your URL to the new address.
	The second time you fetch the same resource, you have all sorts of information to pass back: a (possibly updated) URL, the ETag from the last time, the Last-Modified date from the last time, and of course your User-Agent.
	What you get back is again a dictionary, but the data hasn't changed, so all you got was a 304 status code and no data.