How to Download a File from the Web in Python 3 with a URL Stored in a String

When attempting to download a file from a web server in Python 3, passing a string as the URL may result in an error demanding bytes input. This article presents several methods to circumvent this issue and successfully retrieve the file.

Using urllib.request.urlopen

To obtain the contents of a web page, utilize urllib.request.urlopen(). This function returns a response object, which can then be read into a variable:

import urllib.request

url = 'http://example.com/'
response = urllib.request.urlopen(url)
data = response.read()  # a `bytes` object

Using urlib.request.urlretrieve

The most straightforward way to download and save a file is to utilize the urllib.request.urlretrieve function:

import urllib.request

url = 'http://example.com/file.txt'
urllib.request.urlretrieve(url, 'file.txt')

This method downloads the file from the specified URL and saves it locally as 'file.txt'.

Using urlib.request.urlopen with shutil.copyfileobj

For greater control over the download process, use urllib.request.urlopen() to return a file-like object. This object can then be copied to a real file using shutil.copyfileobj():

import urllib.request
import shutil

url = 'http://example.com/file.zip'
with urllib.request.urlopen(url) as response, open('file.zip', 'wb') as out_file:
    shutil.copyfileobj(response, out_file)

Storing the Download as Bytes

If speed is a priority, the downloaded data can be stored directly into a bytes object and subsequently written to a file, although this is suitable only for small files:

import urllib.request

url = 'http://example.com/icon.png'
with urllib.request.urlopen(url) as response, open('icon.png', 'wb') as out_file:
    data = response.read() # a `bytes` object
    out_file.write(data)

Handling Compressed Files

urllib.request.urlopen() can also be used to handle compressed files, provided the server supports random access:

import urllib.request
import gzip

url = 'http://example.com/archive.gz'
with urllib.request.urlopen(url) as response:
    with gzip.GzipFile(fileobj=response) as uncompressed:
        file_header = uncompressed.read(64) # a `bytes` object

By implementing one of these methods, developers can successfully download files from the web in Python 3, even when the URL is stored as a string.