How to Upload Files with Python’s requests Library
Python is supported by many libraries which simplify data transfer over HTTP. The requests library is one of the most popular Python packages as it’s heavily used in web scraping. It’s also popular for interacting with servers! The library makes it easy to upload data in a popular format like JSON, but also makes it easy to upload files as well.
In this tutorial, we will take a look at how to upload files using Python’s requests library. The article will start by covering the requests library and the post() function signature. Next, we will cover how to upload a single file using the requests package. Last but not least, we upload multiple files in one request.
Uploading a Single File with Python’s Requests Library
This tutorial covers how to send the files, we’re not concerned about how they’re created. To follow along, create three files called my_file.txt , my_file_2.txt and my_file_3.txt .
The first thing we need to do is install our the request library in our workspace. While not necessary, it’s recommended that you install libraries in a virtual environment:
Activate the virtual environment so that we would no longer impact the global Python installation:
Now let’s install the requests library with pip :
Create a new file called single_uploader.py which will store our code. In that file, let’s begin by importing the requests library:
Now we’re set up to upload a file! When uploading a file, we need to open the file and stream the content. After all, we can’t upload a file we don’t have access to. We’ll do this with the open() function.
The open() function accepts two parameters: the path of the file and the mode. The path of the file can be an absolute path or a relative path to where the script is being run. If you’re uploading a file in the same directory, you can just use the file’s name.
The second argument, mode, will take the «read binary» value which is represented by rb . This argument tells the computer that we want to open the file in the read mode, and we wish to consume the data of the file in a binary format:
test_file = open("my_file.txt", "rb") Note: it’s important to read the file in binary mode. The requests library typically determines the Content-Length header, which is a value in bytes. If the file is not read in bytes mode, the library may get an incorrect value for Content-Length , which would cause errors during file submission.
For this tutorial, we’ll make requests to the free httpbin service. This API allows developers to test their HTTP requests. Let’s create a variable that stores the URL we’ll post our files to:
test_url = "http://httpbin.org/post" We now have everything to make the request. We’ll use the post() method of the requests library to upload the file. We need two arguments to make this work: the URL of the server and files property. We’ll also save the response in a variable, write the following code:
test_response = requests.post(test_url, files = "form_field_name": test_file>) The files property takes a dictionary. The key is the name of the form field that accepts the file. The value is the bytes of the opened file you want to upload.
Normally to check if your post() method was successful we check the HTTP status code of the response. We can use the ok property of the response object, test_url . If it’s true, we’ll print out the response from the HTTP server, in this case, it will echo the request:
Free eBook: Git Essentials
Check out our hands-on, practical guide to learning Git, with best-practices, industry-accepted standards, and included cheat sheet. Stop Googling Git commands and actually learn it!
if test_response.ok: print("Upload completed successfully!") print(test_response.text) else: print("Something went wrong!") Let’s try it out! In the terminal, execute your script with the python command:
Your output would be similar to this:
Upload completed successfully! < "args": <>, "data": "", "files": < "form_field_name": "This is my file\nI like my file\n" >, "form": <>, "headers": < "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "189", "Content-Type": "multipart/form-data; boundary=53bb41eb09d784cedc62d521121269f8", "Host": "httpbin.org", "User-Agent": "python-requests/2.25.0", "X-Amzn-Trace-Id": "Root=1-5fc3c190-5dea2c7633a02bcf5e654c2b" >, "json": null, "origin": "102.5.105.200", "url": "http://httpbin.org/post" > As a sanity check, you can verify the form_field_name value matches what’s in your file.
Uploading Multiple Files with Python’s requests Library
Uploading multiple files using requests is quite similar to a single file, with the major difference being our use of lists. Create a new file called multi_uploader.py and the following setup code:
import requests test_url = "http://httpbin.org/post" Now create a variable called test_files that’s a dictionary with multiple names and files:
test_files = < "test_file_1": open("my_file.txt", "rb"), "test_file_2": open("my_file_2.txt", "rb"), "test_file_3": open("my_file_3.txt", "rb") > Like before, the keys are the names of the form fields and the values are the files in bytes.
We can also create our files variables as a list of tuples. Each tuple contains the name of the form field accepting the file, followed by the file’s contents in bytes:
test_files = [("test_file_1", open("my_file.txt", "rb")), ("test_file_2", open("my_file_2.txt", "rb")), ("test_file_3", open("my_file_3.txt", "rb"))] Either works so choose whichever one you prefer!
Once the list of files is ready, you can send the request and check its response like before:
test_response = requests.post(test_url, files = test_files) if test_response.ok: print("Upload completed successfully!") print(test_response.text) else: print("Something went wrong!") Execute this script with the python command:
Upload completed successfully! < "args": <>, "data": "", "files": < "test_file_1": "This is my file\nI like my file\n", "test_file_2": "All your base are belong to us\n", "test_file_3": "It's-a me, Mario!\n" >, "form": <>, "headers": < "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "470", "Content-Type": "multipart/form-data; boundary=4111c551fb8c61fd14af07bd5df5bb76", "Host": "httpbin.org", "User-Agent": "python-requests/2.25.0", "X-Amzn-Trace-Id": "Root=1-5fc3c744-30404a8b186cf91c7d239034" >, "json": null, "origin": "102.5.105.200", "url": "http://httpbin.org/post" > Good job! You can upload single and multiple files with requests !
Conclusion
In this article, we learned how to upload files in Python using the requests library. Where it’s a single file or multiple files, only a few tweaks are needed with the post() method. We also verified our response to ensure that our uploads were successful.
Fetch a file from a local url with Python requests?
I am using Python’s requests library in one method of my application. The body of the method looks like this:
def handle_remote_file(url, **kwargs): response = requests.get(url, . ) buff = StringIO.StringIO() buff.write(response.content) . return True I’d like to write some unit tests for that method, however, what I want to do is to pass a fake local url such as:
class RemoteTest(TestCase): def setUp(self): self.url = 'file:///tmp/dummy.txt' def test_handle_remote_file(self): self.assertTrue(handle_remote_file(self.url)) When I call requests.get with a local url, I got the KeyError exception below:
requests.get('file:///tmp/dummy.txt') /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/urllib3/poolmanager.pyc in connection_from_host(self, host, port, scheme) 76 77 # Make a fresh ConnectionPool of the desired type 78 pool_cls = pool_classes_by_scheme[scheme] 79 pool = pool_cls(host, port, **self.connection_pool_kw) 80 KeyError: 'file' The question is how can I pass a local url to requests.get?
PS: I made up the above example. It possibly contains many errors.
As @WooParadog explained requests library doesn’t know how to handle local files. Although, current version allows to define transport adapters.
Therefore you can simply define you own adapter which will be able to handle local files, e.g.:
from requests_testadapter import Resp class LocalFileAdapter(requests.adapters.HTTPAdapter): def build_response_from_file(self, request): file_path = request.url[7:] with open(file_path, 'rb') as file: buff = bytearray(os.path.getsize(file_path)) file.readinto(buff) resp = Resp(buff) r = self.build_response(request, resp) return r def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None): return self.build_response_from_file(request) requests_session = requests.session() requests_session.mount('file://', LocalFileAdapter()) requests_session.get('file://') I’m using requests-testadapter module in the above example.
Here’s a transport adapter I wrote which is more featureful than b1r3k’s and has no additional dependencies beyond Requests itself. I haven’t tested it exhaustively yet, but what I have tried seems to be bug-free.
import requests import os, sys if sys.version_info.major < 3: from urllib import url2pathname else: from urllib.request import url2pathname class LocalFileAdapter(requests.adapters.BaseAdapter): """Protocol Adapter to allow Requests to GET file:// URLs @todo: Properly handle non-empty hostname portions. """ @staticmethod def _chkpath(method, path): """Return an HTTP status for the given filesystem path.""" if method.lower() in ('put', 'delete'): return 501, "Not Implemented" # TODO elif method.lower() not in ('get', 'head'): return 405, "Method Not Allowed" elif os.path.isdir(path): return 400, "Path Not A File" elif not os.path.isfile(path): return 404, "File Not Found" elif not os.access(path, os.R_OK): return 403, "Access Denied" else: return 200, "OK" def send(self, req, **kwargs): # pylint: disable=unused-argument """Return the file specified by the given request @type req: C @todo: Should I bother filling `response.headers` and processing If-Modified-Since and friends using `os.stat`? """ path = os.path.normcase(os.path.normpath(url2pathname(req.path_url))) response = requests.Response() response.status_code, response.reason = self._chkpath(req.method, path) if response.status_code == 200 and req.method.lower() != 'head': try: response.raw = open(path, 'rb') except (OSError, IOError) as err: response.status_code = 500 response.reason = str(err) if isinstance(req.url, bytes): response.url = req.url.decode('utf-8') else: response.url = req.url response.request = req response.connection = self return response def close(self): pass (Despite the name, it was completely written before I thought to check Google, so it has nothing to do with b1r3k’s.) As with the other answer, follow this with:
requests_session = requests.session() requests_session.mount('file://', LocalFileAdapter()) r = requests_session.get('file:///path/to/your/file') packages/urllib3/poolmanager.py pretty much explains it. Requests doesn’t support local url.
pool_classes_by_scheme = < 'http': HTTPConnectionPool, 'https': HTTPSConnectionPool, > The easiest way seems using requests-file.
“Requests-File is a transport adapter for use with the Requests Python library to allow local filesystem access via file:// URLs.”
This in combination with requests-html is pure magic 🙂
In a recent project, I’ve had the same issue. Since requests doesn’t support the “file” scheme, I’ll patch our code to load the content locally. First, I define a function to replace requests.get :
def local_get(self, url): "Fetch a stream from local files." p_url = six.moves.urllib.parse.urlparse(url) if p_url.scheme != 'file': raise ValueError("Expected file scheme") filename = six.moves.urllib.request.url2pathname(p_url.path) return open(filename, 'rb') Then, somewhere in test setup or decorating the test function, I use mock.patch to patch the get function on requests:
@mock.patch('requests.get', local_get) def test_handle_remote_file(self): . This technique is somewhat brittle — it doesn’t help if the underlying code calls requests.request or constructs a Session and calls that. There may be a way to patch requests at a lower level to support file: URLs, but in my initial investigation, there didn’t seem to be an obvious hook point, so I went with this simpler approach.
I think simple solution for this will be creating temporary http server using python and using it.
- Put all your files in temporary folder eg. tempFolder
- Go to that directory and create a temporary http server in terminal/cmd as per your OS using command python -m http.server 8000 (Note 8000 is port no.)
- This will you give you a link to http server. You can access it from http://127.0.0.1:8000/
- Open your desired file in browser and copy the link to your url.