HTTP Request/Response from a file and back
What is the best way to parse HTTP request/response from a file to some kind of standard python objects and later dump it to a file? What are the standard HTTP Request/Response objects in python. I need some kind of wrapper as output, just like requests.Response or httplib.HTTPResponse and also for request and a way to parse a file and load data to these wrappers/objects. I don’t know if httplib.HTTPResponse is some basic/standard class for that and why there is no HTTPRequest class and moreover i don’t know how to convert them from plain text files to objects and the other way around.
I don’t understand. Do you have some kind of http traffic dump in a file that you wish to «replay»? In that case, what is the file format?
plain text files. I want to load requests, modify them in scripts and dump again to txt files. same for responses.
OK, «plain text file». But . one request per file? Requests/responses interleaved in the file? Are the «chuncked» data reassembled or not? How are stored response containing binary data (like JPEG images?) How are identified various clients to map the correct response with the correct request?
Thx for your comments 🙂 answers: request per file, same for responses. mapping => file’s name — it’s sequential number and client stores pairs to display request and correct response. binary data => right now don’t know, for now just text files are enough I will think about binaries later. chuncked => reassembled. It’s the API I got and I have to integrate with it 🙁 Really appreciate, thx!
It sounds like you might be better served with a straight parser. The stuff from httplib , and really any other library like it, is primarily geared towards interacting in a transactional manner with traffic on the fly from a socket, not pulling static text files and parsing them. They also are way thicker than you seem to need, given all the networking code. You might be able to use them by opening the file and throwing it at the library as a file-like object in place of a socket, but it still seems like you’d have a better and easier time just working with a lightweight parser.
Saving response from Requests to file
I’m using Requests to upload a PDF to an API. It is stored as «response» below. I’m trying to write that out to Excel.
import requests files = response = requests.post("https://pdftables.com/api?&format=xlsx-single",files=files) response.raise_for_status() # ensure we notice bad responses file = open("out.xls", "w") file.write(response) file.close() file.write(response) TypeError: expected a character buffer object with open(filename, mode=’wb’) as localfile: localfile.write(response.content) is more clean and elegant than open and close, IMHO.
Skip the top two answers and head straight to this one. It’d be nice if OP could move the checkmark to that answer—the existing top answer is pretty much useless and the most-upvoted one is of limited usefulness.
3 Answers 3
I believe all the existing answers contain the relevant information, but I would like to summarize.
The response object that is returned by requests get and post operations contains two useful attributes:
Response attributes
- response.text — Contains str with the response text.
- response.content — Contains bytes with the raw response content.
You should choose one or other of these attributes depending on the type of response you expect.
- For text-based responses (html, json, yaml, etc) you would use response.text
- For binary-based responses (jpg, png, zip, xls, etc) you would use response.content .
Writing response to file
When writing responses to file you need to use the open function with the appropriate file write mode.
- For text responses you need to use «w» — plain write mode.
- For binary responses you need to use «wb» — binary write mode.
Examples
Text request and save
# Request the HTML for this web page: response = requests.get("https://stackoverflow.com/questions/31126596/saving-response-from-requests-to-file") with open("response.txt", "w") as f: f.write(response.text) Binary request and save
# Request the profile picture of the OP: response = requests.get("https://i.stack.imgur.com/iysmF.jpg?s=32&g=1") with open("response.jpg", "wb") as f: f.write(response.content) Answering the original question
The original code should work by using wb and response.content :
import requests files = response = requests.post("https://pdftables.com/api?&format=xlsx-single",files=files) response.raise_for_status() # ensure we notice bad responses file = open("out.xls", "wb") file.write(response.content) file.close() But I would go further and use the with context manager for open .
import requests with open('1.pdf', 'rb') as file: files = response = requests.post("https://pdftables.com/api?&format=xlsx-single",files=files) response.raise_for_status() # ensure we notice bad responses with open("out.xls", "wb") as file: file.write(response.content)