Python requests read timed out

My guess is that someone abused a Python script once to download too many files from that server and you are being tar-pitted based on the User-Agent header alone.

Why do I receive a timeout error from Pythons requests, try: #defined request goes here except requests.exceptions.ReadTimeout: # Set up for a retry, or continue in a retry loop You can wrap it like an exception block like this. Since you asked for this only ReadTimeout. Otherwise catch all of them; try: #defined request goes here …

Define timeout in web request with python

This is my scenario. I am making requests to a web page that takes too long in some occasions. I would like that when it takes more than 10 seconds without issuing a response from the server, the request is canceled but without receiving any error. This is my current code and the error that appears to me. when this error appears, my code ends. In summary I want to make a web request and limit that if there is no answer in 10 seconds, I finish the request and continue with my code.

requests.post("www.webpage.com", headers = , data = ,timeout=10) . . . 

when 10 seconds pass I get this error

ReadTimeout: HTTPConnectionPool(host='www.webpage.com', port=80): Read timed out. (read timeout=10) 

I do not put the real url for confidentiality reasons, but normally without setting a timeout, it works

Use an exception to handle the timeout

try: requests.post("www.webpage.com", headers = , data = ,timeout=10) except requests.exceptions.ReadTimeout: print("Server didn't respond within 10 seconds") 

Python requests or urllib read timeout, URL encoding, Browse other questions tagged python pdf python-requests urllib or ask your own question. The Overflow Blog The internet’s Robin Hood uses robo-lawyers to fight parking tickets and spam

Read timed out. error while sending a POST request to a node.js API

I have a node.js API as below to which I send a POST request from python as below,the issue am facing is if I remove the headers= the POST goes thorugh,if not i get a Read timed out. error , can anyone provide guidance on how to fix this timeout error ?

node.js endpoint

app.post("/api/bats_push",(req, res) => < //console.log("Calling bats_push. ") const d = < method: req.method, headers: req.headers, query: req.query, body: '' >req.on('data', (c) => < //console.log(c) d.body = d.body + c >); req.on('end', () => < DATA.push(d); res.end('Saved BATS job details'); //res.status(200).json(< //message: "Saved BATS job details", //posts: req.body //>); >); >); 

Python POST

try: json">,timeout=10.0) r = requests.post(webhook_url,data=json_data.encode("utf8"),verify=False,headers=) print "posted" print(r.status_code, r.reason) print r.url print r.text except Exception as e: print (e) 

 InsecureRequestWarning) HTTPSConnectionPool(host='company.com', port=443): Read timed out. (read timeout=10.0) 

I seems that you are using express.js. I believe that your problem is, that body is actually already parsed. You can check it by reading req.body . The situation is caused because express.js already read whole body (due to the content type) and trying to read body again will cause timeout (event data and event end are not emitted). There are several ways how to fix it.

1. disable express.js body parser - or reconfigure it to ignore json
2. remove reading body code and use directly req.body

app.post("/api/bats_push",(req, res) => < //console.log("Calling bats_push. ") const d = < method: req.method, headers: req.headers, query: req.query, body: req.body >DATA.push(d); res.end('Saved BATS job details'); >); 

according to requests why not use this:

#replace this r = requests.post(webhook_url,data=json_data.encode("utf8"),verify=False,headers=) #by this. assuming that 'data' is a dict r = requests.post(webhook, json=data, verify=False) 

Looks like it could be something related to your SSL. Check if you get the same error sending a request to your localhost with the server running.

From your question,the key word is:

if I remove the headers= the POST goes thorugh.

The reason may clear:it is a wrong way use about the header.

Simpler say: the node.js app check the header before into the logic code.

if do not send the header by ourself,the requests part use the default headers below:

through the requests's code can print the default headers when post.

may resulting in the node.js app think the reqeust not legitimate(the word I do not know how to explain in English).

If the node.js app is developed by yourself, you can try to find the frame's check after create a tcp connection and before the logic code,the way is to read the source code.

If the node.js app is not developed by yourself,try to change the header mixing the default header to find which header key checked by the node.js app.

But in my thought,It is import ,about the node.js app's interface description:

we just use by interface description engouth,just need to know the error from the api header's check, which the description should show but not show to us?

Why is default timeout for python requests.get(), 1 Answer Sorted by: -2 Again, the default timeout is None, you can dig into the library files in your OS (the path is /usr/local/lib/python2.7/site-packages/requests/ in my Mac OSX) two files to look into is api.py and sessions.py for timeout part, line 275 file sessions.py

Источник

Timeout within session while sending requests

I'm trying to learn how I can use timeout within a session while sending requests. The way I've tried below can fetch the content of a webpage but I'm not sure this is the right way as I could not find the usage of timeout in this documentation.

import requests link = "https://stackoverflow.com/questions/tagged/web-scraping" with requests.Session() as s: r = s.get(link,timeout=5) print(r.text) 

2 Answers 2

You can tell Requests to stop waiting for a response after a given number of seconds with the timeout parameter. Nearly all production code should use this parameter in nearly all requests.

requests.get('https://github.com/', timeout=0.001) 

Or from the Documentation Advanced Usage you can set 2 values (connect and read timeout)

The timeout value will be applied to both the connect and the read timeouts. Specify a tuple if you would like to set the values separately:

r = requests.get('https://github.com', timeout=(3.05, 27)) 

Making Session Wide Timeout

Searched throughout the documentation and it seams it is not possible to set timeout parameter session wide.

But there is a GitHub Issue Opened (Consider making Timeout option required or have a default) which provides a workaround as an HTTPAdapter you can use like this:

import requests from requests.adapters import HTTPAdapter class TimeoutHTTPAdapter(HTTPAdapter): def __init__(self, *args, **kwargs): if "timeout" in kwargs: self.timeout = kwargs["timeout"] del kwargs["timeout"] super().__init__(*args, **kwargs) def send(self, request, **kwargs): timeout = kwargs.get("timeout") if timeout is None and hasattr(self, 'timeout'): kwargs["timeout"] = self.timeout return super().send(request, **kwargs) 

And mount on a requests.Session()

s = requests.Session() s.mount('http://', TimeoutHTTPAdapter(timeout=5)) # 5 seconds s.mount('https://', TimeoutHTTPAdapter(timeout=5)) . r = s.get(link) print(r.text) 

or similarly you can use the proposed EnhancedSession by @GordonAitchJay

with EnhancedSession(5) as s: # 5 seconds r = s.get(link) print(r.text) 

In the snippets above, I believe it should be s.mount('http://', TimeoutHTTPAdapter(timeout=5)) and s.mount('https://', TimeoutHTTPAdapter(timeout=5)) .

I'm not sure this is the right way as I could not find the usage of timeout in this documentation.

Scroll to the bottom. It's definitely there. You can search for it in the page by pressing Ctrl + F and entering timeout .

You're using timeout correctly in your code example.

You can actually specify the timeout in a few different ways, as explained in the documentation:

If you specify a single value for the timeout, like this:

r = requests.get('https://github.com', timeout=5)

The timeout value will be applied to both the connect and the read timeouts. Specify a tuple if you would like to set the values separately:

r = requests.get('https://github.com', timeout=(3.05, 27))

If the remote server is very slow, you can tell Requests to wait forever for a response, by passing None as a timeout value and then retrieving a cup of coffee.

r = requests.get('https://github.com', timeout=None)

For example, this raises an exception because 0.2 seconds is not long enough to establish a connection with the server:

import requests link = "https://httpstat.us/200?sleep=5000" with requests.Session() as s: try: r = s.get(link, timeout=(0.2, 10)) print(r.text) except requests.exceptions.Timeout as e: print(e) 

HTTPSConnectionPool(host='httpstat.us', port=443): Read timed out. (read timeout=0.2) 

This raises an exception because the server waits for 5 seconds before sending the response, which is longer than the 2 second read timeout set:

import requests link = "https://httpstat.us/200?sleep=5000" with requests.Session() as s: try: r = s.get(link, timeout=(3.05, 2)) print(r.text) except requests.exceptions.Timeout as e: print(e) 

HTTPSConnectionPool(host='httpstat.us', port=443): Read timed out. (read timeout=2) 

You specifically mention using a timeout within a session. So maybe you want a session object which has a default timeout. Something like this:

import requests link = "https://httpstat.us/200?sleep=5000" class EnhancedSession(requests.Session): def __init__(self, timeout=(3.05, 4)): self.timeout = timeout return super().__init__() def request(self, method, url, **kwargs): print("EnhancedSession request") if "timeout" not in kwargs: kwargs["timeout"] = self.timeout return super().request(method, url, **kwargs) session = EnhancedSession() try: response = session.get(link) print(response) except requests.exceptions.Timeout as e: print(e) try: response = session.get(link, timeout=1) print(response) except requests.exceptions.Timeout as e: print(e) try: response = session.get(link, timeout=10) print(response) except requests.exceptions.Timeout as e: print(e) 

EnhancedSession request HTTPSConnectionPool(host='httpstat.us', port=443): Read timed out. (read timeout=4) EnhancedSession request HTTPSConnectionPool(host='httpstat.us', port=443): Read timed out. (read timeout=1) EnhancedSession request

Источник