Python3 Requests Proxy (Socks5)
This is an example of one of the errors I get during this whole process: Solution: Are you also closing the sessions again after use? I am in a situation where I would like to use http.client for performance reasons, but I’d also like to use a SOCKS proxy for remote access into a cluster.
Python3 Requests Proxy (Socks5)
Edit: I don’t know if this is a thing, but often this happens when i’ve been testing a lot and I have a lot of python scripts running. Usually none active at the same time but I’ll have ran a few of them in a short space of time. I don’t know if that’s why teh error starts happening.
Ok here’s my dilemma. I’ve got a NordVPN account and I want to randomly loop through the ip’s for the requests i’m making to google. This works fine .. sometimes. For some reason after a few request pings my program starts slowing down and then I start getting ‘Max Connection Exceeded’ errors and as far as I can tell the ip’s are fine.
I started thinking that google was blocking the ip but then I request to other website and the same error continues.. I started thinking that maybe the proxy server only allows a certain amount of requests in a certain time frame so I tried some free proxies but I get the same errors again.. Either way if I leave it alone for a while it seems to work fine and then a few dozen pings later and the errors start again.
The only think I can think of and not sure how I can test or even if it’s possible to test remotely (work router) is that maybe some open connections remain open and affects my code and then when the backlog of open connections are thrown out I can resume as normal. I started looking for if I needed to «close» my connection after my get request but it doesn’t seem necessary (although I wasn’t even sure what I was looking for).
This is one version of my code, I’ve tried without sessions, I’ve tried a different way of writing the sessions. All seem to work in the same way:
import requests, random username = 'xxx' password = 'yyy' with open('proxies.txt', 'r') as p: proxy_lost = p.read().splitlines() with open('data.txt', 'r') as d: data = d.read().splitlines() for i in data: result = None while result is None: try: proxy = proxy_list[random.randint(0,len(proxy_list)-1)] + '.nordvpn.com' prox = 'socks5://<>:<>@<>:<>'.format(username, password, proxy, 1080) #I've also tried socks5h with requests.Session() as r: r.proxies['http': prox] r.proxies['https': prox] result = r.get('http://icanhazip.com') print(result.text.strip()) If anyone has any idea whatsoever any help is appreciated. I’ve reached a point where i’m struggling to think of new ideas to try.
This is an example of one of the errors I get during this whole process:
Traceback (most recent call last): File «C:\Users*\AppData\Local\Programs\Python\python37\lib\site-packages\socks.py», line 809, in connect negotiate(self, dest_addr, dest_port) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py», line 444, in _negotiate_SOCKS5 self, CONNECT, dest_addr) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py», line 503, in _SOCKS5_request raise SOCKS5AuthError(«SOCKS5 authentication failed») socks.SOCKS5AuthError: SOCKS5 authentication failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\contrib\socks.py», line 88, in _new_conn **extra_kw File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py», line 209, in create_connection raise err File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py», line 199, in create_connection sock.connect((remote_host, remote_port)) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py», line 47, in wrapper return function(*args, **kwargs) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\socks.py», line 814, in connect raise GeneralProxyError(«Socket error», error) socks.GeneralProxyError: Socket error: SOCKS5 authentication failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py», line 600, in urlopen chunked=chunked) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py», line 354, in _make_request conn.request(method, url, **httplib_request_kw) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py», line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py», line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py», line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py», line 1026, in _send_output self.send(msg) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\http\client.py», line 966, in send self.connect() File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py», line 181, in connect conn = self._new_conn() File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\contrib\socks.py», line 110, in _new_conn «Failed to establish a new connection: %s» % error urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: SOCKS5 authentication failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py», line 449, in send timeout=timeout File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py», line 638, in urlopen _stacktrace=sys.exc_info()[2]) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\retry.py», line 399, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: SOCKShttpconnectionPool(host=’icanhazip.com’, port=80): Max retries exceeded with url: / (Caused by NewConnectionError(‘: Failed to establish a new connection: SOCKS5 authentication failed’))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File «C:\Users*\Documents_Scripts\Find Proxies\FindProxies.py», line 23, in result = requests.get(‘http://icanhazip.com’, proxies=proxies) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py», line 75, in get return request(‘get’, url, params=params, **kwargs) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py», line 60, in request return session.request(method=method, url=url, **kwargs) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py», line 533, in request resp = self.send(prep, **send_kwargs) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py», line 646, in send r = adapter.send(request, **kwargs) File «C:\Users*\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py», line 516, in send raise ConnectionError(e, request=request) requests.exceptions.connectionerror: SOCKSHTTPConnectionPool(host=’icanhazip.com’, port=80): Max retries exceeded with url: / (Caused by NewConnectionError(‘: Failed to establish a new connection: SOCKS5 authentication failed’))
Are you also closing the sessions again after use?
How to Use Socks Proxy with Request Module in Python
How to Use Socks Proxy with Request Module in Python
import requests from bs4 import BeautifulSoup as soup url = 'https://www.socks-proxy.net/' response = requests.get(url) bsobj = soup(response.content) proxies= set() for ip in bsobj.findAll('table')[0].findAll('tbody')[0].findAll('tr'): cols = ip.findChildren(recursive = False) cols = [element.text.strip() for element in cols] #print(cols) proxy = ':'.join([cols[0],cols[1]]) proxy = 'socks4://'+proxy proxies.add(proxy) print(proxy) Output:
Now that we have the list of Proxy IP Addresses in a variable proxies, we’ll go ahead and rotate it using a Round Robin method and send get request.
import requests from itertools import cycle #If you are copy pasting proxy ips, put in the list below #proxies = ['121.129.127.209:80', '124.41.215.238:45169', '185.93.3.123:8080', '194.182.64.67:3128', '106.0.38.174:8080', '163.172.175.210:3128', '13.92.196.150:8080'] proxy_pool = cycle(proxies) url = 'https://httpbin.org/ip' for i in range(1,10): #Get a proxy from the pool proxy = next(proxy_pool) print("Request #%d"%i) try: response = requests.get(url,proxies=) print(response.json()) except: #Most free proxies will often get connection errors. You will have retry the entire request using another proxy to work. #We will just skip retries as its beyond the scope of this tutorial and we are only downloading a single url print("Skipping. Connnection error") Output:
As you can see only 3 of our request were successful that is because most free proxies expire quickly. If you need to scrape large amount of data it is recommended to use paid proxies.