Selenium python proxy chrome auth

Need Proxy?

BotProxy: Rotating Proxies Made for professionals. Really fast connection. Built-in IP rotation. Fresh IPs every day.

Selenium Chrome Proxy Authentication

Setting chromedriver proxy with Selenium using Python

If you need to use proxy with python and Selenium library with chromedriver you usually use the following code:

chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port) driver = webdriver.Chrome(chrome_options=chrome_options) 

It works fine unless proxy requires authentication. if the proxy requires you to login with a username and password it will not work. In this case you have to use more tricky solution that is explained below. By the way if you are going to use BotProxy rotating proxy you can whitelist your server IP address and botproxy will not require HTTP auth to work.

HTTP Proxy Authentication with Chromedriver in Selenium

To set up proxy authentication we will generate a special file and upload it to chromedriver dynamically using the following code below. The example is given for BotProxy rotating proxy server, but you can substitute PROXY_HOST and other constants with your values. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with user/password pair.

import os import zipfile from selenium import webdriver PROXY_HOST = 'x.botproxy.net' # rotating proxy PROXY_PORT = 8080 PROXY_USER = 'proxy-user' PROXY_PASS = 'proxy-password' manifest_json = """   "version": "1.0.0", "manifest_version": 2, "name": "Chrome Proxy", "permissions": [ "proxy", "tabs", "unlimitedStorage", "storage", "", "webRequest", "webRequestBlocking" ], "background":  "scripts": ["background.js"] >, "minimum_chrome_version":"22.0.0" > """ background_js = """ var config =  mode: "fixed_servers", rules:  singleProxy:  scheme: "http", host: "%s", port: parseInt(%s) >, bypassList: ["localhost"] > >; chrome.proxy.settings.set(, function() <>); function callbackFn(details)  return  authCredentials:  username: "%s", password: "%s" > >; > chrome.webRequest.onAuthRequired.addListener( callbackFn, , ['blocking'] ); """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS) def get_chromedriver(use_proxy=False, user_agent=None): path = os.path.dirname(os.path.abspath(__file__)) chrome_options = webdriver.ChromeOptions() if use_proxy: pluginfile = 'proxy_auth_plugin.zip' with zipfile.ZipFile(pluginfile, 'w') as zp: zp.writestr("manifest.json", manifest_json) zp.writestr("background.js", background_js) chrome_options.add_extension(pluginfile) if user_agent: chrome_options.add_argument('--user-agent=%s' % user_agent) driver = webdriver.Chrome( os.path.join(path, 'chromedriver'), chrome_options=chrome_options) return driver def main(): driver = get_chromedriver(use_proxy=True) #driver.get('https://www.google.com/search?q=my+ip+address') driver.get('https://httpbin.org/ip') if __name__ == '__main__': main() 

Function get_chromedriver returns configured selenium webdriver that you can use in your application. This code is tested and works just fine with BotProxy HTTP Rotating Proxy.

Источник

Chromedriver Proxy with Selenium using Python

Selenium is an open-source tool that helps automate web browser interactions for website testing, scraping and more. It’s useful when you need to automate the browser to perform a number of tasks, such as clicking on buttons, scrolling, etc. Even if primarily Selenium is used for website testing, it can also be used for web scraping because it helps locate the required public data on a website.

It provides a single interface that lets you write scripts in programming languages like Python, Ruby, Java, NodeJS, PHP, Perl, and C#.

Selenium automates frequent and recurrent functional, performance, and compatibility testing. This gives developers near-instant feedback for faster debugging, leaving them with more time to code business logic for newer versions/features.

Modern web development needs Selenium testing because:

• It automates repeated testing tasks of smaller components of larger code-bases
• It’s integral to agile development and CI/CD
• It frees resources from manual testing
• It’s consistently reliable; catches bugs that human testers might miss
• You can test your web application at scale
• It’s precise; the customizable error reporting is an added plus
• It’s reusable; you can refactor and reuse an end-to-end test script every time a new feature gets deployed.
• It’s scalable; over time, you can develop an extensive library of repeatable test cases for a product

Selenium Webdriver also known as Selenium 2.0

WebDriver executes test scripts through browser-specific drivers. It consists of API, Library, Driver and Frameworks. It supports libraries for integration with natural or programming language test frameworks.

Basically the WebDriver has a local end (‘client’) which sends the commands (test scripts) to a browser-specific driver. The driver executes these commands on its browser-instance. That way if the test script calls for execution on Chrome and Firefox, the ChromeDriver will execute the test script on Chrome; on the other side the GeckoDriver will do the same on Firefox.

Selenium Chrome Proxy Authentication

When you need to use a proxy with Python and Selenium library with chromedriver you usually use the following code (Without any username and password):

chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port) driver = webdriver.Chrome(chrome_options=chrome_options) 

That works fine unless proxy requires authentication. If the proxy requires you to log in with a username and password you have to use one of the solutions explained below.

1. HTTP Proxy Authentication with Chromedriver in Selenium

In order to set up proxy authentication we will generate a special file and upload it to chromedriver dynamically using the following code below. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with username and password.

import os import zipfile from selenium import webdriver PROXY_HOST = '192.168.10.10' # rotating proxy or host PROXY_PORT = 9000 # port PROXY_USER = 'proxy-user' # username PROXY_PASS = 'proxy-password' # password manifest_json = """ < "version": "1.0.0", "manifest_version": 2, "name": "Chrome Proxy", "permissions": [ "proxy", "tabs", "unlimitedStorage", "storage", "", "webRequest", "webRequestBlocking" ], "background": < "scripts": ["background.js"] >, "minimum_chrome_version":"22.0.0" > """ background_js = """ var config = < mode: "fixed_servers", rules: < singleProxy: < scheme: "http", host: "%s", port: parseInt(%s) >, bypassList: ["localhost"] > >; chrome.proxy.settings.set(, function() <>); function callbackFn(details) < return < authCredentials: < username: "%s", password: "%s" >>; > chrome.webRequest.onAuthRequired.addListener( callbackFn, , ['blocking'] ); """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS) def get_chromedriver(use_proxy=False, user_agent=None): path = os.path.dirname(os.path.abspath(__file__)) chrome_options = webdriver.ChromeOptions() if use_proxy: pluginfile = 'proxy_auth_plugin.zip' with zipfile.ZipFile(pluginfile, 'w') as zp: zp.writestr("manifest.json", manifest_json) zp.writestr("background.js", background_js) chrome_options.add_extension(pluginfile) if user_agent: chrome_options.add_argument('--user-agent=%s' % user_agent) driver = webdriver.Chrome( os.path.join(path, 'chromedriver'), chrome_options=chrome_options) return driver def main(): driver = get_chromedriver(use_proxy=True) driver.get('https://httpbin.org/ip') # any url you want to crawl

Function get_chromedriver returns configured selenium webdriver that you can use in your application.

2. Using Selenium-Wire Package

Selenium Wire extends Selenium’s Python bindings to give you access to the underlying requests made by the browser. You author your code in the same way as you do with Selenium, but you get extra APIs for inspecting requests and responses and making changes to them on the fly.

Example code from the documentation:

HTTP proxies

from seleniumwire import webdriver options = < 'proxy': < 'http': 'http://user:[email protected]:8888', 'https': 'https://user:[email protected]:8888', 'no_proxy': 'localhost,127.0.0.1' > > driver = webdriver.Chrome(seleniumwire_options=options) 

SOCKS proxies

from seleniumwire import webdriver options = < 'proxy': < 'http': 'socks5://user:[email protected]:8888', 'https': 'socks5://user:[email protected]:8888', 'no_proxy': 'localhost,127.0.0.1' > > driver = webdriver.Chrome(seleniumwire_options=options) 

Another recommended package is webdriver-manager. It’s a package that helps with the management of binary drivers for different browsers. There’s no need to manually download a new version of a web driver after each update.

You can install the webdriver-manager using the pip command:

pip install webdrive-manager

Selenium is a great tool for public web scraping, especially when learning the basics. With the help of ProxyEmpire’s Residential And Mobile Proxies, web scraping becomes even more efficient.

Источник