Python selenium proxy geckodriver

Как установить прокси-аутентификацию (пользователь и пароль) с помощью Python + Selenium

Я использую Firefox WebDriver в Python 2.7 с Selenium. Моя программа python запускает браузер Firefox и посещает разные веб-сайты, когда я запускаю программу. Но мне нужно установить прокси с аутентификацией, чтобы при посещении программой любого веб-сайта она посещала прокси-сервер.

Есть несколько подобных вопросов на SO. Но для Selenium Firefox WebDriver of Python нет конкретного решения.

Firefox сохраняет свою конфигурацию прокси в профиле. Вы можете установить прокси в профиле и использовать этот профиль Firefox. поэтому я думаю, что вам нужно изменить существующий профиль Firefox. Вы найдете много ресурсов, как изменить профиль FF для прокси.

Не могли бы вы дать мне ссылку на то, как я могу изменить профиль firefox из программы python, сохранить его и использовать повторно .

Насколько я знаю, селен не может сделать это сам по себе. нужно искать больше для последнего результата. но я думаю, что вы можете справиться с этим через python

6 ответов

Установите селеновую проволоку

Импортировать

from seleniumwire import webdriver 

Авторизация через прокси

options = < 'proxy': < 'http': 'http://username:password@host:port', 'https': 'https://username:password@host:port', 'no_proxy': 'localhost,127.0.0.1,dev_server:8080' >> driver = webdriver.Firefox(seleniumwire_options=options) 

Предупреждение
Загляните в папку кэша selenium-wire. У меня была проблема, потому что она занимала все мое дисковое пространство. Вы должны иногда удалять его в своем скрипте, когда хотите.

В дополнение к запуску Firefox с профилем, в котором сохранены учетные данные. Вы можете сделать это, загрузив расширение, которое пишет в loginTextbox и password1Textbox из chrome://global/content/commonDialog.xul (окно предупреждения).

Уже есть некоторые расширения, которые сделают эту работу. Например: Close Proxy Authentication

from selenium import webdriver from base64 import b64encode proxy = fp = webdriver.FirefoxProfile() fp.add_extension('closeproxy.xpi') fp.set_preference('network.proxy.type', 1) fp.set_preference('network.proxy.http', proxy['host']) fp.set_preference('network.proxy.http_port', int(proxy['port'])) # . ssl, socks, ftp . fp.set_preference('network.proxy.no_proxies_on', 'localhost, 127.0.0.1') credentials = ':'.format(**proxy) credentials = b64encode(credentials.encode('ascii')).decode('utf-8') fp.set_preference('extensions.closeproxyauth.authtoken', credentials) driver = webdriver.Firefox(fp) 

Источник

Using Proxies With Python Selenium

Selenium is a powerful browser automation library that allows you to build bots and scrapers that can load and interact with web pages in the browser. As a result, Selenium is very popular amongst the Python web scraping community.

In this guide for The Python Selenium Web Scraping Playbook, we will look at how to integrate proxies into our Python Selenium based web scraper.

There are number of different types of proxies which you need to integrate differently with Selenium, so we will walk through how to integrate each type:

Using Proxies With Selenium​

The first and simplest type of proxy to integrate with Python Selenium are simple HTTP proxies (in the form of a IP address) that don’t require authentication. For example:

Depending on which type of browser you are using the integration method is slightly different.

Integrating Proxy With Selenium Chrome Browser​

To integrate this proxy IP into a Selenium scraper that uses a Chrome Browser we just need to set the —proxy-server arguement in our WebDriver options:

  from selenium import webdriver  ## Example Proxy PROXY = "11.456.448.110:8080"  ## Create WebDriver Options to Add Proxy chrome_options = WebDriver.ChromeOptions() chrome_options.add_argument(f'--proxy-server=PROXY>') chrome = webdriver.Chrome(chrome_options=chrome_options)  ## Make Request Using Proxy chrome.get("http://httpbin.org/ip") 

Now when we run the script we can see that Selenium is using the defined proxy IP:

Integrating Proxy With Selenium Firefox Browser​

To integrate this proxy IP into a Selenium scraper that uses a FireFox Browser we need to use the Proxy and ProxyType classes from the Selenium Webdriver library:

  from selenium import webdriver from selenium.webdriver.common.proxy import Proxy, ProxyType  ## Define Proxy proxy = Proxy(  'proxyType': ProxyType.MANUAL, 'httpProxy': "11.456.448.110:8080", 'noProxy': '' >)  ## Create Driver firefox_driver = webdriver.Firefox(proxy = proxy, executable_path=r"/root/geckodriver")  ## Make Request Using Proxy firefox_driver.get("http://httpbin.org/ip") 

Now when we run the script we can see that Selenium is using the defined proxy IP:

HTTP Proxy Authentication

This method works fine when you don’t need to add an authentication username and password to the proxy. We will look at how to use authenticated proxies in another section.

Using Authenticated Proxies With Selenium​

The above method doesn’t work if you need to use proxies that require username and password authentication.

It is very common for commercial proxy providers to sell access to their proxy pools by giving you single proxy endpoint that you send your requests too and authenticate your account using a username and password .

  "http://USERNAME:PASSWORD@proxy-server:8080" 

There are a couple ways to solve this, but one of the easiest is to use the Selenium Wire extension which makes it very easy to use proxies with Selenium.

First, you need to install Selenium Wire using pip:

Then update your scraper to use the seleniumwire webdriver instead of the default selenium webdriver :

from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager ## Define Your Proxy Endpoints proxy_options = ‘proxy’: ‘http’: ‘http://USERNAME:PASSWORD@proxy-server:8080’, ‘https’: ‘http://USERNAME:PASSWORD@proxy-server:8080’, ‘no_proxy’: ‘localhost:127.0.0.1’ > > ## Set Up Selenium Chrome driver driver = webdriver.Chrome(ChromeDriverManager().install(), seleniumwire_options=proxy_options) ## Send Request Using Proxy driver.get(‘http://httpbin.org/ip’)

Now when we run the script we can see that Selenium is using a proxy IP:

Selenium Wire has a lot of other powerful functionality, so if you would like to learn more then check out our full Selenium Wire guide here.

Integrating Proxy APIs​

Over the last few years there has been a huge surge in proxy providers that offer smart proxy solutions that handle all the proxy rotation, header selection, ban detection and retries on their end. These smart APIs typically provide their proxy services in a API endpoint format.

However, these proxy API endpoints don’t integrate well with headless browsers when the website is using relative links as Selenium will try to attach the relative URL onto the proxy API endpoint not the websites root URL. Resulting, in some pages not loading correctly.

As a result, when integrating your Selenium scrapers it is recommended that you use their proxy port integration over the API endpoint integration when they provide them (not all do have a proxy port integration).

For example, in the case of the ScrapeOps Proxy Aggregator we offer a proxy port integration for situations like this.

The proxy port integration is a light front-end for the API and has all the same functionality and performance as sending requests to the API endpoint but allow you to integrate our proxy aggregator as you would with any normal proxy.

The following is an example of how to integrate the ScrapeOps Proxy Aggregator into your Selenium scraper using

from seleniumwire import webdriver from webdriver_manager.chrome import ChromeDriverManager SCRAPEOPS_API_KEY = ‘APIKEY’ ## Define ScrapeOps Proxy Port Endpoint proxy_options = ‘proxy’: ‘http’: f’http://scrapeops:SCRAPEOPS_API_KEY>@proxy.scrapeops.io:5353′, ‘https’: f’http://scrapeops:SCRAPEOPS_API_KEY>@proxy.scrapeops.io:5353′, ‘no_proxy’: ‘localhost:127.0.0.1’ > > ## Set Up Selenium Chrome driver driver = webdriver.Chrome(ChromeDriverManager().install(), seleniumwire_options=proxy_options) ## Send Request Using ScrapeOps Proxy driver.get(‘http://quotes.toscrape.com/’)

Full integration docs for Python Selenium and the ScrapeOps Proxy Aggregator can be found here.

To use the ScrapeOps Proxy Aggregator, you first need an API key which you can get by signing up for a free account here which gives you 1,000 free API credits.

More Web Scraping Tutorials​

So that’s how you can use both authenticated and unauthenticated proxies with Selenium to scrape websites without getting blocked.

If you would like to learn more about Web Scraping with Selenium, then be sure to check out The Selenium Web Scraping Playbook.

Or check out one of our more in-depth guides:

Источник

Блог

Прокси-сервер не работает на python selenium с firefox geckodriver

#python #selenium #proxy #geckodriver

#python #selenium #прокси #geckodriver

Вопрос:

У меня есть работающий скребок, но я не могу понять, почему запросы не проходят через установленный мной прокси. Я попробовал три разных подхода, которые нашел, но ни один из них не сработал. Может быть, это что-то в настройке сервера, что я должен проверить?

Вот три варианта, которые я попробовал, но безуспешно.

 def SetProxy1(): options = Options() myProxy = '191.6.69.137:20183' proxy = Proxy(< 'proxyType': ProxyType.MANUAL, 'httpProxy': myProxy, 'ftpProxy': myProxy, 'sslProxy': myProxy, >) options.headless = True options.add_argument('-headless') driver = webdriver.Firefox(options=options, proxy=proxy) return driver def SetProxy2(): options = Options() options.add_argument("--headless") proxy = "191.6.69.137" proxy_port = 20183 proxy_profile = webdriver.FirefoxProfile() proxy_profile.set_preference("network.proxy.type", 1) proxy_profile.set_preference("network.proxy.http",proxy) proxy_profile.set_preference("network.proxy.http_port",int(proxy_port)) proxy_profile.set_preference("network.proxy.https",proxy) proxy_profile.set_preference("network.proxy.https_port",int(proxy_port)) proxy_profile.set_preference("network.proxy.ssl",proxy) proxy_profile.set_preference("network.proxy.ssl_port",int(proxy_port)) proxy_profile.set_preference("network.proxy.ftp",proxy) proxy_profile.set_preference("network.proxy.ftp_port",int(proxy_port)) proxy_profile.set_preference("network.proxy.socks",proxy) proxy_profile.set_preference("network.proxy.socks_port",int(proxy_port)) proxy_profile.update_preferences() browser = webdriver.Firefox(firefox_profile=proxy_profile, firefox_options=options) return browser def SetProxy3(): PROXY = "191.6.69.137" PORT = 20183 desired_capability = webdriver.DesiredCapabilities.FIREFOX desired_capability['proxy']=< "proxyType":"manual", "httpProxy":PROXY, "httpProxyPort": PORT, "ftpProxy":PROXY, "ftpProxyPort": PORT, "sslProxy":PROXY, "sslProxyPort" : PORT > options = Options() options.add_argument("--headless") driver = webdriver.Firefox(firefox_options=options,capabilities=desired_capability) return driver 

Ответ №1:

Это функциональный прокси для selenium

Для Firefox

 from selenium.webdriver.common.proxy import Proxy, ProxyType def setProxy(): myProxy = "xx.xx.xx.xx:xxxx" proxy = Proxy(< 'proxyType': ProxyType.MANUAL, 'httpProxy': myProxy, 'ftpProxy': myProxy, 'sslProxy': myProxy, 'noProxy': '' # set this value as desired >) driver = webdriver.Firefox(proxy=proxy) # Example how to use with out function :: driver.get("http://www.google.com") return driver 

 from selenium.webdriver.common.proxy import Proxy, ProxyType def setProxy(): prox = Proxy() prox.proxy_type = ProxyType.MANUAL prox.http_proxy = "ip_addr:port" prox.socks_proxy = "ip_addr:port" prox.ssl_proxy = "ip_addr:port" capabilities = webdriver.DesiredCapabilities.CHROME prox.add_to_capabilities(capabilities) driver = webdriver.Chrome(desired_capabilities=capabilities) return driver 

Комментарии:

1. это для Chrome .. вопросы были о Firefox / GeckoDriver

2. Я просто упустил суть. Я обновляю для обоих браузеров [Firefox и Chrome]

3. это именно мой пример SetProxy1 (), или я что-то упускаю?

Источник