Python selenium скроллинг страницы

Selenium Scroll Down Webpage

Do you want the Web Browser to scroll to the end of the page while using Python Selenium?

You can do that with code, the trick is to inject Javascript code to be webpage. After you load a webpage, scroll down the page by injecting javascript.You can scroll down a specific amount or all the way to the bottom.

Related course:

Scroll down webpage

Example

Before you start make sure the Selenium Web Driver is installed and that you have the selenium module installed. The web driver must be the appropriate web driver for the browser (same version). For Firefox that’s the geckoDriver, for Chrome that’s the ChromeDriver. The version of the driver must be intended for the browser version, an outdated version most likely wont work.

The selenium scroll down code is shown below. It cals the method execute_script() with the javascript to scroll to the end of the web page.

#_*_coding: utf-8_*_
from selenium import webdriver
import time

browser=webdriver.Firefox()
browser.get(«https://en.wikipedia.org»)
browser.execute_script(«window.scrollTo(0,document.body.scrollHeight)»)
time.sleep(3)
browser.close()

First the required modules are loaded. You’ll need the selenium module and the time module.

#_*_coding: utf-8_*_
from selenium import webdriver
import time

Then initialize the web browser. This can be Firefox or another supported browser (Chrome, Edge, Safari)

Get your webpage wit the get() method, which has a parameter the URL to load.

browser.get(«https://en.wikipedia.org»)

And finally scroll down the complete body height or a specific height.

browser.execute_script(«window.scrollTo(0,document.body.scrollHeight)»)

selenium switch to window

Источник

Python Selenium – How to Scroll Down a Page?

If we want to move down to a page or want to search for something on a page that is not in view, we use scrolling to reach there. Is it possible to scroll a page automatically with selenium? Selenium‘s main feature does not have an option for scrolling. But we can achieve it with some additional javascript features enabled in Python by using the driver object. In selenium, it is possible to scroll down the page in three different ways. Today we will try to know about all three possible options.

Set Up the Environment

Since I have a passion for traveling, I love to read travel blogs. Today we will try to scroll a website called “The 50 most visited tourist attractions in the world”.

So, let us start the process. First, we need to import the WebDriver from selenium and then create a driver object from it. Next, we need to specify the path of the ChromeDriver as we will be using a chrome browser to scroll the page. The maximize_window() method is available to have a better view. Then we will try to connect to the website using the driver.get() method. We will be using implicit wait for 10 seconds. A cookie policy will appear at the bottom of the page when we will be connected. We need to find the WebElement of “OK, got it” button to accept it and then click it.

from selenium import webdriver driver = webdriver.Chrome(executable_path = r'G:/chromedriver_win32/chromedriver.exe') driver.maximize_window() driver.get('https://www.lovehomeswap.com/blog/latest-news/the-50-most-visited-tourist-attractions-in-the-world') driver.implicitly_wait(10) cookie = driver.find_element_by_link_text('OK, got it') cookie.click()

Scroll Down the Page by Pixel

It is possible to scroll a page with pixel number. There is a method called execute_script() which enables us to scroll a page. The command goes like this:

driver.execute_script("window.scrollBy(0,500)","")

Here we need to input two parameters in the scrollBy() method. 0 is the starting pixel position or default pixel and 500 is the pixel position we want to scroll to. By changing these values, it is possible to scroll down from one place to another place. The second parameter of the execute_script() method will remain empty. Let’s try to do it on the website.

driver.execute_script("window.scrollBy(0,3000)","")

As we have set the 2 nd parameter of the scrollBy() method from 500 to 3000 we can see the scroll bar at the right-hand side does not remain in its default position. It has scrolled down a bit where the 3000-pixel position lies. By changing the second parameter we can visit certain places on a page with the help of this method.

Scroll Down the Page Till the Element Found

Now we want to search for a specific element in the webpage but we don’t know the exact pixel position for that point. How can we scroll down to that specific element? that is also possible with the following command.

driver.execute_script("arguments[0].scrollIntoView();",Element)

To work with this command, at first we need to identify the element that we want to view, and then we will store it to a variable. Again we will use execute_script() method and it will take two parameters as well. We will input “ arguments[0].scrollIntoView(); ” as the first parameter and the variable that contains the identified element as the second parameter. Hopefully, the scroll bar will automatically move to the place where the element is located.

Let’s try to find the element «Niagra Falls» from the web page. We want to set our scroller to view this element automatically. Following code will be good enough.

niagara_falls = driver.find_element_by_link_text('Niagara Falls') driver.execute_script("arguments[0].scrollIntoView();",niagara_falls)

Here, we tried to find the element by link text and created a variable niagara_falls with the located WebElement . In the next line, we executed the command with driver.execute_script method.

Niagara Falls appears at the top of the page. sometimes it may not be visible due to the “Log in” bar. Then you need to scroll up a little to get the view.

Scroll Down Till the End of the Page

It is also possible with the execute_script method to scroll down to the end of any page. the command will look like this:

driver.execute_script("window.scrollBy(0,document.body.scrollHeight)")

Here again, we will use javascript statement inside the execute_script() method. The scrollBy() method will take two parameters. The first one is 0 as the initial starting point and the second one should be javascript defined “ document.body.scrollHeight ” as it helps the scroller to reach the ending point of the page. If we run the code we will see the page like this.

We can see from the right-hand side that the scrollbar reached the ending point of the page.

So, that’s all about the methods we use nowadays in selenium python to automatically scroll down the browser. I hope you will find this link useful to learn more about Selenium.

Yassin Mahmud is a blockchain enthusiast and content creator in the blockchain space. Driven by his passion for distributed technology, he switched to the blockchain sector from the MERN stack development. With over half a decade of experience at present, he is trying to contribute to the blockchain community with his writing. When Yassin is not writing, he can be found walking along the beaches or touring spectacular travel destinations.

Be on the Right Side of Change 🚀

• The world is changing exponentially. Disruptive technologies such as AI, crypto, and automation eliminate entire industries. 🤖
• Do you feel uncertain and afraid of being replaced by machines, leaving you without money, purpose, or value? Fear not! There a way to not merely survive but thrive in this new world!
• Finxter is here to help you stay ahead of the curve, so you can keep winning as paradigms shift.

Learning Resources 🧑‍💻

⭐ Boost your skills. Join our free email academy with daily emails teaching exponential with 1000+ tutorials on AI, data science, Python, freelancing, and Blockchain development!

Join the Finxter Academy and unlock access to premium courses 👑 to certify your skills in exponential technologies and programming.

New Finxter Tutorials:

Finxter Categories:

Источник