Scrappy для python 3

Scrapy 2.9 documentation¶

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Getting help¶

Having trouble? We’d like to help!

• Try the FAQ – it’s got answers to some common questions.
• Looking for specific information? Try the Index or Module Index .
• Ask or search questions in StackOverflow using the scrapy tag.
• Ask or search questions in the Scrapy subreddit.
• Search for questions on the archives of the scrapy-users mailing list.
• Ask a question in the #scrapy IRC channel,
• Report bugs with Scrapy in our issue tracker.
• Join the Discord community Scrapy Discord.

First steps¶

Understand what Scrapy is and how it can help you.

Get Scrapy installed on your computer.

Write your first Scrapy project.

Learn more by playing with a pre-made Scrapy project.

Basic concepts¶

Learn about the command-line tool used to manage your Scrapy project.

Write the rules to crawl your websites.

Extract the data from web pages using XPath.

Test your extraction code in an interactive environment.

Define the data you want to scrape.

Populate your items with the extracted data.

Post-process and store your scraped data.

Output your scraped data using different formats and storages.

Understand the classes used to represent HTTP requests and responses.

Convenient classes to extract links to follow from pages.

Learn how to configure Scrapy and see all available settings .

See all available exceptions and their meaning.

Built-in services¶

Learn how to use Python’s builtin logging on Scrapy.

Collect statistics about your scraping crawler.

Send email notifications when certain events occur.

Inspect a running crawler using a built-in Python console.

Solving specific problems¶

Get answers to most frequently asked questions.

Learn how to debug common problems of your Scrapy spider.

Learn how to use contracts for testing your spiders.

Get familiar with some Scrapy common practices.

Tune Scrapy for crawling a lot domains in parallel.

Learn how to scrape with your browser’s developer tools.

Read webpage data that is loaded dynamically.

Learn how to find and get rid of memory leaks in your crawler.

Download files and/or images associated with your scraped items.

Deploying your Scrapy spiders and run them in a remote server.

Adjust crawl rate dynamically based on load.

Check how Scrapy performs on your hardware.

Learn how to pause and resume crawls for large spiders.

Use asyncio and asyncio -powered libraries.

Extending Scrapy¶

Understand the Scrapy architecture.

Customize how pages get requested and downloaded.

Customize the input and output of your spiders.

Extend Scrapy with your custom functionality

See all available signals and how to work with them.

Understand the scheduler component.

Quickly export your scraped items to a file (XML, CSV, etc).

Learn the common API and some good practices when building custom Scrapy components.

Use it on extensions and middlewares to extend Scrapy functionality.

All the rest¶

See what has changed in recent Scrapy versions.

Learn how to contribute to the Scrapy project.

Understand Scrapy versioning and API stability.

© Copyright 2008–2023, Scrapy developers. Revision 52c07264 . Last updated on May 08, 2023.

Источник

Installation guide¶

Scrapy requires Python 3.7+, either the CPython implementation (default) or the PyPy implementation (see Alternate Implementations ).

Installing Scrapy¶

If you’re using Anaconda or Miniconda, you can install the package from the conda-forge channel, which has up-to-date packages for Linux, Windows and macOS.

To install Scrapy using conda , run:

conda install -c conda-forge scrapy 

Alternatively, if you’re already familiar with installation of Python packages, you can install Scrapy and its dependencies from PyPI with:

We strongly recommend that you install Scrapy in a dedicated virtualenv , to avoid conflicting with your system packages.

Note that sometimes this may require solving compilation issues for some Scrapy dependencies depending on your operating system, so be sure to check the Platform specific installation notes .

For more detailed and platform specifics instructions, as well as troubleshooting information, read on.

Things that are good to know¶

Scrapy is written in pure Python and depends on a few key Python packages (among others):

• lxml, an efficient XML and HTML parser
• parsel, an HTML/XML data extraction library written on top of lxml,
• w3lib, a multi-purpose helper for dealing with URLs and web page encodings
• twisted, an asynchronous networking framework
• cryptography and pyOpenSSL, to deal with various network-level security needs

Some of these packages themselves depend on non-Python packages that might require additional installation steps depending on your platform. Please check platform-specific guides below .

In case of any trouble related to these dependencies, please refer to their respective installation instructions:

Using a virtual environment (recommended)¶

TL;DR: We recommend installing Scrapy inside a virtual environment on all platforms.

Python packages can be installed either globally (a.k.a system wide), or in user-space. We do not recommend installing Scrapy system wide.

Instead, we recommend that you install Scrapy within a so-called “virtual environment” ( venv ). Virtual environments allow you to not conflict with already-installed Python system packages (which could break some of your system tools and scripts), and still install packages normally with pip (without sudo and the likes).

See Virtual Environments and Packages on how to create your virtual environment.

Once you have created a virtual environment, you can install Scrapy inside it with pip , just like any other Python package. (See platform-specific guides below for non-Python dependencies that you may need to install beforehand).

Platform specific installation notes¶

Windows¶

Though it’s possible to install Scrapy on Windows using pip, we recommend you to install Anaconda or Miniconda and use the package from the conda-forge channel, which will avoid most installation issues.

Once you’ve installed Anaconda or Miniconda, install Scrapy with:

conda install -c conda-forge scrapy 

To install Scrapy on Windows using pip :

This installation method requires “Microsoft Visual C++” for installing some Scrapy dependencies, which demands significantly more disk space than Anaconda.

1. Download and execute Microsoft C++ Build Tools to install the Visual Studio Installer.
2. Run the Visual Studio Installer.
3. Under the Workloads section, select C++ build tools.
4. Check the installation details and make sure following packages are selected as optional components:

• MSVC (e.g MSVC v142 — VS 2019 C++ x64/x86 build tools (v14.23) )
• Windows SDK (e.g Windows 10 SDK (10.0.18362.0))

Now, you should be able to install Scrapy using pip .

Ubuntu 14.04 or above¶

Scrapy is currently tested with recent-enough versions of lxml, twisted and pyOpenSSL, and is compatible with recent Ubuntu distributions. But it should support older versions of Ubuntu too, like Ubuntu 14.04, albeit with potential issues with TLS connections.

Don’t use the python-scrapy package provided by Ubuntu, they are typically too old and slow to catch up with latest Scrapy.

To install Scrapy on Ubuntu (or Ubuntu-based) systems, you need to install these dependencies:

sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev 

• python3-dev , zlib1g-dev , libxml2-dev and libxslt1-dev are required for lxml
• libssl-dev and libffi-dev are required for cryptography

Inside a virtualenv , you can install Scrapy with pip after that:

The same non-Python dependencies can be used to install Scrapy in Debian Jessie (8.0) and above.

macOS¶

Building Scrapy’s dependencies requires the presence of a C compiler and development headers. On macOS this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run:

There’s a known issue that prevents pip from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions:

• (Recommended)Don’t use system Python. Install a new, updated version that doesn’t conflict with the rest of your system. Here’s how to do it using the homebrew package manager:
  • Install homebrew following the instructions in https://brew.sh/
  • Update your PATH variable to state that homebrew packages should be used before system packages (Change .bashrc to .zshrc accordingly if you’re using zsh as default shell):

  echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc 

  brew update; brew upgrade python 

  This method is a workaround for the above macOS issue, but it’s an overall good practice for managing dependencies and can complement the first method.

  After any of these workarounds you should be able to install Scrapy:

  PyPy¶

  We recommend using the latest PyPy version. For PyPy3, only Linux installation was tested.

  Most Scrapy dependencies now have binary wheels for CPython, but not for PyPy. This means that these dependencies will be built during installation. On macOS, you are likely to face an issue with building the Cryptography dependency. The solution to this problem is described here, that is to brew install openssl and then export the flags that this command recommends (only needed when installing Scrapy). Installing on Linux has no special issues besides installing build dependencies. Installing Scrapy with PyPy on Windows is not tested.

  You can check that Scrapy is installed correctly by running scrapy bench . If this command gives errors such as TypeError: . got 2 unexpected keyword arguments , this means that setuptools was unable to pick up one PyPy-specific dependency. To fix this issue, run pip install ‘PyPyDispatcher>=2.1.0’ .

  Troubleshooting¶

  AttributeError: ‘module’ object has no attribute ‘OP_NO_TLSv1_1’¶

  After you install or upgrade Scrapy, Twisted or pyOpenSSL, you may get an exception with the following traceback:

  […] File "[…]/site-packages/twisted/protocols/tls.py", line 63, in from twisted.internet._sslverify import _setAcceptableProtocols File "[…]/site-packages/twisted/internet/_sslverify.py", line 38, in TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1, AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'

  The reason you get this exception is that your system or virtual environment has a version of pyOpenSSL that your version of Twisted does not support.

  To install a version of pyOpenSSL that your version of Twisted supports, reinstall Twisted with the tls extra option:

  © Copyright 2008–2023, Scrapy developers. Revision 52c07264 . Last updated on May 08, 2023.

  Источник

  Scrappy для python 3

  An open source and collaborative framework for extracting the data you need from websites.

  In a fast, simple, yet extensible way.

  Maintained by Zyte (formerly Scrapinghub) and many other contributors

  pip install scrapy cat > myspider.py 
  import scrapy class BlogSpider(scrapy.Spider): name = 'blogspider' start_urls = ['https://www.zyte.com/blog/'] def parse(self, response): for title in response.css('.oxy-post-title'): yield 'title': title.css('::text').get()> for next_page in response.css('a.next'): yield response.follow(next_page, self.parse)
   
  Build and run your 
  web spiders
   
  pip install shub shub login Insert your Zyte Scrapy Cloud API Key: # Deploy the spider to Zyte Scrapy Cloud shub deploy # Schedule the spider for execution shub schedule blogspider Spider blogspider scheduled, watch it running here: https://app.zyte.com/p/26731/job/1/8 # Retrieve the scraped data shub items 26731/1/8 
  "title": "Improved Frontera: Web Crawling at Scale with Python 3 Support"> "title": "How to Crawl the Web Politely with Scrapy"> . 
   
  or use Scrapyd to host the spiders on your own server
   
  Fast and powerful
   
  write the rules to extract the data and let Scrapy do the rest
   
  Easily extensible
   
  extensible by design, plug new functionality easily without having to touch the core
   
  Portable, Python
   
  written in Python and runs on Linux, Windows, Mac and BSD
   
  Healthy community
   
   
  - 43,100 stars, 9,600 forks and 1,800 watchers on GitHub
   
  - 5.500 followers on Twitter
   
  - 18,000 questions on StackOverflow
   
   
  Want to know more?
   
  Maintained by Zyte (formerly Scrapinghub) and many other contributors
   
  Источник
   
  Читайте также:  Php console new line