Web scraping & data extraction · Zyte (open-source project)
Scrapy
Open-source Python framework you run yourself to write, schedule and scale web crawlers and spiders.
Scrapy is a Python library, not a hosted product: teams write "spiders" as Python classes that define how to follow links and parse responses, and Scrapy handles the concurrency, request scheduling, retries and pipeline plumbing (writing to files, databases or APIs). It does not include proxies, JavaScript rendering or anti-bot bypass out of the box — those are added through extensions (e.g. a headless-browser middleware) or by pairing it with a proxy provider or unblocking API. It runs on any machine or container a team controls, so there is no vendor-imposed request pricing, only the compute and proxy costs a team chooses to add. Zyte, the company maintaining Scrapy, also sells hosted deployment (Scrapy Cloud) for teams that want managed infrastructure around the same codebase.
At a glance
| Vendor | Zyte (open-source project) |
|---|---|
| Pricing model | Open source + paid options |
| Free tier | Yes |
| Deployment | Self-hosted |
| Open source | Yes (BSD-3-Clause) |
| Best for | Engineering teams who want full control of crawl logic and are willing to run and maintain their own infrastructure. |
Pricing
Free and open source; no vendor pricing. Costs are whatever compute, proxies or hosting a team adds around it.
Pricing has not been verified yet — see the vendor's site.
Features
- Asynchronous request scheduling and throttling
- Built-in HTML/XML selectors (CSS and XPath)
- Item pipelines for cleaning and storing scraped data
- Middleware system for proxies, retries and custom headers
- Built-in export to JSON, CSV and XML
- Extensible with headless-browser middlewares for JS-heavy sites
- Command-line tooling for generating and running spiders
Integrations
Profile last reviewed September 21, 2026