scrapy-stealth

Overview

scrapy-stealth extends Scrapy with browser impersonation, proxy rotation, fingerprint cycling, and intelligent retry strategies — designed for large-scale, production-grade crawling.

🧠 Why scrapy-stealth?

Scrapy is fast and powerful, but modern websites use advanced anti-bot protections such as:

• TLS fingerprinting
• Browser behavior detection
• Rate limiting and IP blocking

scrapy-stealth helps by adding:

📊 Comparison

✨ Features

• 🔌 Pluggable engine system (scrapy, stealth)
• 🧠 Per-request engine selection via request.meta
• 🌐 Proxy support and rotation
• 🧬 Browser fingerprint rotation
• 🔁 Smart retry logic
• 🛡️ Anti-bot detection (status + content-based, Cloudflare, Akamai)
• 🧠 Smart browser selection — start with fast basic / turbo, auto-retry once with visible Chrome when a JS challenge or ban is detected
• ⚡ Thread-safe async integration
• 🖥️ Real-browser engine (CDP) for JS-heavy pages
• 🔄 Intelligent session recycle — after consecutive bans, browser restarts Chrome; basic/turbo clear HTTP sessions
• 🚫 Static asset blocking — skip images, fonts, CSS, and media for faster, lighter browser fetches
• 🎯 Proxy bypass list — send chosen domains straight to the origin instead of through the proxy (--proxy-bypass-list)
• 🧭 Custom DNS overrides — pin hosts to fixed IPs (connect via IP, keep hostname for TLS/SNI/Host) to dodge poisoned or geo-shifted public DNS
• 📸 Built-in snapshot decorator (scrapy_stealth.decorators.snapshot)

📦 Installation

pip install scrapy-stealth

💡 Requires Python 3.11+ and Scrapy 2.12–2.x

⚙️ Setup

# 1. Enable the middleware
DOWNLOADER_MIDDLEWARES = {
    "scrapy_stealth.middlewares.StealthDownloaderMiddleware": 950,
}

# 2. (Optional) Route ALL requests through stealth automatically — no meta needed per request
STEALTH_ENABLED = True
STEALTH_DRIVER = "turbo"  # "basic" (default), "turbo", "browser", or "auto"
STEALTH_AUTO_FALLBACK = True  # retry basic/turbo JS challenges once with browser (headless=False)

# 3. (Optional) Proxy list — seeded as engine default; rotated on ban-streak session recycle
#    Supported schemes: http, https, socks4, socks5
STEALTH_PROXIES = [
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://user:pass@proxy3:8080",  # with authentication
    "socks5://proxy4:1080",
]

# 4. (Optional) Pin hosts to fixed origin IPs (bypass public DNS)
#    Connects to the IP while keeping the hostname for TLS SNI / Host / certs
STEALTH_DNS_OVERRIDES = {
    "example.com": "203.0.113.10",
    "www.example.com": "203.0.113.10",
}

class MySpider(scrapy.Spider):
    name = "example"

    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_stealth.middlewares.StealthDownloaderMiddleware": 950,
        },
        "STEALTH_ENABLED": True,
        "STEALTH_DRIVER": "turbo",
        "STEALTH_AUTO_FALLBACK": True,
        "STEALTH_PROXIES": [
            "http://proxy1:8080",
            "http://user:pass@proxy2:8080",
            "socks5://proxy3:1080",
        ],
        "STEALTH_DNS_OVERRIDES": {
            "example.com": "203.0.113.10",
        },
    }

🚀 Quick Start

yield scrapy.Request(
    url="https://example.com",
    meta={"stealth": {}},
)

# settings.py or custom_settings
STEALTH_ENABLED = True
STEALTH_DRIVER = "turbo"
STEALTH_AUTO_FALLBACK = True  # optional: basic/turbo -> browser on JS challenge

# No meta needed — all requests go through stealth
yield scrapy.Request(url="https://example.com")

# Opt out for a specific request
yield scrapy.Request(url="https://api.internal/health", meta={"stealth": False})

# settings.py or custom_settings — enable fallback for all stealth requests
STEALTH_ENABLED = True
STEALTH_DRIVER = "turbo"  # primary: fast HTTP driver
STEALTH_AUTO_FALLBACK = True  # retry once with browser (headless=False) on JS challenge / ban

# Or per-request — same behaviour without a global setting
yield scrapy.Request(
    url="https://example.com",
    meta={"stealth": {"driver": "auto"}},
)

🔧 Global Configuration

Customise package-wide defaults via the shared config instance. All settings must be applied at module level, before the spider class — the engine client is created at middleware initialisation, so changes inside start_requests or parse will have no effect.

# myspider.py
import scrapy
from scrapy_stealth.config import config

config.DEFAULT_ENGINE = "stealth"  # "scrapy" (native) or "stealth" (browser impersonation)
config.DEFAULT_PROFILE = "chrome_147"  # browser profile when meta["stealth"]["profile"] is not set
config.DEFAULT_TIMEOUT = 30  # stealth request timeout in seconds
config.STEALTH_DRIVER = "turbo"  # "basic" (default), "turbo", "browser", or "auto"
config.STEALTH_AUTO_FALLBACK = True  # basic/turbo -> browser on JS challenge (headless=False)
config.HTTP2 = True  # False for servers that only support HTTP/1.1
config.BLOCK_CODES |= {407}  # extend blocked status codes (|= keeps defaults)
config.BLOCK_KEYWORDS.append("banned")  # extend blocked body-text patterns
config.BROWSER_HEADLESS = True  # browser driver: headless mode (False = visible window, more stealthy)
config.BROWSER_SETTLE_S = 4.0  # browser driver: seconds to wait after navigation for JS to finish
config.BROWSER_EXECUTABLE_PATH = "/usr/bin/brave-browser"  # custom browser binary (default: auto-detect Chrome)
config.STEALTH_RECYCLE_AFTER_BANS = 5  # recycle Chrome / HTTP sessions after 5 consecutive bans
config.BROWSER_STATIC_ASSETS_BLOCK = True  # block images/fonts/CSS/media (skipped when snapshot=True)
config.BROWSER_PROXY_BYPASS_LIST = ["example.com", "*.internal"]  # these bypass the proxy
config.STEALTH_DNS_OVERRIDES = {"example.com": "203.0.113.10"}  # pin host → origin IP


class MySpider(scrapy.Spider):
    name = "example"
    ...

# ❌ wrong — too late, the engine client is already created
class MySpider(scrapy.Spider):
    def start_requests(self):
        config.HTTP2 = False  # has no effect
        ...

You can also read any value programmatically:

config.get("DEFAULT_ENGINE")          # "scrapy"
config.get("MISSING_KEY", "default")  # "default"

For one-off overrides on a single request, set meta["stealth"]["driver"] or meta["stealth"]["http2"] (see Per-Request Configuration below).

⚙️ Per-Request Configuration

All options are passed via request.meta["stealth"].

The presence of meta["stealth"] (a dict) activates the stealth engine. Omit the key to use the default Scrapy engine. When STEALTH_ENABLED = True, all requests are stealth by default — pass meta={"stealth": False} to opt out for a specific request.

yield scrapy.Request(
    url,
    meta={
        "stealth": {
            "driver": "turbo",
            # optional overrides — otherwise profile/proxy come from defaults and
            # rotate automatically when the session recycles after consecutive bans
            "profile": "chrome_147",
            "proxy": "http://user:pass@proxy:8080",
            "stealth_timeout": 60,
            "http2": True,
            "dns": "203.0.113.10",  # or {"example.com": "203.0.113.10"}
        }
    },
)

🧭 Custom DNS Overrides

Pin a hostname to a fixed origin IP so the package dials that address directly instead of trusting public DNS. The request URL stays as https://example.com/... — TLS SNI, the Host header, and certificate verification still use the hostname.

STEALTH_DNS_OVERRIDES = {
    "shop.example.com": "203.0.113.10",
    "cdn.example.com": "203.0.113.11",
}

yield scrapy.Request(
    "https://shop.example.com/item/1",
    meta={"stealth": {"driver": "turbo", "dns": "203.0.113.10"}},
)

# Or a full mapping:
meta = {"stealth": {"dns": {"shop.example.com": "203.0.113.10"}}}

🧠 Smart Browser Selection

Pick the right driver automatically: stay on fast HTTP impersonation (basic / turbo) for normal pages, and escalate to real Chrome only when the response looks like a JS challenge or session ban (403/429/503, Cloudflare “Just a moment”, Akamai, DataDome, and similar signals).

The fallback always opens a visible Chrome window (headless=False) for better evasion — regardless of BROWSER_HEADLESS or any prior meta["stealth"]["headless"] value.

STEALTH_ENABLED = True
STEALTH_DRIVER = "turbo"  # or "basic" — primary HTTP driver
STEALTH_AUTO_FALLBACK = True  # off by default; set True to enable smart selection globally

yield scrapy.Request(
    url,
    meta={"stealth": {"driver": "auto"}},  # primary = STEALTH_DRIVER, else basic
)

meta = {"stealth": {"driver": "browser"}}

meta = {"stealth": {"driver": "auto", "fallback": False}}
# or globally: STEALTH_AUTO_FALLBACK = False

🖥️ Browser Engine

For sites protected by Cloudflare JS challenges or heavy JavaScript rendering, use the browser driver. It runs a real Chrome instance via the DevTools Protocol (no WebDriver), keeping one persistent browser and opening a new tab per request.

📸 Screenshots

Capture a PNG screenshot of any page rendered by the browser driver and save it to disk.

yield scrapy.Request(
    url,
    meta={
        "stealth": {
            "driver": "browser",
            "snapshot": True,
        }
    },
    callback=self.parse,
)

from scrapy_stealth.decorators import snapshot

class MySpider(scrapy.Spider):

    @snapshot
    def parse(self, response): ...

    @snapshot(path="stealth_shots/page.png")
    def parse(self, response): ...

    @snapshot(path=lambda r: r.url.split("/")[-1] + ".png")
    def parse(self, response): ...

def parse(self, response):
    shot: bytes | None = response.meta.get("snapshot_content")
    if shot is None:
        return  # screenshot was not requested or capture failed

    # Save manually
    with open("page.png", "wb") as f:
        f.write(shot)

    # Pass to a pipeline via item
    yield {"url": response.url, "screenshot": shot}

🔁 Automatic Rotation

Profile + proxy stay stable for speed (session reuse). After STEALTH_RECYCLE_AFTER_BANS consecutive bans, the session recycles and a new default profile + proxy (from STEALTH_PROXIES) are chosen automatically.

# settings.py
STEALTH_PROXIES = ["http://proxy1:8080", "http://proxy2:8080"]
STEALTH_RECYCLE_AFTER_BANS = 5  # default

# spider
yield scrapy.Request(url, meta={"stealth": {}})

Scrapy stats

After the crawl (or mid-run via crawler.stats), inspect:

# e.g. in spider_closed
stats = spider.crawler.stats.get_stats()
print(stats.get("stealth/bans"), stats.get("stealth/recycles"))

🧩 Strategies

from scrapy_stealth.strategies.proxy import ProxyRotator

proxy_rotator = ProxyRotator([
    "http://proxy1:8080",
    "http://proxy2:8080",
])

yield scrapy.Request(
    url,
    meta={
        "stealth": {
            "proxy": proxy_rotator.get(),
        }
    },
)

from scrapy_stealth.strategies.fingerprint import ProfileRotator

fp = ProfileRotator()

yield scrapy.Request(
    url,
    meta={
        "stealth": {
            "profile": fp.get(),
        }
    },
)

from scrapy_stealth.strategies.retry import RetryHandler

retry = RetryHandler()


def parse(self, response):
    if retry.should_retry(response):
        yield retry.build(response.request)
        return

🛡️ Anti-Bot Detection

from scrapy_stealth.detectors.antibot import AntiBotDetector

detector = AntiBotDetector()

if detector.is_blocked(response):
    print("Blocked!")

📊 Examples

import scrapy


class ExampleSpider(scrapy.Spider):
    name = "example"
    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_stealth.middlewares.StealthDownloaderMiddleware": 950,
        },
        "STEALTH_ENABLED": True,
        "STEALTH_DRIVER": "turbo",
    }

    def start_requests(self):
        yield scrapy.Request("https://example.com")

    def parse(self, response):
        yield {"title": response.css("title::text").get(), "url": response.url}

# from a Scrapy project
scrapy crawl stealth_demo

# or one-off
scrapy runspider examples/full_spider.py

"""Full scrapy-stealth spider example.

Run from a Scrapy project after installing scrapy-stealth::

    scrapy crawl stealth_demo

Or as a one-off script::

    scrapy runspider examples/full_spider.py
"""

from __future__ import annotations

import scrapy

from scrapy_stealth.decorators import snapshot
from scrapy_stealth.detectors import AntiBotDetector


class StealthDemoSpider(scrapy.Spider):
    """Demonstrates global stealth settings + per-request driver overrides."""

    name = "stealth_demo"

    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_stealth.middlewares.StealthDownloaderMiddleware": 950,
        },
        # Route every request through stealth (no meta needed by default).
        "STEALTH_ENABLED": True,
        "STEALTH_DRIVER": "turbo",  # "basic" | "turbo" | "browser"
        # Optional: seeded as engine default; rotated on ban-streak recycle.
        # "STEALTH_PROXIES": [
        #     "http://user:pass@proxy1:8080",
        #     "socks5://proxy2:1080",
        # ],
        # Optional: pin hosts to fixed origin IPs.
        # "STEALTH_DNS_OVERRIDES": {"example.com": "93.184.216.34"},
        "STEALTH_RECYCLE_AFTER_BANS": 5,
        "BROWSER_HEADLESS": True,
        "BROWSER_SETTLE_S": 4.0,
        "BROWSER_STATIC_ASSETS_BLOCK": True,
        "LOG_LEVEL": "INFO",
        "ROBOTSTXT_OBEY": False,
        "TELNETCONSOLE_ENABLED": False,
    }

    start_urls = [
        "https://example.com",
        "https://httpbin.org/html",
    ]

    def start_requests(self):
        # 1) Default turbo path (STEALTH_ENABLED + STEALTH_DRIVER).
        yield scrapy.Request(
            self.start_urls[0],
            callback=self.parse,
            dont_filter=True,
        )

        # 2) Force basic for a lightweight page.
        yield scrapy.Request(
            self.start_urls[1],
            callback=self.parse,
            meta={"stealth": {"driver": "basic"}},
            dont_filter=True,
        )

        # 3) Real browser + snapshot for JS / challenge-heavy pages.
        yield scrapy.Request(
            self.start_urls[0],
            callback=self.parse_browser,
            meta={
                "stealth": {
                    "driver": "browser",
                    "settle": 3.0,
                    "snapshot": True,
                    "static_assets_block": False,  # keep assets for a real screenshot
                }
            },
            dont_filter=True,
        )

        # 4) Opt out of stealth for a specific request.
        # yield scrapy.Request(
        #     "https://httpbin.org/get",
        #     callback=self.parse,
        #     meta={"stealth": False},
        # )

    def parse(self, response: scrapy.http.Response):
        detector = AntiBotDetector()
        if detector.is_blocked(response):
            self.logger.warning("Blocked response from %s", response.url)
            return

        yield {
            "url": response.url,
            "status": response.status,
            "title": response.css("title::text").get(),
            "driver": response.flags,  # e.g. ['scrapy-stealth', 'turbo']
            "bytes": len(response.body),
        }

    @snapshot  # saves PNG under stealth_snapshots/ when snapshot=True was set
    def parse_browser(self, response: scrapy.http.Response):
        yield {
            "url": response.url,
            "status": response.status,
            "title": response.css("title::text").get(),
            "driver": response.flags,
            "has_snapshot": bool(response.meta.get("snapshot_content")),
        }

    def closed(self, reason: str):
        stats = self.crawler.stats.get_stats()
        self.logger.info(
            "Done (%s) — requests=%s successes=%s bans=%s recycles=%s driver=%s",
            reason,
            stats.get("stealth/requests"),
            stats.get("stealth/successes"),
            stats.get("stealth/bans"),
            stats.get("stealth/recycles"),
            stats.get("stealth/driver"),
        )

⚡ Performance Insight

Using stealth selectively offers key structural improvements:

• ⚡ Faster crawling (Scrapy for simple pages)
• 💰 Lower proxy cost
• 🛡️ Better success rate on protected pages

📜 Dynamic Release Tracker

Recent Artifact Versions (GitHub API)

🤝 Contributing & Project Links

See CHANGELOG.md for a full history of changes, browse GitHub Releases, or review guidelines in CONTRIBUTING.md to contribute.