D4Vinci/Scrapling

Claims below include README evidence. A matching quote or worker agreement does not establish that a claim is correct. Category majority among 2 workers. Version 1.

Scrapling is a Python web scraping framework that combines fast HTTP requests, stealth browsers, parsing and concurrent crawling in one library.

Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

It covers the full scraping workflow, from extracting a few fields to collecting a whole website as clean Markdown, from Python, a terminal or an AI agent.

From extracting a few fields to collecting a whole website as clean Markdown, Scrapling covers the full scraping workflow.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

It combines fast HTTP requests, stealth browsers, JavaScript rendering and Cloudflare challenge handling behind one interface.

Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Parses with CSS, XPath, text and regex search plus adaptive element tracking.

CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Concurrent spiders with proxy rotation, retries, pause and resume, and JSON, JSONL, CSV or XML exports.

Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Exposes an MCP server, an Agent Skill, CLI commands and an interactive shell.

An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Keeps persistent sessions and can capture the background API calls behind a page.

persistent sessions, and background API capture

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Blocks chosen domains and known ad or tracker domains in browser-based fetchers.

Block requests to specific domains (and their subdomains) or enable built-in ad blocking

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Can route DNS through DNS-over-HTTPS to prevent leaks when proxies are used.

Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Turns a page into clean, LLM-ready Markdown.

Turn any page into clean, sanitized, LLM-ready Markdown with one line

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Spider templates for rule-based crawling, sitemaps, XML and CSV feeds, and Shopify stores.

`CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, `XMLFeedSpider`/`CSVFeedSpider` for iterating XML/RSS and CSV feeds

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Handles Cloudflare Turnstile challenges

Scrapling handles Cloudflare Turnstile

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Adaptive element tracking and similar-element matching

adaptive element tracking, similar-element matching, and clean Markdown output

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Concurrent spiders with proxy rotation, retries and pause/resume

Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Exports to JSON, JSONL, CSV and XML

JSON/JSONL/CSV/XML exports

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Limitation: Robots.txt compliance is an optional flag rather than the default.

Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Limitation: Captured background API history is bounded and stores text response bodies.

History is bounded and saves text response bodies.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

Alternative to Scrapy

**Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects.

Quote matched the README or GitHub description at extraction. Same claim and quote from 1 worker(s). Interpretation not reviewed by a maintainer.

developer-tools · data