brainstate

discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens

apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Python★ 9.6kApache-2.0GitHub ↗

History and source changes · Reports and owner responses · Compare

Similar

  1. apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    TypeScript★ 26.3kApache-2.0GitHub ↗

    Why this match

    topics apify, automation, crawler; language differs: Python → TypeScript

  2. unclecode/crawl4ai

    Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

    Python★ 86.9kApache-2.0GitHub ↗

    Why this match

    topics crawler, playwright, python

  3. scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python★ 65.2kBSD-3-ClauseGitHub ↗

    Why this match

    topics crawler, crawling, python; license differs: Apache-2.0 → BSD-3-Clause

  4. gocolly/colly

    Elegant Scraper and Crawler Framework for Golang

    Go★ 25.6kApache-2.0GitHub ↗

    Why this match

    topics crawler, crawling, scraper; language differs: Python → Go

  5. alirezamika/autoscraper

    A Smart, Automatic, Fast and Lightweight Web Scraper for Python

    Python★ 8.1kMITGitHub ↗

    Why this match

    topics automation, crawler, python; license differs: Apache-2.0 → MIT

  6. codelucas/newspaper

    newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

    Python★ 15.2kMITGitHub ↗

    Why this match

    topics crawler, crawling, python; license differs: Apache-2.0 → MIT

  7. D4Vinci/Scrapling

    Scrapling — Python repository. Updated description pending.

    Python★ 95.8kBSD-3-ClauseGitHub ↗

    Why this match

    topics automation, crawler, crawling; license differs: Apache-2.0 → BSD-3-Clause

  8. getmaxun/maxun

    Turn any website into a structured API. Extract, automate, search and monitor the web.

    TypeScript★ 18.1kAGPL-3.0GitHub ↗

    Why this match

    topics automation, crawler, crawling; language differs: Python → TypeScript; license differs: Apache-2.0 → AGPL-3.0