brainstate

discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens

apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

TypeScript★ 26.3kApache-2.0GitHub ↗

History and source changes · Reports and owner responses · Compare

Similar

  1. apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

    Python★ 9.6kApache-2.0GitHub ↗

    Why this match

    topics apify, automation, crawler; language differs: TypeScript → Python

  2. unclecode/crawl4ai

    Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf

    Python★ 86.9kApache-2.0GitHub ↗

    Why this match

    topics crawler, playwright, scraper; language differs: TypeScript → Python

  3. gocolly/colly

    Elegant Scraper and Crawler Framework for Golang

    Go★ 25.6kApache-2.0GitHub ↗

    Why this match

    topics crawler, crawling, scraper; language differs: TypeScript → Go

  4. lexmount/moli

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    Rust★ 22.5kApache-2.0GitHub ↗

    Why this match

    topics headless, playwright, puppeteer; language differs: TypeScript → Rust

  5. scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python★ 65.2kBSD-3-ClauseGitHub ↗

    Why this match

    topics crawler, crawling, scraping; language differs: TypeScript → Python; license differs: Apache-2.0 → BSD-3-Clause

  6. jo-inc/camofox-browser

    Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

    JavaScript★ 12.1kMITGitHub ↗

    Why this match

    topics automation, javascript, nodejs; language differs: TypeScript → JavaScript; license differs: Apache-2.0 → MIT

  7. alirezamika/autoscraper

    A Smart, Automatic, Fast and Lightweight Web Scraper for Python

    Python★ 8.1kMITGitHub ↗

    Why this match

    topics automation, crawler, scraper; language differs: TypeScript → Python; license differs: Apache-2.0 → MIT

  8. getmaxun/maxun

    Turn any website into a structured API. Extract, automate, search and monitor the web.

    TypeScript★ 18.1kAGPL-3.0GitHub ↗

    Why this match

    topics automation, crawler, crawling; license differs: Apache-2.0 → AGPL-3.0