brainstate

discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens

gocolly/colly

Elegant Scraper and Crawler Framework for Golang

Go★ 25.6kApache-2.0GitHub ↗

History and source changes · Reports and owner responses · Compare

Similar

  1. code4craft/webmagic

    A scalable web crawler framework for Java.

    Java★ 11.7kApache-2.0GitHub ↗

    Why this match

    topics crawler, framework, scraping; language differs: Go → Java

  2. scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python★ 65.2kBSD-3-ClauseGitHub ↗

    Why this match

    topics crawler, crawling, framework; language differs: Go → Python; license differs: Apache-2.0 → BSD-3-Clause

  3. codelucas/newspaper

    newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

    Python★ 15.2kMITGitHub ↗

    Why this match

    topics crawler, crawling, scraper; language differs: Go → Python; license differs: Apache-2.0 → MIT

  4. BruceDone/awesome-crawler

    A collection of awesome web crawler,spider in different languages

    ★ 7.4kMITGitHub ↗

    Why this match

    topics crawler, scraper, spider; license differs: Apache-2.0 → MIT

  5. apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    TypeScript★ 26.3kApache-2.0GitHub ↗

    Why this match

    topics crawler, crawling, scraper; language differs: Go → TypeScript

  6. lorien/awesome-web-scraping

    List of libraries, tools and APIs for web scraping and data processing.

    JavaScript★ 8.2kGitHub ↗

    Why this match

    topics crawler, crawling, scraping; language differs: Go → JavaScript

  7. jhao104/proxy_pool

    Python ProxyPool for web spider

    Python★ 23.8kMITGitHub ↗

    Why this match

    topics crawler, spider; language differs: Go → Python; license differs: Apache-2.0 → MIT

  8. apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

    Python★ 9.6kApache-2.0GitHub ↗

    Why this match

    topics crawler, crawling, scraper; language differs: Go → Python