discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens
BruceDone/awesome-crawler
A collection of awesome web crawler,spider in different languages
History and source changes · Reports and owner responses · Compare
Similar
- gocolly/colly
Elegant Scraper and Crawler Framework for Golang
Why this match
topics crawler, scraper, spider; license differs: MIT → Apache-2.0
- jhao104/proxy_pool
Python ProxyPool for web spider
Why this match
topics crawler, spider
- s0md3v/Photon
Incredibly fast crawler designed for OSINT.
Why this match
topics crawler, spider; license differs: MIT → GPL-3.0
- firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Why this match
topics crawler, scraper, web-crawler; license differs: MIT → AGPL-3.0
- ssssssss-team/spider-flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Why this match
topics crawler, spider, web-crawler
- unclecode/crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Why this match
topics crawler, scraper, web-crawler; license differs: MIT → Apache-2.0
- binux/pyspider
A Powerful Spider(Web Crawler) System in Python.
Why this match
topics crawler; license differs: MIT → Apache-2.0
- codelucas/newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Why this match
topics crawler, scraper