discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
History and source changes · Reports and owner responses · Compare
Similar
- gocolly/colly
Elegant Scraper and Crawler Framework for Golang
Why this match
topics crawler, crawling, scraping; language differs: JavaScript → Go
- scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Why this match
topics crawler, crawling, scraping; language differs: JavaScript → Python
- D4Vinci/Scrapling
Scrapling — Python repository. Updated description pending.
Why this match
topics crawler, crawling, crawling-python; language differs: JavaScript → Python
- alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Why this match
topics crawler, scraping, web-scraping; language differs: JavaScript → Python
- ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
Why this match
topics crawler, scraping, scraping-python; language differs: JavaScript → Python
- code4craft/webmagic
A scalable web crawler framework for Java.
Why this match
topics crawler, scraping; language differs: JavaScript → Java
- apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Why this match
topics crawler, crawling, scraping; language differs: JavaScript → TypeScript
- jhao104/proxy_pool
Python ProxyPool for web spider
Why this match
topics crawler, spider; language differs: JavaScript → Python