discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
History and source changes · Reports and owner responses · Compare
Similar
- scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Why this match
topics crawler, scraping, web-scraping; license differs: Apache-2.0 → BSD-3-Clause
- code4craft/webmagic
A scalable web crawler framework for Java.
Why this match
topics crawler, scraping; language differs: Python → Java
- firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Why this match
topics crawler, html-to-markdown, llm; language differs: Python → TypeScript; license differs: Apache-2.0 → AGPL-3.0
- datawhalechina/happy-llm
📚 从零开始构建大模型
Why this match
topics llm, rag; language differs: Python → Jupyter Notebook
- unclecode/crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Why this match
topics crawler, llm, rag
- codelucas/newspaper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Why this match
topics crawler, news-aggregator; license differs: Apache-2.0 → MIT
- alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Why this match
topics crawler, scraping, web-scraping; license differs: Apache-2.0 → MIT
- ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
Why this match
topics crawler, llm, rag; license differs: Apache-2.0 → MIT