Web Crawling and Scraping Tools

Explore powerful open source web crawling and scraping solutions for developers. Find scalable, self-hosted tools for data extraction, web indexing, and research.

Open Source Projects tagged "Web Crawling and Scraping Tools"

firecrawl

52,2514,515

Firecrawl is an API that turns any URL into clean, LLM‑ready data. It crawls sites (no sitemap required), scrapes dynamic pages, and outputs markdown, HTML, structured JSON, screenshots, links, and metadata.

website-crawler

llm-data-extraction

web-crawling

Last updated: 8/26/20254515 contributors