andyliszewski/webcrawl-mcp

What it does

webcrawl-mcp is an MCP server that provides web scraping, crawling, and search capabilities. It prioritizes local-first processing using trafilatura to extract content from static web pages, with an optional Firecrawl integration for handling JavaScript-heavy sites when needed. The tool offers four main functions: scrape individual pages, search using DuckDuckGo, map website structures, and crawl multiple pages.

How to set up

Install the package from the GitHub repository and configure it as an MCP server. Basic setup involves cloning the repository and following the installation instructions. For optional Firecrawl functionality, provide API credentials when JavaScript rendering is required for specific pages.

Key features

• Local-first web scraping using trafilatura for efficient content extraction
• DuckDuckGo search integration for web queries
• Optional Firecrawl fallback for JavaScript-rendered content
• Web crawling and mapping capabilities
• Four tools: scrape, search, map, and crawl
• Privacy-focused with local processing as default