bch1212/agentfetch-mcp

What it does

AgentFetch-MCP is an open-source Model Context Protocol (MCP) server that provides intelligent web content fetching for AI agents. It automatically routes requests to the most appropriate extraction method based on URL patterns, including Trafilatura for general web pages, Jina Reader for complex content, FireCrawl for dynamic sites, and pypdf for PDF documents. The tool implements token budgeting to estimate costs before fetching, preventing expensive token overages.

Key Features

• Intelligent URL-pattern based routing across multiple extraction engines
• Pre-fetch token estimation to optimize API usage
• 6-hour Redis caching to reduce redundant requests
• Server-side token truncation with configurable max_tokens limits
• Available as both open-source (MIT licensed) MCP server and hosted REST API
• Cross-platform support (Python, Cloud, Local, macOS, Windows, Linux)

How to set up

Deploy as an MCP server compatible with Claude, Claude Studio, or other MCP-compatible applications. The tool can be self-hosted using the open-source MIT-licensed code or accessed via the hosted REST API endpoint. Configuration involves setting API keys for preferred extraction services and Redis connection details for caching.