Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.
To add evalsmith-mcp to Claude, run the install command below or copy the ready-made config for Claude Code, Claude Desktop, Cursor and VS Code further down. Restart your client after adding it and its tools show up in your next conversation. It needs 4 environment variables, listed below.
Install
npx -y evalsmith-mcp@0.1.0Environment variables
EVALSMITH_JUDGE_BASE_URLSet in your MCP configEVALSMITH_JUDGE_MODELSet in your MCP configEVALSMITH_JUDGE_API_KEYSet in your MCP configLOG_LEVELSet in your MCP configInstall with your client
claude mcp add evalsmith --env EVALSMITH_JUDGE_BASE_URL=<YOUR_EVALSMITH_JUDGE_BASE_URL> --env EVALSMITH_JUDGE_MODEL=<YOUR_EVALSMITH_JUDGE_MODEL> --env EVALSMITH_JUDGE_API_KEY=<YOUR_EVALSMITH_JUDGE_API_KEY> --env LOG_LEVEL=<YOUR_LOG_LEVEL> -- npx -y evalsmith-mcp@0.1.0
Run this in a terminal to add the server to Claude Code.
What this does
evalsmith-mcp exposes a set of tools to Claude over the Model Context Protocol. After you add it to claude_desktop_config.json (snippet above) and restart Claude Desktop, those tools become callable inside any conversation.
Requirements
This server needs the following environment variables to be set before it can run:
EVALSMITH_JUDGE_BASE_URL EVALSMITH_JUDGE_MODEL EVALSMITH_JUDGE_API_KEY LOG_LEVEL
Set these via the env object in your MCP config (see claude_desktop_config.json snippet above).
Frequently asked questions
How do I install evalsmith-mcp?
Run the Claude Code command in the install section to add evalsmith-mcp to Claude Code, or copy the ready-made config for your client from the install section on this page. Restart your client afterwards and the tools appear in your next conversation.
Which clients work with evalsmith-mcp?
This page carries setup and config for Claude Code, Claude Desktop, Cursor and VS Code. Any other MCP client that accepts a standard server config can connect to evalsmith-mcp as well.
Does evalsmith-mcp need an API key?
Yes. Before it can run, set these environment variables: EVALSMITH_JUDGE_BASE_URL, EVALSMITH_JUDGE_MODEL, EVALSMITH_JUDGE_API_KEY, LOG_LEVEL. They go in the env block of your MCP config.
Imported from the Official MCP Registry. The details above come from the registry entry; the maker can claim this listing to correct them, add docs, and see its traffic.
Is this your MCP server?
Claim this listing to see its traffic, fix details yourself, and skip the review queue on future updates.
Add this badge to your README
Show visitors this server is listed in the ClaudeRules directory. The badge links straight to this listing.
[](https://clauderules.net/mcp/io-github-diegoperezg7-evalsmith-mcp)<a href="https://clauderules.net/mcp/io-github-diegoperezg7-evalsmith-mcp" target="_blank" rel="noopener noreferrer"><img src="https://clauderules.net/api/badge/io-github-diegoperezg7-evalsmith-mcp" alt="Listed on clauderules.net" /></a>More Search MCP servers
GitHub
Connect AI assistants to GitHub - manage repos, issues, PRs, and workflows through natural language.
Serena MCP: the IDE for your agent
A powerful toolkit for coding, providing semantic retrieval and editing capabilities.
mediar-ai/screenpipe
🎖️ 🦀 🏠 🍎 Local-first system capturing screen/audio with timestamped indexing, SQL/embedding storage, semantic search, LLM-powered history analysis, and event-triggered actions - enables building c
Figma-Context-MCP
Give your coding agent access to your Figma data. Implement designs in any framework in one-shot.