kubeflow/mcp-apache-spark-history-server

> **Connect AI agents and engineers to Apache Spark History Server for intelligent job analysis, performance monitoring, and investigation**
---
> [!IMPORTANT]
> ### ✨ NEW — Spark History Server CLI is now available
>
>
> A standalone Go binary that queries Spark History Server **directly from your terminal** — no MCP, no AI framework, no daemon process. Inspect jobs, compare runs, investigate failures, and script against the Spark REST API.
>
> **Get started with the SHS CLI →**
---
This project provides **two interfaces** to your Spark History Server data:
| | 🛠️ SHS CLI (shs) | ⚡ MCP Server |
|---|---|---|
| **For** | Engineers, shell scripts, CI/CD, coding agents | AI agents and MCP-compatible clients |
| **Mental model** | "I know the command I want to run" | "Agent, investigate this Spark app" |
| **Install** | Single static binary — no dependencies | Python 3.12+, uv |
| **Get started** | CLI docs → | MCP docs → |
📺 **See it in action:**
---
🛠️ SHS CLI (shs) — For Engineers & Scripts
A standalone Go binary. Query your Spark History Server directly from the terminal, shell scripts, or CI/CD pipelines. Also works as a **skill** for coding agents like Claude Code and Kiro.
Install
Quick Start
**CLI documentation** for full usage, or check out a real-world example of Claude Code comparing two **TPC-DS 3TB benchmark** runs.
---
⚡ MCP Server — For AI Agents
An MCP (Model Context Protocol) server that exposes Spark History Server data as tools for AI agents. Agents query your Spark infrastructure using natural language — the server handles tool selection, multi-server routing, and structured data retrieval.
**Use the MCP server when** you want an AI agent to conduct multi-step investigations, synthesize findings across tools, or answer natural-language questions about your Spark applications.
Install
The package is published to PyPI.
Coding Agent Integration
Register the server with a single command. Both examples run it over **stdio** via uvx.
With no config file present, the server defaults to a Spark History Server at http://localhost:18080;
point it elsewhere with a config file or SHS_SERVERS__LOCAL__URL.
**Claude Code** (claude mcp add):
**Kiro CLI** (kiro-cli mcp add):
Verify in either client with claude mcp list / kiro-cli mcp list, then ask the agent to *"list the available Spark applications."*
The server also ships **prompts** — guided, multi-step workflows you run as a command. In Claude Code: /mcp__spark-history__investigate_failure . In Kiro CLI: /prompts investigate_failure (or @investigate_failure). See Prompts for the full list and arguments.
Passing server flags and environment
The commands above have two layers: the **client's** own options and the arguments/environment forwarded to **spark-mcp**. spark-mcp itself takes a single flag, --config / -c; everything else is set through SHS_* environment variables.
| To pass to spark-mcp… | Claude Code | Kiro CLI |
|---|---|---|
| A flag (e.g. --config) | append after --: … spark-mcp --config /path/config.yaml | add --args pairs: --args --config --args /path/config.yaml |
| An environment variable | --env KEY=value (before --transport) | --env KEY=value |
For example, to point at a remote Spark History Server with an explicit config file:
Configure
Basic configuration below. Create a file named config.yaml:
Config file location
The server looks for its config file in the following order and uses the first one it finds:
1. The --config / -c flag (e.g. spark-mcp --config /path/to/config.yaml)
2. The SHS_MCP_CONFIG environment variable
3. ./config.yaml in the current working directory
4. ~/.config/spark-mcp/config.yaml (honors $XDG_CONFIG_HOME when set)
If none exist, the server starts with built-in defaults that can be overridden by SHS_* environment variables. When a path is given explicitly via the flag or SHS_MCP_CONFIG but the file is missing, the server fails fast instead of falling back.