DataEval/dingo

👋 join us on Discord and WeChat
If you like Dingo, please give us a ⭐ on GitHub!
Introduction
**Dingo is A Comprehensive AI Data, Model and Application Quality Evaluation Tool**, designed for ML practitioners, data engineers, and AI researchers. It helps you systematically assess and improve the quality of training data, fine-tuning datasets, and production AI systems.
---
🚀 Enterprise Dingo SaaS Version
Need a **production-grade data quality platform**? Try Dingo SaaS Enterprise Edition!
✨ Compared to the open-source version, SaaS provides:
- 🌐 **Web UI** - Visual evaluation interface, no coding required
- 🔐 **Access Control** - JWT + Google OAuth 2.0
- 📊 **Visual Reports** - Interactive charts, trend analysis, export features
- 🔌 **RESTful API** - Seamless integration with existing systems
SaaS — Dashboard
SaaS — Playground
SaaS — Evaluation output
📝 How to Get Free SaaS Code
👉 **Apply for Dingo SaaS Repository Access**
Review time: 1-5 business days | Suitable for enterprise data governance, team collaboration
---
Why Dingo?
🎯 **Production-Grade Quality Checks** - From pre-training datasets to RAG systems, ensure your AI gets high-quality data
🗄️ **Multi-Source Data Integration** - Seamlessly connect to Local files, SQL databases (PostgreSQL/MySQL/SQLite), HuggingFace datasets, and S3 storage
🔍 **Multi-Field Evaluation** - Apply different quality rules to different fields in parallel (e.g., ISBN validation for isbn, text quality for title)
🤖 **RAG System Assessment** - Comprehensive evaluation of retrieval and generation quality with 5 academic-backed metrics
🧠 **LLM & Rule & Agent Hybrid** - Combine fast heuristic rules (30+ built-in) with LLM-based deep assessment
🚀 **Flexible Execution** - Run locally for rapid iteration or scale with Spark for billion-scale datasets
📊 **Rich Reporting** - Detailed quality reports with GUI visualization and field-level insights
Architecture Diagram
Quick Start
Installation
Example Use Cases of Dingo
1. Evaluate LLM chat data
2. Evaluate Dataset
Command Line Interface
Evaluate with Rule Sets
Evaluate with LLM (e.g., GPT-4o)
---
MCP Server
Dingo includes a built-in Model Context Protocol (MCP) server for AI agent integration:
For detailed setup and Cursor/Claude Desktop integration, see the dedicated documentation:
English · 简体中文 · 日本語
Video Demonstration
To help you get started quickly with Dingo MCP, we've created a video walkthrough:
https://github.com/user-attachments/assets/aca26f4c-3f2e-445e-9ef9-9331c4d7a37b
This video demonstrates step-by-step how to use Dingo MCP server with Cursor.
---
📚 Data Quality Metrics
Dingo provides **100+ evaluation metrics** across multiple dimensions, combining rule-based speed with LLM-based depth.
Metric Categories
| Category | Examples | Use Case |
|----------|----------|----------|
| **Pretrain Text Quality** | Completeness, Effectiveness, Similarity, Security | LLM pre-training data filtering |
| **SFT Data Quality** | Honest, Helpful, Harmless (3H) | Instruction fine-tuning data |
| **RAG Evaluation** | Faithfulness, Context Precision, Answer Relevancy | RAG system assessment |
| **Hallucination Detection** | HHEM-2.1-Open, Factuality Check | Production AI reliability |
| **Classification** | Topic categorization, Content labeling | Data organization |
| **Multimodal** | Image-text relevance, VLM quality, OCR visual evaluation | Vision-language data |
| **Security** | PII detection, Perspective API toxicity | Privacy and safety |
📊 **View Complete Metrics Documentation →**
📖 **RAG Evaluation Guide →** | **中文版**
🔍 **Hallucination Detection Guide →** | **中文版**
✅ **Factuality Assessment Guide →** | **中文版**
👁️ **VLM Render Judge Guide →** | **中文版**
Most metrics are backed by academic research to ensure scientific rigor.
Quick Metric Usage
**Customization**: All prompts are defined in dingo/model/llm/ directory (organized by category: text_quality/, rag/, hhh/, etc.). Extend or modify them for domain-specific requirements.