onlymuneeb38-glitch/ios-agent-skills-evaluator

🚀 The Ultimate Testbed for iOS AI Agent Capabilities
Welcome to **iOS Agent Skills Pro** — a comprehensive, open-source evaluation framework designed to stress-test the cognitive and operational skills of AI agents operating on iOS devices. This repository is not just another testing toolkit; it is a **neural gymnasium** where your AI agents train, compete, and evolve through 11 meticulously crafted tasks, 260+ real-world navigation scenarios, and 850+ granular assertions across three cutting-edge models.
Whether you are a machine learning researcher, a QA engineer, or a developer building autonomous iOS assistants, this framework provides the structured chaos needed to validate agent performance in production-like environments.
---
🧠 Repository Vision: Beyond Benchmarks
Traditional agent testing is like teaching a parrot to recite phrases. **iOS Agent Skills Pro** is different — it is the **Swiss Army knife of agent evaluation**. We simulate ambiguous user intents, broken UI states, and multi-step reasoning chains. Our 850+ assertions act as **microscopic neural checkpoints**, ensuring your agent doesn't just *appear* smart but *actually* navigates iOS interfaces with human-like adaptability.
**Key differentiators from existing repos:**
- Not a static benchmark but a **living test suite** that grows with iOS updates
- Supports **Claude API and OpenAI API** for model-agnostic evaluation
- Generates **radar charts of agent weaknesses** (e.g., fails on dynamic lists, succeeds on static buttons)
- Includes **responsive UI validation** — tests how agents react to device rotation, split-screen, and dark mode
---
📥 Quick Start: Download and Setup
Prerequisites
- iOS Simulator (Xcode 15+)
- Python 3.10+
- OpenAI API key or Anthropic API key
Installation in 30 Seconds
1. Download the repository from the link above
2. Run pip install -r requirements.txt
3. Configure your API keys in config.yaml
---
🔮 Architecture Overview (Mermaid Diagram)
*Figure 1: The agent evaluation pipeline — from user intent to assertion verification.*
---
⚙️ Example Profile Configuration
Create a profile.yaml file to customize your agent's behavioral parameters:
**Why this matters:** The max_steps parameter prevents infinite loops during tricky tasks like "find the WiFi password in Settings > General > About." The forbidden_actions field mimics real-world safety guardrails.
---
💻 Example Console Invocation
Run a full evaluation suite with a single command:
Expected output:
*This real-time feedback loop is the **heartbeat** of the framework — you know exactly where your agent stumbles.*
---
📱 Operating System Compatibility
| iOS Version | Status | Notes |
|-------------|--------|-------|
| iOS 18.0+ | ✅ Full Support | All 260+ scenarios verified |
| iOS 17.4 - 17.9 | ✅ Supported | ClipKit and Live Activities may vary |
| iOS 16.6 - 17.3 | ⚠️ Partial | Some gesture-based assertions disabled |
| iOS 15.x | ❌ Deprecated | Minimum requirement is iOS 16.6 |
**Emoji Legend:** ✅ = Certified | ⚠️ = Limited | ❌ = Not Supported
The framework auto-detects the OS version and adjusts assertion thresholds accordingly — a **feature born from frustration** with brittle test suites that break on every iOS update.
---
🌟 Comprehensive Feature List
Core Capabilities
- **11 Task Categories** — from basic tapping to complex multi-step workflows involving Apple Pay, Shortcuts, and Siri integration