funkyfunc/filegoblin

> **"Ingesting the messy world, spitting out clean context."**
**filegoblin** is a single-binary CLI that "gobbles" almost any file or website — PDF, Word, Excel, PowerPoint, email, EPUB, HTML, whole repos — and converts it into clean, LLM-ready markdown.
Philosophy
- **It just works.** filegoblin document.pdf > context.md needs no manual. Power flags exist, but the default path is the product.
- **Format invisibility.** You shouldn't have to think about extensions; detection is extension-first with magic-byte fallback, so renamed and extension-less files route correctly.
- **Fail loudly, never silently.** An empty result exits non-zero, parse failures are reported in-band, and invalid flags are hard errors. An agent can trust every exit code.
- **Local and private.** Parsing and PII scrubbing happen entirely on your machine. No cloud APIs, no runtimes, no pip install.
- **Not just a concatenator.** cat * > out.txt exists; filegoblin adds structure (real markdown from Word, one consistent table dialect, readability-grade web extraction), token budgeting, and redaction.
---
📦 Installation
Every release comes in two flavours:
- **with the PDF vision models** for layout and reading order, formulas as LaTeX, tables, and OCR on scanned pages (~510 MB);
- **lite**, without them: the fast geometric converter, plus Apple Vision OCR on macOS.
Everything else is the same.
| Channel | Lite | With the models |
|---|---|---|
| GitHub release (latest) | filegoblin- archive | filegoblin-full- archive: one binary with the models compiled in |
| PyPI | pip install filegoblin | pip install "filegoblin[models]" |
| npm | npm i -g filegoblin | npm i -g filegoblin filegoblin-models |
| Cargo | cargo install --git https://github.com/funkyfunc/filegoblin filegoblin | then filegoblin --install-models |
Builds are published for macOS (Apple Silicon and Intel), Linux (x86_64 and aarch64), and Windows (x86_64).
**Getting the models later.**
- filegoblin --install-models downloads this version's model archive from the GitHub release, checks its SHA-256 and unpacks it into ~/.filegoblin/models. You can pass a directory instead, and set FILEGOBLIN_MODELS_URL to point at an internal mirror. filegoblin only downloads when you ask it to.
- filegoblin --models-status shows where each model was found.
**Behind a corporate network.** The PyPI and npm model packages carry the weights themselves, split into pieces under the registries' 100 MB per-file limit. They install from a company mirror (Artifactory, Nexus, CodeArtifact) like any other package, with no access to github.com.
**Air-gapped machines.** Copy the full download. Alternatively, put the filegoblin-vision-models.tar.gz archive on a shared read-only path and name it in FILEGOBLIN_MODELS. That variable takes a PATH-style list. Licenses are in LICENSES.md inside the archive.
**From source** — needs a Rust toolchain (1.87+):
That gives you everything in the "always on" list below, including web fetching and
crawling. The optional engines are opt-in because each one pulls in a heavy
dependency:
| Want | Add | Build needs |
|---|---|---|
| OCR on Linux/Windows | --features ocr | just get-brains for the weights (macOS uses Apple Vision, no flag) |
| PDF vision models (layout, reading order, formulas → LaTeX, tables, scan OCR) | --features vision (on in release builds) | The weights: filegoblin --install-models (the release's filegoblin-vision-models.tar.gz, SHA-256 checked) or just build-vision-models. filegoblin finds them in models/ next to the binary, in ~/.filegoblin/models, or in $FILEGOBLIN_MODELS. Add bundled-models with FILEGOBLIN_MODELS_DIR set to compile them into one binary |
| Audio/video transcription | --features transcribe | cmake + a C/C++ compiler; downloads a ~574 MB model on first use |
| Metal-accelerated transcription (macOS only) | --features transcribe-metal | Xcode Command Line Tools. **Does not build on Linux/Windows** |
| WASM plugins | --features wasm | — |
| BM25 search (--search) | --features rag | — |
Without the models, --vision smart (the default) behaves exactly like the fast converter, with one note per run on stderr; --vision off silences it. With the models, it uses them only on the pages that need them. Anyone building from source can make a single self-contained binary with --features vision,bundled-models and FILEGOBLIN_MODELS_DIR set.
---
**Measuring quality.** just gauntlet runs the 29-case adversarial gauntlet