ErxStudio: a personal AI daemon
A local-first AI daemon and desktop app that hosts GGUF models in-process with llama.cpp, runs tool-using agents, and serves one HTTP API to every other front end.
Overview
ErxStudio is one product with two halves. The daemon is a FastAPI service that owns the GPU: it loads local GGUF models in-process through llama-cpp-python, falls back to an API model (Anthropic, OpenRouter or any OpenAI-compatible provider) when allowed, and exposes a single HTTP API. The GUI is a PySide6 window over that API, with chat, model and log tabs. A dependency-free web page serves the same chats to other devices on the home network. The daemon also speaks llama.cpp-server and OpenAI-compatible dialects, so existing front ends can use it as their backend and pick up its agents, tools and privacy rules. Source not public.
Architecture
| Layer | Implementation | Purpose |
|---|---|---|
| Engine | llama-cpp-python on a dedicated thread | One local model at a time, VRAM-aware offload and context sizing |
| Routing | local, api or auto policy per request |
Local first, API fallback only when the chat permits it |
| Agent runs | Background asyncio tasks with event logs | Runs outlive the HTTP request; clients detach and re-attach |
| Tools | LangChain tools, MCP client, bubblewrap sandbox | File, shell, git, web and RAG tools with per-call approval |
| Storage | aiosqlite with numbered SQL migrations | Conversations, messages, run events, agents, settings |
| RAG | LlamaIndex with a CPU embedding model | Per-project collections injected when history outgrows the window |
Technical deep dive
Engine. The model library is a folder of GGUF files. A header reader pulls each file’s
architecture, trained context, chat template and recommended sampling from its metadata, so a GGUF
needs no other config. At load time the backend plans GPU offload against free VRAM, retrying with
fewer layers if the load fails, and picks the largest power-of-two context whose q8_0 KV cache
fits beside the weights, accounting for hybrid and sliding-window architectures. Every llama.cpp
call runs on one thread so the async event loop never blocks.
Speculative decoding. I replaced Llama.generate with a custom decode loop that verifies
drafts from prompt-lookup n-grams, a smaller draft model, or the model’s own multi-token-prediction
head. Drafts are checked position by position through the real sampler chain, so penalties, DRY
and grammars still apply, and the KV cache is cut back on the first mismatch. The draft length
adapts, and drafting pauses after repeated misses. The MTP path ports llama.cpp’s C++ driver,
which the Python bindings don’t wrap: it patches default-params functions for the loading
thread and calls C++ symbols by their mangled names. MTP is also switched off when it would push
layers to the CPU, where it is slower than plain decoding.
Runs and approvals. Each chat message becomes a run: an asyncio task that saves every turn and
tool result as it happens, and gives every event a sequence number. Clients follow it over SSE and
can resume from any sequence, so a reply started on the desktop can be watched and approved from a
phone. Messages sent mid-run are either injected before the next model call (“steer”) or queued.
Before a run’s first file-changing tool call, a shadow git repository snapshots the workspace
without touching the project’s own .git, which makes every agent edit undoable.
Long contexts. As a chat fills the window, old tool output shrinks to a preview, then the oldest history is summarized by the chat’s own model (undoably), including inside long agent tasks.
Security. Every request needs a bearer token, and Host headers are allow-listed against DNS
rebinding. Shell and git tools run under bubblewrap with a read-only system, an empty home
directory, a cleared environment and no network by default. If bubblewrap is missing, commands are
refused rather than run unsandboxed.
Key challenge: keeping one GPU-bound, single-sequence engine responsive to many clients and long-running agents at once. The answer was to decouple everything from the HTTP request: an explicit engine state machine with drained unloads, local runs queued on the engine while API runs proceed in parallel, and persisted run events that make disconnects, daemon restarts and cross-device approval routine.