Running a Full AI Agent Stack on My MacBook. Here’s What It Actually Takes.

Somewhere over the Atlantic, on a flight back to Zürich, I found myself thinking about how much of my daily AI workflow depended on things I didn’t control. API keys. Servers on the other side of the world. Data sent to places I’d never see. I had a good model, I had good tools, and I was completely offline at 38,000 feet, which meant I had exactly none of it.

I have to admit I didn’t go home and immediately rebuild everything from scratch. That took a few more weeks of the same quiet frustration before I actually sat down and started tinkering.

What I ended up building over a handful of weekends is something I now use every day: a fully local, fully private AI agent stack that runs entirely on my MacBook Pro. No API keys. No internet dependency. Nothing leaving the machine. It sounds like a weekend project that got away from itself, and to be honest, that is exactly what happened. But the result is something I’m genuinely happy with.

Here’s how it works and, more importantly, why I made the choices I made.

The Engine: oMLX

The core of the setup is oMLX — a background server that wraps Apple’s MLX framework and exposes a standard API at 127.0.0.1:8000. That address is key: it binds only to localhost, never to the network. Nothing outside the machine can reach it.

MLX is Apple’s machine learning framework built specifically for Apple Silicon, and what makes it interesting is unified memory. On a MacBook Pro with 128 GB of RAM, that memory is shared between CPU and GPU — which means you can load models that would be physically impossible on a typical GPU setup. The 35B parameter model I use for most work takes about 20 GB. A 122B parameter model for deep reasoning takes roughly 65 GB. These are not toy numbers.

oMLX manages loading and evicting models from that unified memory, which in practice means I can switch between models without manually juggling files or restarting processes. It handles that behind the scenes.

Four Models, Each With a Job

I settled on four models, and the allocation is intentional.

The 122B model is for deep reasoning work — the kind of task where you want a very long, careful chain of thought. It loads alone because it needs almost all the available memory. I use it rarely, but when I do, the quality difference is real.

The 35B everyday model is what I spend most time with. It handles regular work, runs my Claude Code sessions when I want them fully local, and is capable enough for most things I throw at it. It loads in about 20 GB, which leaves headroom.

A second 35B model handles sandboxed coding work — isolated from the everyday model so the two tasks don’t bleed into each other in memory or context.

The 4B model is a scout: fast, cheap to load, useful for quick summarizations and lightweight tasks where spinning up the 35B would be overkill.

Five Agents, Each With a Role

The models are the hardware. The agents are the software — five distinct processes, each with carefully scoped permissions.

The Brain is the orchestrator. It’s the single front door for every request I make. It has no web access and no credentials. It routes tasks to the appropriate agents and synthesizes results, but it touches nothing directly.

websum is a web fetcher that runs in a Docker container. It has internet access — and nothing else. No filesystem, no credentials, no ability to act on anything. Its only job is to retrieve a URL and return text.

The coder agent runs in a separate Docker sandbox. It can see exactly one project folder and can reach the local model server — and nothing else. It commits locally but cannot push to remote. It cannot exfiltrate code.

agent-approve is the human gate for anything irreversible: pushing code, sending a message, deleting a file. It pulls credentials from macOS Keychain only at the moment of execution — never storing them in memory, never exposing them to any AI step.

mailbox and brief are read-only digest agents — they retrieve and summarize, nothing more.

The Quarantine Idea (and Why It Matters)

This is the part I’m most happy with, and it came from thinking carefully about a specific problem: prompt injection.

Prompt injection is when an AI reads content from the web or an email that contains hidden instructions — text designed to hijack whatever the AI does next. It’s a real risk for any agentic system that fetches external content and then acts on it.

However, there’s a clean solution if you build the system right from the beginning: the AI step that reads untrusted content must hold no tools and no credentials. The step that holds credentials must run no AI.

In practice this means websum — the thing that touches the actual web — is completely toothless. It can read a page. It cannot do anything with what it reads except return text. That text then gets screened and handed to Brain, which has no internet access. If a malicious web page tries to inject instructions, the worst outcome is that Brain receives some weird text. It cannot act on it in any dangerous way because it holds nothing dangerous.

The quarantine is structural. It’s not a check or a filter — it’s an impossibility. And that’s a much more comfortable place to be.

What It Actually Feels Like to Use

Setting this up took longer than I expected. There are pieces that need to slot together — the oMLX server, the Docker containers for websum and coder, the Keychain integration for agent-approve, the model downloads (which can be temperamental — there’s a Hugging Face backend issue that occasionally causes hangs, fixable with one environment variable, but still annoying to hit the first time). I would not call it a weekend project for someone who isn’t comfortable with a terminal.

However: once it runs, it runs. I have a capable AI stack that works on flights, in venues with questionable WiFi, and in situations where I’d rather not have my work logged somewhere I can’t see. The 35B model handles most things I would have previously sent to a cloud API. The 122B model, when I need it, is genuinely impressive.

The whole thing fits in a bag. Nothing leaves the machine. That still feels good every time I open it.

Stay safe,
Raffael