The constraint

Most people who run models locally have more than one machine and use one at a time. The idea is to pool them: contribute capacity, draw on it, and treat a desktop, a laptop and a tablet as one endpoint. The hard part is not the inference — it is that consumer machines sit behind NAT, sleep without warning, and differ wildly in what they can hold.

What runs today

The aim is a BitTorrent-style protocol for inference. What actually runs is a QUIC overlay with NAT-friendly reverse connections: seeders dial out, and the gateway routes inference back down the live session. That is not a shortcut — hole-punching does not survive symmetric NAT, which is what a lot of home routers do, so a design that depends on it works in a demo and fails at a friend’s house. Public, private and federated networks are all first-class. It presents as one OpenAI-compatible endpoint for the whole pool — existing tools point at it unchanged — and behind that a node either serves local weights through MLX or llama.cpp, or brokers a compatible upstream. The caller cannot tell which, and that indifference is the point: it is what lets a pool of uneven machines act as one address instead of a list of special cases.

Decisions worth defending

• The iPad is a peer, not a client. The shipped iOS app loads a local model and joins the network as a node that others can route to. Treating it as a thin client would have been easier and would have missed the point.
• Drive the batching primitive directly, rather than vendor a wrapper. Continuous batching uses mlx-lm’s own BatchGenerator; the existing project that solves this is credited as prior art in NOTICE instead of copied. Fewer layers to debug when a Metal queue misbehaves.
• Quote the ratio, not the rate — then go and reproduce it. Continuous batching gives one node about 4.3× aggregate throughput at 32 concurrent streams, with single-stream latency unchanged (M1 16GB, Qwen3-1.7B 4-bit). That figure has now been taken three times — once in June and twice again three months later across two MLX releases — landing at 4.29×, 4.23× and 4.30×. A 30B MoE gains about 1.9× at 16 streams on an M1 Max 64GB, because dense models batch better; those are different machines at different concurrencies, and quoting one as though it covered both would be the easiest lie on this page.
• Measure it on a machine that is busy, because that is the only kind there is. Both re-runs were taken while the host was also serving Plex — load above 3 on eight cores. That is not a caveat to apologise for, it is the condition the software ships into: nobody pooling the machines they already own has a spare one sitting idle. The absolute rate moved with the load, 193.7 tok/s to 182.8, exactly as it should. The ratio did not. That is the whole argument for quoting a ratio and not a rate, and it is why the number is worth putting in writing.

What measuring it taught me

Two findings changed the design more than any feature did. Small local models fail at structured tool calling for reasons of output format, not ability — the same 1.5B model scored 0/3 with native tool calls and 5/5 with the tools described in the prompt, on identical hardware. And a total time figure is close to meaningless unless it says whether the model was already resident: the same 30B on the same machine, four hours apart, took 370s and 34s. The per-task breakdown shows why — the first run was paying to load the model.

Where it stands

Open source under Apache-2.0, on PyPI, with a shipped iOS and iPadOS app, at mycellm.ai. The coding agent built on it ships in the repo as an example.