← All articles

Clustered memory, now and next: a survey of local options

Large open models are usually limited by memory before they're limited by compute. When one box can't hold the weights and the KV cache, you need several boxes to act like one. Here's what people do locally today and what we hope to support. This post extends clustering and sharing models.

Status: Hypermesh serves models on single enrolled hosts today. Support for multi-node and clustered-memory hosts is planned, and the list below covers what we're evaluating. It isn't a commitment.

What works locally today

Apple silicon over Thunderbolt 5 (RDMA). Starting with macOS 26.2, Macs with Thunderbolt 5 can use RDMA over Thunderbolt. Apple says it was co-developed for MLX's distributed backend. The open-source exo project uses it to split models across Macs. Its documentation says every node must connect directly to every other node and all nodes must run the exact same macOS version. A university project report found that keeping control traffic on Ethernet and running data over the Thunderbolt mesh made their four-node cluster stable.

Multi-GPU in one box (PCIe and peer links). This is the most common and best-understood setup. Inference engines split layers or tensors across cards in a single machine. It's the easiest step up from one GPU.

Ethernet clusters. Tools like llama.cpp's RPC backend spread weights and KV cache across machines in proportion to each one's memory. The project calls this proof-of-concept and warns it isn't secure on open networks. Faster Ethernet and RDMA NICs help when a model is split by tensors rather than by layers.

Unified-memory boxes. Machines with large shared CPU-GPU memory, such as high-memory Macs and the NVIDIA Jetson AGX Orin class, can hold bigger models per node. That reduces how much splitting you need in the first place.

What's emerging: CXL and memory pooling

CXL lets servers attach and share memory over a fabric. Recent research uses CXL pools as a shared KV-cache tier for LLM serving. Examples include a September 2026 Kubernetes driver study and the Beluga KV-cache paper. The authors present these as research and feasibility results. Most of this targets datacenter hardware, and it's an emerging option for local hosts, not a mainstream one. llama.cpp contributors have also proposed topology-aware placement across NUMA, CXL, and GPU memory.

What we aim to support

In rough order of readiness:

  1. Multi-GPU single hosts.
  2. Small Apple silicon clusters over Thunderbolt 5.
  3. Ethernet and RDMA clusters of Linux hosts.
  4. Pooled memory such as CXL, once the hardware reaches the kind of machines our hosts run.

Each would ship as a supported host class after testing. We won't publish speed numbers until we've measured them ourselves.

Try it

Hypermesh connects hosts who have enrolled, checked hardware with renters who need time or tokens on it. Renters pay in USDC (or buy USDC through Stripe’s onramp), and hosts are paid through Stripe Connect. Running a cluster you'd like us to look at? See which host types we support and how to request one.


Sources: Apple, “TN3205: Low-latency communication with RDMA over Thunderbolt”; Apple Machine Learning Research, “Local and distributed LLM inference with MLX and RDMA”; exo; UMFT, “EXO deployment on Mac Studio network”; llama.cpp “RPC” and “memory distribution discussion”; Fan et al., “Composable CXL Memory…”; “Beluga”. Performance claims belong to their authors.