SARA: Sharded Activation Reduction Architecture for Multi-CPU LLM Inference

GitHub (open implementation): source code

Introduction

Most LLM infrastructure is built for the datacenter problem: many users, large batches, and maximum aggregate throughput. Robotics is a different problem.

A robot usually has only one real client: itself. At any moment the model is generating one stream of thought: internal monologue, planning, tool selection, or outward dialogue. In that setting, the key metric is not requests per second. It is token latency for a single sequential stream.

At the same time, affordable robotic systems often have access to inexpensive CPU resources. What affordable robots often do not have is a large, expensive, and power-hungry GPU budget. That motivated us to build SARA, the Sharded Activation Reduction Architecture: a distributed inference path that uses multiple CPUs to reduce single-stream token latency.

SARA is designed for a narrow but important deployment regime:

one active LLM client
autoregressive generation, where each token depends on the previous one
multiple inexpensive CPUs available across networked nodes
low operational complexity preferred over datacenter-scale orchestration

The result is a system that is small, explicit, and practical. Instead of distributing an entire general-purpose serving stack, SARA distributes just the parts of transformer inference that naturally decompose into additive partial results.

Why robotics needs a different optimization target

In a server setting, batching is king. If many requests arrive together, the serving system can amortize expensive operations and push aggregate throughput much higher. That is exactly why systems such as vLLM emphasize continuous batching, high-throughput serving, and large-scale distributed parallelism.

A robotic brain rarely lives in that world. Most of the time there is exactly one stream that matters: the robot's own current reasoning loop. That makes the cost of waiting for the next token painfully visible. Slower token generation directly affects:

internal planning latency
deliberation during tool use
response time in spoken dialogue

For this use case, spare CPUs are more useful than multi-user schedulers. The problem becomes:

Can we turn multiple low-cost CPU nodes into one faster sequential inference path for a single model stream?

SARA answers yes, by sharding attention and FFN work across ranks and reducing the resulting partial contributions back into the residual stream.

The core idea

SARA exploits a simple fact about transformer blocks: some expensive parts can be partitioned into independent local computations whose results add back together after a linear projection.

For each transformer layer, SARA splits:

attention work by KV heads, with the corresponding query heads assigned through the query-to-KV replication factor
feed-forward work by FFN channels

Each rank computes only its local slice. The master then sums the full-width residual contributions from all ranks.

At a high level, one token step looks like this:

x = token_embedding(token)

for each layer l:
    x_hat = RMSNorm_att(x)
    broadcast Q8(x_hat) to all ranks
    each rank computes its local attention partial p_att[l, rank]
    x = x + sum_r p_att[l, r]

    x_hat = RMSNorm_ffn(x)
    broadcast Q8(x_hat) to all ranks
    each rank computes its local FFN partial p_ffn[l, rank]
    x = x + sum_r p_ffn[l, r]

logits = W_out * RMSNorm_out(x)

The name "Sharded Activation Reduction Architecture" comes directly from this loop:

the current activation is broadcast in quantized form, conserving network bandwidth
each shard computes a local contribution
the contributions are reduced on the master

The master is not just a coordinator. It also owns rank 0 and computes its own shard locally, then adds the workers' returned partials.

What SARA is, and what it is not

SARA is deliberately opinionated.

It is:

single-stream oriented
CPU-first
latency-focused
static-shard
explicit master-worker reduction

It is not:

a multi-tenant serving platform
a general remote device abstraction
a datacenter scheduler
a system designed primarily around GPU saturation

That distinction matters, because many existing LLM systems solve a different optimization problem.

Comparison with existing approaches

What SARA shares with other distributed inference systems

SARA is not alien to the existing literature or ecosystem. It shares important ground with model-parallel inference systems:

work is partitioned across devices or processes
local partial results are combined into a single forward pass
synchronization still exists during sequential token generation
performance depends on the balance between compute and communication

In that sense, SARA belongs to the same broad family as tensor-parallel inference.

Where SARA differs from general-purpose GPU serving stacks

Systems such as vLLM are designed around high-throughput serving, continuous batching, and rich distributed parallelism for many incoming requests. That is the right answer for datacenter serving. It is not automatically the right answer for a robot with one active reasoning stream and a few affordable CPUs.

SARA takes the opposite path:

no continuous batching
no multi-user scheduler
no assumption that the best hardware is a large GPU
no attempt to solve every deployment mode at once

The goal is narrower and, for robotics, often more relevant: reduce token latency for one stream by using multiple CPU shards effectively.

Where SARA differs from llama.cpp RPC

The closest comparison in spirit is probably llama.cpp RPC, but the designs are still quite different.

llama.cpp RPC exposes remote ggml devices and can distribute model weights and KV cache across local and remote devices. Its RPC backend has been described as experimental, with security and robustness caveats for untrusted networks.

SARA takes a much narrower route. Each participant can load the model locally, the shard split is fixed up front, and runtime communication consists of quantized activations and quantized partial contributions for attention and FFN. There is no general remote device layer to manage, no remote tensor cache to reason about during token generation, and no broad placement policy to tune.

That narrower scope is why we believe SARA is simpler, more robust, and more efficient for this specific use case. It is not trying to be a universal inference fabric. It is trying to make one robot think faster with a few cooperating CPUs.

A fair way to phrase the claim is this:

We believe SARA is a better fit for narrow single-user robotic brain deployments because its protocol surface, sharding policy, and runtime dataflow are smaller and more explicit than those of general-purpose remote-device frameworks.

That is a use-case claim, not a universal one. And for deployment work, use-case fit matters more than fashionable generality.

Efficiency and benchmark results

We evaluated SARA using two separate CPU nodes connected over Gigabit Ethernet. Both nodes used the same CPU model and operating system, so the distributed configuration added a second, comparable pool of compute resources over a standard network connection.

Experimental setup

Baseline: single-node CPU inference, generating one sequential token stream
Distributed configuration: two CPU nodes, each running a shard of the same model under SARA's master-worker architecture
Interconnect: Gigabit Ethernet (1 Gb/s)
Hardware and software: identical CPU model and operating system on both nodes
Models tested: open-weight Mistral 7B and Qwen3 24B
Metric: inference speedup relative to the corresponding single-node baseline; higher is better

Speedup is computed as:

Speedup = distributed generation throughput (tokens/s) / single-node generation throughput (tokens/s)

For a comparable sequential generation workload, this ratio also describes the improvement in average per-token generation time.

Results

Model	Nodes	Interconnect	Speedup vs. one node	Two-node scaling efficiency*
Mistral 7B	2	Gigabit Ethernet	1.951×	97.55%
Qwen3 24B	2	Gigabit Ethernet	1.948×	97.40%

*Scaling efficiency is defined here as measured speedup divided by the ideal 2.0× speedup for two equally capable nodes.

In other words, adding a second node delivered almost twice the single-node inference throughput for a single autoregressive stream, even with activation exchange and partial-result reduction crossing a standard Gigabit Ethernet link.

Why these results matter

For two equivalent CPU nodes, 2.0× is the ideal linear-scaling reference. Real distributed inference incurs synchronization, network transfer, and reduction costs, so performance below that ideal is expected.

The observed 1.951× and 1.948× speedups correspond to approximately 97.6% and 97.4% of ideal two-node scaling, respectively. These results suggest that, for the tested models and configurations, SARA's communication and coordination overheads are small relative to the compute work being distributed.

The fact that the measurements were obtained over Gigabit Ethernet is particularly relevant to robotics: this is a common, inexpensive interconnect, not a specialized high-bandwidth datacenter fabric. SARA therefore offers a practical route to lower token latency using modest networked CPU resources.

Open implementation and proprietary SARA

The linked GitHub implementation is intentionally limited by design. It exposes the core approach and provides a concrete, inspectable implementation of SARA's sharding and reduction path, but it does not include all optimizations or capabilities of the proprietary implementation.

The proprietary version of SARA achieves higher performance, with speedups of up to 1.98× in some two-node configurations relative to single-node inference. That figure is configuration-dependent and should be read as a best observed result, not a universal guarantee.

Licensing is open for discussion. If you are interested in licensing the proprietary SARA implementation for a robotics product or another CPU inference deployment, contact us to discuss your requirements and licensing options.

Practical advantages for robotic systems

For robotic brain workloads, SARA has several attractive properties.

1. It targets the right metric

The design is about sequential token latency for one active stream, not aggregate datacenter throughput.

2. It uses inexpensive hardware effectively

Ordinary CPUs on separate nodes can cooperate to reduce latency without requiring a large GPU. That matters for cost, power, thermals, and deployment flexibility.

3. It works over conventional networking

The two-node tests demonstrate near-linear scaling over Gigabit Ethernet, without relying on specialized datacenter interconnects.

4. It keeps the runtime small

The protocol is compact. The control flow is easy to trace. There are fewer moving parts than in a general-purpose serving stack.

Conclusion

SARA is a focused answer to a focused problem.

In robotics, the LLM is usually serving one client: the robot itself. That makes single-stream token latency the metric that matters. SARA attacks that metric directly by sharding attention heads and FFN channels across multiple CPUs, shipping quantized activations, and reducing full-width partial results back into the residual stream.

In tests with two identical-CPU, same-OS nodes connected over Gigabit Ethernet, SARA achieved 1.951× speedup on Mistral 7B and 1.948× on Qwen3 24B versus single-node inference. The proprietary implementation reaches up to 1.98× in certain configurations. These figures show how close a narrowly focused, CPU-first distributed inference architecture can come to ideal two-node scaling for a single generation stream.

The open implementation demonstrates the architecture while remaining intentionally limited; the proprietary implementation extends its performance capabilities and is available for licensing discussions upon request.

No related post found