How to Simulate a 16-Device Cluster on Your Laptop (And Learn 4D Parallelism for Free)

Community Article
Published October 25, 2026

If you want to learn how modern LLMs are trained across dozens of GPUs or TPUs, the standard advice is pretty discouraging: rent an 8xH100 node for $25/hour and hope your bash script doesn’t hang on NCCL_INFO.

Most of us don’t have a multi-node cluster sitting under our desks. Because of that, concepts like Tensor Parallelism (TP), Pipeline Parallelism (PP), Context Parallelism (CP), and FSDP (ZeRO-3) stay abstract diagrams in papers rather than code you can actually break and fix.

Here is a trick that doesn't get talked about nearly enough: XLA (the compiler underneath JAX) doesn't care if your "devices" are $30,000 GPUs or threads on your laptop CPU. With a single environment flag, you can tell XLA to pretend your laptop has 16 separate accelerators, arrange them into a 4D mesh, and compile the exact same communication collectives (all-reduce, all-gather, reduce-scatter, collective-permute) that run on a real GPU cluster or TPU pod.

In this post, we’ll spin up a 16-device simulated cluster on a regular CPU, train a small Llama-style model with 4D parallelism, verify that its gradients match a single device down to float32 round-off, and load that 16-device checkpoint back onto a single device for inference.

Step 1: Tricking XLA into seeing 16 devices

Before JAX initializes its runtime, you can pass --xla_force_host_platform_device_count=N via XLA_FLAGS. Here is the exact helper from jaxotron/cli/train.py:

import os

def setup_cpu_devices(n: int) -> None:
    """Simulate n devices on the host CPU. Must run before JAX is imported."""
    if n:
        os.environ["JAX_PLATFORMS"] = "cpu"
        flags = [
            f for f in os.environ.get("XLA_FLAGS", "").split()
            if "xla_force_host_platform_device_count" not in f
        ]
        os.environ["XLA_FLAGS"] = " ".join(
            flags + [f"--xla_force_host_platform_device_count={n}"]
        )

If you call setup_cpu_devices(16) and then import jax, calling jax.devices() returns [CpuDevice(id=0), CpuDevice(id=1), ..., CpuDevice(id=15)]. Each simulated device gets its own buffer slices and executes the exact SPMD (Single Program, Multiple Data) program XLA generates for multi-device hardware.

To try this hands-on without writing boilerplate, grab Jaxotron, a minimal, ~2,500-line pure-JAX training codebase built specifically for hacking on 4D parallelism:

git clone https://github.com/rishiraj/jaxotron
cd jaxotron
pip install -e .

(Or install directly from GitHub: pip install "jaxotron @ git+https://github.com/rishiraj/jaxotron")

Step 2: Arranging 16 devices into a 5D Mesh

When you train a large model at scale, different parallelism strategies have very different communication appetites:

  • Tensor Parallelism (tp) splits individual matrix multiplications inside every single layer. It talks constantly, so it needs your fastest links (NVLink or TPU ICI).
  • Context Parallelism (cp) splits the sequence length across devices and shares Key/Value tensors.
  • Fully Sharded Data Parallelism (fsdp) shards weights and optimizer states across devices, gathering weights before each layer.
  • Pipeline Parallelism (pp) splits layers into stages and passes microbatch activations point-to-point from stage i to stage i+1.
  • Data Parallelism (dp) replicates weights and only averages gradients once at the end of a step.

In jaxotron/parallel/mesh.py, all five axes live in a single grid ordered from slowest/outermost (dp) to fastest/innermost (tp):

import jax
import numpy as np
from jax.sharding import Mesh

MESH_AXES = ("dp", "pp", "fsdp", "cp", "tp")

def create_mesh(pcfg, devices=None) -> Mesh:
    devices = list(jax.devices()) if devices is None else list(devices)
    shape = (pcfg.dp, pcfg.pp, pcfg.fsdp, pcfg.cp, pcfg.tp)
    device_array = np.asarray(devices, dtype=object).reshape(shape)
    return Mesh(device_array, MESH_AXES)

Let’s run a 4D training job (DP=2 × PP=2 × CP=2 × TP=2 = 16 devices) on our laptop and pass --show_mesh to see which coordinates each simulated CPU device owns:

jaxotron-train --cpu_devices 16 --dp 2 --pp 2 --cp 2 --tp 2 \
    --seq_len 128 --global_batch_size 16 --micro_batch_size 4 \
    --steps 60 --show_mesh

You’ll see the synthetic arithmetic-progression dataset generate automatically, followed by the mesh map and training loop:

Jaxotron | jax 0.4.35 | backend=cpu | devices=16 (cpu) | processes=1
Mesh: {'dp': 2, 'pp': 2, 'fsdp': 1, 'cp': 2, 'tp': 2}  ->  DP=2 x PP=2 x CP=2 x TP=2
Pipeline bubble: 20% of pipeline time is idle (raise micro_batch_size / --pp_microbatches to shrink it)
device      dp    pp    cp    tp   kind
---------------------------------------
0            0     0     0     0   cpu
1            0     0     0     1   cpu
2            0     0     1     0   cpu
...
15           1     1     1     1   cpu
Compiled the train step in 19.3s
step      1 | loss 6.9893 | grad_norm 2.541 | lr 0.00e+00 | ...
step     30 | loss 5.3702 | grad_norm 1.812 | lr 1.00e-03 | ...
step     60 | loss 2.9933 | grad_norm 0.943 | lr 5.00e-04 | ...

Look at device 0 and device 1: they share (dp=0, pp=0, cp=0) and only differ on tp. That means they hold the same pipeline stage (layers 0 and 1), read the same batch rows and sequence chunk, and split the attention heads and MLP columns between them.

Step 3: Does 4D parallelism actually compute the same gradients?

A classic bug when writing distributed training code is that your loss goes down, so you assume your sharding math is right, when in reality a subtle mask or reduction bug is silently corrupting your gradients.

The only way to be sure is to run one forward and backward pass on a single device, then run the exact same batch and random seed across every parallel layout and compare the gradients tensor-by-tensor.

Jaxotron ships with a CLI tool (jaxotron-compare) that compiles 8 different layouts on 16 simulated CPU devices and prints a live scorecard:

jaxotron-compare --cpu_devices 16
strategy             devices  KB/device  loss     max |Δgrad|  XLA collectives (mesh axis)
------------------------------------------------------------------------------------------
1-device baseline    1           1344.2  4.86102  0.00e+00     none
DP = 4               4           1344.2  4.86102  1.19e-07     all-reduce(dp)x12
FSDP = 4 (ZeRO-3)    4            336.6  4.86102  2.38e-07     all-gather(fsdp)x21, reduce-scatter(fsdp)x10
TP = 4               4            340.2  4.86102  2.38e-07     all-reduce(tp)x17, all-gather(tp)x2
TP = 4 + SP          4            340.2  4.86102  2.38e-07     all-gather(tp)x9, reduce-scatter(tp)x6
CP = 4 (seq split)   4           1344.2  4.86102  1.79e-07     all-reduce(cp)x14, all-gather(cp)x4
PP = 4 (GPipe)       4            478.2  4.86102  1.19e-07     all-reduce(pp)x4, collective-permute(pp)x5
4D (2x2x2x2)         16           369.2  4.86102  2.98e-07     all-reduce(dp/pp/cp/tp), all-gather(cp), collective-permute(pp)

Notice two things in that table:

  1. KB/device drops as expected: FSDP=4 and TP=4 cut per-device parameter memory by 4x; PP=4 cuts the stacked transformer blocks by 4x while keeping the embeddings replicated.
  2. max |Δgrad| is < 3e-7 across the board: whether you split the batch, the weights, the attention heads, the sequence, the layers, or all four at once, the gradients are identical to a single device up to floating-point associativity.

Step 4: Train on 16 devices, generate on 1

In many distributed frameworks, saving a checkpoint with TP=2, PP=2 writes out 4 separate shard files tied to that exact topology. If you want to load the model on a single GPU (or your laptop) to generate text, you have to run an offline checkpoint-merging script first.

Because Jaxotron uses stacked layer parameters and Orbax with target NamedSharding specs (jaxotron/checkpoint.py), checkpoints are completely topology-independent. Our 16-device 4D training run earlier saved its checkpoint to checkpoints/run. We can sample from it immediately on 1 CPU device using KV-cached generation:

jaxotron-generate --cpu --checkpoint_dir checkpoints/run --prompt "5 8 11 14" --temperature 0

Orbax reads the 16-way sharded arrays from disk, reshards them onto a single device on the fly, and continues the arithmetic progression (17 20 23 26...).

Community

Sign up or log in to comment