The One Body Problem
Why scaling stalls at the optimizer — and what changes when integration is done by rules instead of gradients
“Just buy more GPUs.”
That is the first answer most people reach for when a monolithic model gets too big to train, and it is not a foolish one. For a long stretch of the last decade it was simply correct.
But anyone who has actually watched a large training run knows that the bottleneck is often not the forward pass, and not the backward pass. It is the line that comes after both of them.
optimizer.step()
What that one line costs
Adam keeps two extra state tensors for every parameter — a first moment and a second moment. In a standard mixed-precision setup, a single parameter carries roughly sixteen bytes:
per parameter (mixed precision)
fp16 weight 2 bytes
fp16 gradient 2 bytes
fp32 master weight 4 bytes
fp32 first moment (m) 4 bytes
fp32 second moment (v) 4 bytes
─────────────────────────────────
16 bytes
A one-billion-parameter model therefore needs roughly sixteen gigabytes simply to hold the state of its own training. And every single step, all of it is read, updated, and written back, element by element.
Three properties of that operation matter more than its raw cost.
1. It is bound by memory bandwidth, not arithmetic
The Adam update is a handful of multiplies and adds per parameter, wrapped around several reads and a write. The arithmetic is trivial; the data movement is not. An operation shaped like this is limited by how fast bytes travel between memory and the compute units — not by how many floating-point operations per second the chip can theoretically sustain.
This matters because those two numbers have not grown at the same rate. Over the past two decades peak compute throughput has risen far faster than memory bandwidth, and each new generation of accelerator widens the gap. A step whose real cost is moving data does not get proportionally shorter when you buy a faster chip — and buying more chips does not help at all if each one is already waiting on its own memory.
2. It is globally synchronous
Now scale out. Data parallelism across many devices means that before any device can take its step, gradients must be reduced across all of them — every parameter, every step.
step t: forward → backward → ALL-REDUCE (every parameter) → update → step t+1
▲
every device waits here
The traffic each device moves in that collective is proportional to the size of the model, and the collective itself takes longer as more devices join it. Doubling the number of GPUs does not halve the time per step. Past some point, adding devices adds more synchronization than it removes work.
3. It cannot be parallelized along the step axis
This is the property people forget, and it is the hardest one. Update t+1 depends on the result of update t. That dependency is not an implementation detail to be engineered away — it is what gradient descent is.
training time = number of steps × time per step
Compute buys you a shorter second term, and only partly, for the reasons above. It buys you nothing at all on the first. You cannot run step 900,000 before step 899,999, no matter how large the cluster is.
The usual answers, and what they leave in place
None of this is news, and the field has good engineering answers. It is worth being precise about what each one actually changes.
Sharding the optimizer state (ZeRO and its relatives) splits those sixteen bytes per parameter across devices so that no single device holds all of it. This is a real and large win for memory. It does not remove the synchronization; it usually adds communication, trading bandwidth for capacity.
Sparse activation (Mixture-of-Experts) means only a fraction of the parameters participate in any given forward pass — a real win for inference cost and for compute per token during training. But training still backpropagates a single loss through the router and the selected experts, and those experts are still parameters of one model moving toward one objective.
Pipeline and tensor parallelism cut the model across devices so each holds a slice. They split where the computation happens. They do not split what the computation is optimizing.
Each of these is a way to carry one body with more hands. None of them makes it stop being one body.
Why does all of it have to move together?
Here is the question underneath the engineering. Why must the whole thing be updated at once, every step?
Because the whole thing is tied to a single differentiable objective. One loss runs through every parameter, and a gradient is by definition the derivative of that one number with respect to everything. So everything is coupled to everything else, and everything must move together, in lockstep, for the entire run.
That coupling is what produces all three properties above. The linear cost in N, the global synchronization, the strict sequencing — none of them are properties of Adam. Adam is a perfectly reasonable way to carry the load. They are properties of the load.
The real bottleneck is not the optimizer. It is the one body the optimizer is made to carry.
Seen that way, the ceiling stops looking like a hardware problem. More hardware removes none of the three properties, because none of them came from the hardware. It is a structural problem, and it wants a structural answer.
HIPLE does not slice gradient descent more cleverly
HIPLE stands for Hierarchical Independent Parallel Learning Engine. Given everything above, it would be easy to assume its answer is a better way to distribute training. It is not — and that is worth saying plainly, because it is the assumption people reach for first.
The individual engines train perfectly normally.
Each engine is a small, independently trained model, on its own domain and its own data, with ordinary backpropagation and ordinary Adam. There is no reason to change that and no attempt to. Gradient descent is an excellent tool at the scale where its costs are tolerable, and the entire point is to keep it at that scale.
Two things follow, and they are the whole argument.
First, N per engine is small. The three properties from the first half of this essay never grow to a size that matters. Sixteen bytes per parameter is not a problem when the parameter count is modest; a bandwidth-bound elementwise update is not a problem when the tensors fit comfortably; a sequential dependency across steps is not a problem when the steps are short.
Second, the engines share no loss function. There is no single number they are jointly minimizing. And where there is no shared objective, there are no gradients to exchange. The all-reduce in the diagram above does not get optimized — it does not exist.
monolithic HIPLE ────────── ───── one loss no shared loss one gradient over everything each engine its own global all-reduce every step nothing to reduce retrain the whole to change part revise the part
Then what holds them together?
If the engines do not share an objective, something else has to combine their judgments. The obvious move — the one the field reaches for — is to learn the combination: train a router, a gating network, a meta-learner on top. But that puts a single differentiable objective back over everything, and with it every property we just removed.
So HIPLE does not do that. Integration is not done by learning. It is done by a rules engine — Rules and Law.
No gradient flows through that layer. There is no backpropagation in it, no optimizer state, no global synchronization step. Instead, explicit and auditable rules take each engine’s judgment, integrate them, and decide which judgments to accept as knowledge.
Case
│
┌──────┼──────┐ each trained independently,
▼ ▼ ▼ no shared loss, no shared gradient
Engine Engine Engine
A B C
│ │ │
└──────┼──────┘
▼
Rules and Law gradient-free integration
(no backprop, verified against external evidence
no optimizer state)
│
┌──────┴──────┐
▼ ▼
Decision Abstain
That changes what maintenance means.
Conflicting evidence revises a rule, not the model. When new external evidence contradicts something the system holds, the rule is re-evaluated and revised. The engines are not retrained to absorb the correction.
A new domain is a new engine plus rules. Adding coverage does not mean retraining the whole system, because there is no whole system to retrain — only engines that do not depend on one another, and a rule set that grows.
Disagreement produces no answer. When the engines’ judgments conflict, or the evidence runs out, the rules do not resolve the conflict by averaging it away. They return nothing. The system holds.
That last one is not a fallback; it is the reason the separation is worth its cost. A learned integration layer is trained to produce an output, and it will produce one — confidently — for an input it has no business judging. A rule that requires agreement can decline.
An honest boundary
This approach buys a great deal, and it is not free.
It requires a domain with knowable structure. Medicine has one; physiology and anatomy have been accumulating structure for centuries. Industrial equipment, law, finance, and much of science have enough. Problems whose structure is genuinely unknown still need broad relational search, and HIPLE does not pretend otherwise.
It also moves work rather than deleting it. Integration quality becomes a rules problem instead of a learning problem. Rules have to be written, verified, and grown — real labor that gradient descent would otherwise have done implicitly. The claim is not that this is less work. It is that this work is auditable and the other kind is not.
And the boundaries between engines are a search prior, not a wall. The world is not perfectly separable: chest pain can originate in the abdomen, referred pain crosses anatomical regions, and cancer ignores the partition entirely. So the divisions have to be soft and overlapping, with explicit cross-cutting links living in the symbolic layer, able to activate another engine when evidence points across a boundary. Where no clean route exists, the system broadens its search, escalates, or abstains.
That commitment has to be measured, not asserted. Two numbers gate whether a partition can be trusted: cross-boundary recall — does the system still find what sits between its own divisions — and critical-miss rate. A structure that scores well on average while dropping the cases that cross its seams has not earned the efficiency it claims.
Learning stays local; integration is done by rules
The first half of this essay is an argument about arithmetic: the optimizer’s update is bandwidth-bound, globally synchronous, and strictly sequential, and no amount of hardware changes any of the three.
The second half is an argument about why. All three follow from a single differentiable objective binding every parameter into one body — which means the ceiling was never really about compute at all.
Scale the parts that scale well. Keep each learner small enough that gradient descent stays cheap. And where the parts must be joined, join them with something you can read, verify, and revise — something that is permitted to say it does not know.
What we need is not a machine that spins one large model faster. It is a structure that never tied all of it to one gradient to begin with.
— Kyewook Lee
HIPLE — Hierarchical Independent Parallel Learning Engine
Patent pending — U.S. Provisional Application No. 64/122,459