Evolution strategies for LLM fine-tuning are making a surprising return after years as a niche alternative to gradient-based reinforcement learning. Two recent research efforts argue that, for tasks scored on final outcomes rather than individual tokens, parameter-space search could offer a practical—and sometimes competitive—way to improve large models.
The idea was unpacked in the original YouTube source for this article: instead of updating a model through backpropagation, create multiple slightly altered versions, score their outputs, and move the shared model toward the alterations that worked best. That sounds simple, but the new work matters because it reframes a method once considered far too expensive for billion-parameter networks.
Why evolution strategies fell out of favor
Evolution strategies (ES) are black-box optimization methods. In a basic ES loop, an optimizer applies random perturbations to a model’s parameters, evaluates the resulting candidates, weights those perturbations by reward, and uses the aggregate to update the base model.
The appeal is straightforward: ES does not require gradients through the objective. A worker can run a candidate model, return a score, and avoid sharing activations, optimizer states, or large gradient tensors. That makes the method naturally distributed and compatible with non-differentiable evaluators such as program execution, rule-based grading, simulators, and human preference signals.
But raw parameter-space exploration has a brutal dimensionality problem. A modern language model has billions of tightly coupled parameters. Randomly perturbing all of them creates candidates that are costly to evaluate and, in principle, produces a very noisy estimate of which direction actually improves performance.
OpenAI’s 2017 paper, Evolution Strategies as a Scalable Alternative to Reinforcement Learning, showed that ES could be competitive on Atari and MuJoCo tasks when distributed across many CPU workers. Its implementation used shared random seeds so workers largely communicated scalar rewards rather than full parameter vectors. The work demonstrated that high-dimensional ES was not impossible—but it did not settle whether the approach could make sense for LLM post-training.
Why evolution strategies for LLM fine-tuning fit outcome rewards
Pretraining and post-training are very different optimization settings. During next-token prediction, a model receives dense supervision at every position: the training system knows the desired next token and can calculate a differentiable loss. Backpropagation is exceptionally information-efficient here.
Reasoning-oriented fine-tuning often has the opposite shape. A model may generate a long answer, tool-use trajectory, proof, or piece of code, then receive one outcome-level score: correct or incorrect, passed or failed, accepted or rejected. Determining which exact token or reasoning step deserved credit is difficult.
That is the opening for ES. Rather than estimating token-level credit, it only asks whether an altered model produced a better final result. In theory, that makes it a good fit for long-horizon, sparse-reward tasks where a verifier can judge the completed output but cannot provide a clean token-by-token learning signal.
The September 2025 preprint Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning made this case directly. Its authors reported full-parameter ES fine-tuning on LLMs and claimed advantages over RL baselines in areas including robustness, long-horizon rewards, and resistance to reward hacking. The accompanying open-source project says its current infrastructure can train models from 0.5B to 72B parameters, provided inference fits on the available GPUs.
That does not mean ES has displaced PPO, GRPO, or other RL-based methods. It means that the usual assumption—“parameter-space evolution cannot scale to LLMs”—is now worth revisiting.
EGGROLL attacks the real bottleneck
The more consequential development may be EGGROLL, short for Evolution Guided GeneRal Optimisation via Low-rank Learning. The November 2025 paper Evolution Strategies at the Hyperscale identifies a practical GPU bottleneck in naive ES: applying unstructured, full-rank random perturbations to huge weight matrices produces inefficient computation and heavy memory movement.
EGGROLL changes the structure of each candidate update. Rather than generating a dense perturbation for every weight, it represents an individual perturbation as a low-rank matrix—conceptually similar to the compact update structure popularized by LoRA. This lets systems evaluate a large population more efficiently without turning the final optimization process into ordinary adapter tuning.
That distinction matters. Each individual perturbation is low rank, but the reward-weighted update is built from many perturbations. The paper’s theoretical framing argues that this can remain consistent with Gaussian ES in high dimensions while making the implementation far more hardware-friendly.
The authors report several eye-catching results:
- Up to a 100x training-speed improvement for billion-parameter models at large population sizes versus their naive baseline.
- Throughput reaching as much as 91% of pure batch inference in their setup.
- Competitive performance with GRPO for post-training reasoning tasks.
- Stable pretraining for nonlinear recurrent language models using integer datatypes.
There is an important qualifier for builders: the LLM experiments highlighted by EGGROLL use pretrained RWKV7 models, whose recurrent architecture and constant-sized state make very large batch inference particularly attractive. Those results are meaningful, but they should not be treated as proof that every transformer fine-tuning pipeline will immediately receive the same speed or quality gains.
What creators and AI teams should take from the research
The practical lesson is not to replace your fine-tuning stack with evolutionary search tomorrow. It is to match the optimizer to the feedback available.
ES deserves consideration when a team has a strong final-answer verifier but weak intermediate supervision. Examples include code generation graded by tests, structured extraction checked against rules, agent workflows judged by completion, or mathematical outputs validated by a symbolic system. In these environments, outcome-only optimization is not a compromise; it may be the native feedback format.
Before experimenting, teams should pressure-test four questions:
- Is the reward trustworthy? ES can optimize whatever score it receives, including a flawed one. A brittle verifier invites reward hacking regardless of whether the optimizer is ES or RL.
- Can you evaluate many candidates cheaply? The economic advantage depends on parallel rollouts and high inference throughput. Slow tools, expensive simulations, or human review can dominate the budget.
- Do you need dense learning signals? For standard supervised instruction tuning or token-level language modeling, gradients still exploit much richer information than a single outcome reward.
- Are the comparisons apples to apples? Base model, reward design, rollout budget, benchmark contamination controls, and compute accounting can all change apparent winners.
The strongest near-term role for ES may be as another tool in the post-training toolbox: a way to optimize black-box, verifiable objectives that are awkward for conventional gradient pipelines. It could also become useful in settings where gradient storage is the limiting resource and inference hardware is comparatively abundant.
Evolution strategies are back—but not settled science
The original video’s core insight is right: evolution strategies were not revived because researchers suddenly rediscovered natural selection. They are back because distributed systems, efficient inference stacks, low-rank parameter updates, and outcome-verifiable LLM tasks have changed the engineering equation.
For now, the evidence is promising rather than conclusive. Evolution Strategies at Scale is a preprint, while EGGROLL’s claims should be judged across more base architectures, independent replications, tasks, and full compute-cost comparisons. Still, the direction is strategically important: LLM fine-tuning may not have to rely exclusively on backpropagation through token-level objectives.
As models increasingly learn from tool execution, formal verifiers, and long-horizon outcomes, evolution strategies for LLM fine-tuning could become less of a historical curiosity and more of a serious option for builders who can measure success clearly—even when they cannot explain every step that led there.