Multi-agent reinforcement learning · from scratch
Five species — herbivores, predators, decomposers, pollinators, and engineers — each learn their own policy network at the same time, in a world that's adapting right back. No species is told what to do; they discover how to forage, hunt, flee, and reproduce by trial and error. Then the interesting part: the training run kept collapsing — and fixing it was the real project.
This is the actual learned behavior, recorded from the simulator. Predators stalk and thin the herd; the population settles into a coevolved balance. Flip to Untrained to see the same world with random actions — the difference is the learning.
Every dot is one training update's score (how healthy the whole ecosystem is). The original run climbs to a strong 194.8… then the predator policy quietly collapses, nothing controls the herbivores, they overgraze, and the whole food web craters into the negative — wasting 70% of the compute. The fixed run stays alive the whole way.
I traced the collapse to three independent failures and fixed each one — structurally, not with band-aids:
A near-deterministic policy made the PPO probability ratio overflow
float32 → inf → NaN, permanently
poisoning the weights. Clamping the log-ratio before exp()
fixed it in both the NumPy and PyTorch backends.
Herbivores spread across the whole map, so the per-cell breeding cap never engaged and they ran to 67×. A global ecological carrying capacity damps the runaway — verified to plateau instead of boom-bust.
The predator's sparse reward made it over-sharpen into a useless deterministic policy. An adaptive entropy controller plus a decaying learning rate kept it exploring — taking the run from always collapsing to stable in most seeds.
Built from scratch in NumPy + PyTorch: a vectorized 5-species ecosystem, PPO with GAE, multi-seed evaluation, checkpoint/resume, and a regression test suite. Trained on a GPU across five rounds of diagnosis and tuning.