A visual explanation of reinforcement learning

Science communicator Aatish Bhatia and neuroscientist Arkarup Banerjee released chapter two of their Myriad project, presenting an interactive simulation that demonstrates how neural circuits learn behaviors from scratch. The demonstration, detailed in aatishb.com's report, constructs an autonomous digital creature called a spark whose movement relies on four simple neurons. By pairing randomized trial variations with reward feedback, the project illustrates how reinforcement principles reshape synthetic brain structures into purposeful navigation systems.

This technical analysis covers the computational architecture of the spark agent, its parameter space navigation, and the algorithmic mechanics converting performance scores into structural wiring updates. Software engineers, roboticists, and machine learning practitioners will see how minimal reward signals guide optimization through complex decision spaces without manual supervision. The work bridges biological neural dynamics with automated algorithmic optimization.

01Four neurons and three tuning parameters

The digital spark operates with an economy of parts, housing four neurons divided between sensory input and motor action. Two front neurons act as eyes that register ambient light intensity, while two motor neurons drive flippers that propel the organism. Physical steering emerges directly from the mechanical interactions between these two independent propulsion channels.

Synaptic connectivity collapses into three adjustable variables rather than thousands of individual connection weights. Straight connections link sensors and flippers on the same side, crossed connections bind opposite sides, and a baseline drive parameter governs velocity in darkness. Tuning requires adjusting these weights inside a three-dimensional coordinate cube until the agent maintains a target light score above 0.5 over one hundred trials.

02Five developmental stages across two thousand trials

To remove human intervention, the creators introduced an automated learning algorithm that adjusts connections through repeated trials. Before each attempt, the three connection values shift slightly in random directions. When a run yields a higher score, the algorithm cements that shift, whereas inferior performance triggers a punitive adjustment in the opposite direction.

The spark begins as an untrained novice spinning in place before crawling forward after fifty trials. At two hundred fifty attempts, the juvenile picks up significant speed yet consistently veers away from the target beacon. Adolescent sparks reach five hundred trials charging toward illumination without braking, while adult mastery at two thousand trials produces an expert that tracks moving light and turns on a dime.

03Tangled parameter trajectories across six thousand runs

Plotting two thousand learning iterations produces a continuous coordinate path through the internal configuration cube rather than across physical ground. Different novice sparks begin with arbitrary initial connection weights scattered across the cube volume, tracing unique trajectories during optimization. The path zigzags sharply because every incremental adjustment introduces small amounts of random perturbation.

Simulating hundreds of autonomous learners over six thousand cumulative trials generates a dense web of overlapping trajectories. Despite distinct starting points, every path curves toward a shared attraction point in parameter space marked with an affinity icon. Averaging direction vectors across these jittery paths exposes how identical reward metrics funnel divergent agents into uniform behavioral strategies.

04Environmental shifts between fleeing and fighting

Learning dynamics transform when the environmental significance of the light source changes from sustenance to hazard. If light simulates a toxic chemical or predator, maximizing contact becomes fatal. In response, altering the reward function causes the spark to rewire its three parameters away from attraction and toward survival strategies.

When avoidance yields rewards, the spark quickly abandons light-seeking behavior and develops two distinct defensive reactions. One parameter combination produces high-speed flight that steers the agent away from light pulses as rapidly as possible, while another triggers freezing behavior where the creature plays dead. Conditioning rewards on approach velocity generates aggressive fighting behavior, proving that environmental incentives reshape behavior without modifying anatomy.

05Dopamine reinforcement and eighty-six billion neurons

The digital experiment mirrors the neurological mechanisms that govern biological skill acquisition in nature. Young songbirds learn courtship melodies by babbling random vocalizations and assessing how closely their sounds match memories of adult songs. Successful vocal approximations trigger dopamine releases in the avian brain, while dissonant deviations suppress dopamine, allowing birds to master complex acoustics without parental instruction.

The simulated creature models vehicle dynamics developed by neuroscientist Valentino Braitenberg, using a mathematical sigmoid transfer function to convert sensor signals into motor velocity. This minimalist four-neuron creature stands in sharp contrast to human physiology, where brains contain approximately eighty-six billion neurons and over one hundred trillion connections. Despite this difference in biological scale, reinforcement feedback serves as the compass that guides neural wiring through vast configuration spaces.