Image credit: Olivier Porchet, Biorobotics Laboratory, EPFL
When a fish holds its position against a current in a river, its brain must figure out how fast to swim and how to steer to offset the water flow. Most fish use vision to register the world sliding past, detect optic flow speed and direction, and their brains turn these signals into compensatory swimming. Neuroscientists call this stabilizing reflex the optomotor response (OMR), from neural activity imaging. The retina captures signals of optic-flow direction, central pretectal neurons interpret direction, and spinal nerves drive muscle contractions.
The catch is that one cannot easily change the living brain to test how these circuits work. Although advances in imaging now allow detailed recording, and even manipulation, of neurons alongside behavior, rewiring connections to ask what a particular link actually does remains almost impossible in the living, complex animal.
A joint team from EPFL (Switzerland), Duke University (USA), and the Instituto Superior Técnico (Portugal) took on this challenge by creating a realistic larval zebrafish simulation and a biomimetic robot, published in Science Robotics. The team found a way to replicate the body and known neural circuit architectures of live zebrafish, first as a physics-based simulation, then as a free-swimming robot that autonomously navigates upstream using vision and these bio-inspired neural circuits.
The results revealed the minimal set of neural components needed for the OMR and, more interestingly, showed that this balanced neural circuit allows autonomous upstream navigation even in poor visibility. It also highlighted that the fish’s body and eyes are part of the neural computation.
A blueprint from the living fish brain
The starting point was work in the Naumann Lab at Duke, where detailed behavioral studies and whole-brain calcium imaging of larval zebrafish exposed to visual stimuli that mimicked riverbed optic flow, paired with circuit modeling, yielded a best-fit wiring diagram of the brain-scale OMR pathways.
Drawing on other insights about neural processing in the vertebrate retina and spinal cord, Dr. Xiangxiao Liu and Luca Zunino from the EPFL team used this neural circuit model to develop a neuromechanical simulation, simZFish, that not only recreates the larval fish body, complete with eyes and fins, but also takes this experimentally derived brain blueprint to be the simulation’s brain, opening new paths for neuroscience and brain-inspired robotics.
simZFish: a brain you can take apart
Built in the physics-based Webots simulator, simZFish reproduces a six-day-old larva at 1:1 scale: a tiny 4 mm body, weighing only 0.3 mg with seven segments, driven by six simulated motors, a head with two sideways-facing cameras for eyes, and realistic water fluid dynamics. With just the right head-to-tail weight balance, the simulated simZFish moves just like real larval zebrafish. Its artificial brain replicates the entire neural circuit found in fish, from light-changing pixels to muscle activation.
The pretectum is a visual brain region that contains neurons that receive direct input from the retina, computing motion directions. The artificial retina detects motion and feeds four types of direction-selective ganglion cells, which drive pretectal neurons that integrate and process visual information from both eyes, and downstream hindbrain motor command neurons that set how often the fish swims and which way it turns. Finally, to emulate how the real fish swims in intermittent bouts, a “bout gate” releases a burst of tail beats, producing the characteristic burst-and-glide swimming of real larval zebrafish.
Because every part is simulated, researchers can change any aspect of simZFish’s body or neural connection weights, delete or add neurons, or change the eye’s lens and see the consequences immediately. In contrast, with animal experiments, one can only record correlations of neural activation if the fish happens to execute the behavior in question, but cannot exclude that some other processing was going on or easily change or interact with the internal structure. simZFish turns that black box into an open, well-lit one with identified components, allowing the team to pinpoint the “minimal essential elements” for OMR behavior.
Figure 1. The larval zebrafish-sized simZFish can swim around in simulated water in a virtual Petri dish and be presented with an unlimited number of visual environments. SimZFish has two laterally placed virtual cameras that can ‘see’ the virtual environment (top-left boxes), a sensorimotor controller, and six motors linked in series in the tail, enabling it to capture and respond to visual motion information.
The body is part of the computation
One specific insight from building this bio-inspired system concerned retinal-brain connectivity. With cameras on the sides of simZFish’s head, optic flow generated when fish are dragged in a river produces conflicting swirls across the visual fields, highlighting a version of the classic “aperture problem”. Therefore, feeding the motion information from the entire simulated retina into the circuit caused those signals to cancel out, breaking the OMR behavior. When the team restricted input to the lower posterior part of the visual field, the behavior snapped back into place.
Strikingly, that is exactly the region that most strongly drives the OMR in real zebrafish, and it matches the large, lower-posterior receptive fields neuroscientists have recorded in the real fish’s pretectum. The insight is not so much that the simulation “reveals” the circuit, but that embodiment, in this case the perspective distortion of the laterally placed eyes, explains why the circuit is wired the way it is: the layout of the body and eyes dictates a configuration that captures the most useful motion information with the fewest connections, an efficient solution evolution appears to have found as well.
Closing the loop: a prediction sends the neuroscientists back to the microscope
The simulation also predicted that the original neural circuit model was incomplete because it could not readily test behavioral responses to new stimuli. However, when each simulated eye was shown motion in the opposite direction, a shearing stimulus the model had never encountered, simZFish1.0 turned far more than expected. When our lead neurobiologist, Dr. Matthew Loring, showed these visual stimuli to real zebrafish, they barely turned.
That mismatch sent the neuroscientists at Duke back to the microscope. Using volumetric two-photon calcium imaging, they recorded tens of thousands of neurons. They found that certain binocular neurons in the pretectum, specific response types that the circuit model relies on, existed in multiple previously overlooked subtypes. The team updated the too-simplistic simZFish1.0 model using neurons predicted by the simulation, which come in forward- and backward-tuned subtypes, with responses to one eye suppressed when the other eye sees forward motion.
Adding these subtypes and updating the connectivity produced simZFish 2.0, which reproduced the real fish’s behavior far better, including the response to the artificial shearing stimulus. “simZFish narrows the search, and the animal experiments verify the predictions, and together they complete the puzzle of the neural circuit,” said EPFL’s Auke Ijspeert, who directs the Biorobotics Laboratory (BioRob) at EPFL.
The final test came when the team released simZFish into a simulated flowing river, dragging its body with realistic forces, now with a realistic riverbed rather than artificial moving stripes. Strikingly, no matter which direction the simZFish started in, it eventually turned and swam upstream. “It was remarkable to observe that our experimentally derived neural circuits were allowing the simulation to navigate upstream autonomously”, marveled Eva Naumann, Assistant Professor of Neurobiology and Secondary Biomedical Engineering at Duke University.
Into the wild: a robot that keeps up with the current using vision alone
To see whether a neural circuit tuned in a clean simulation could cope with the real world, the team scaled the tiny simZFish design up into ZBot, an 80-centimeter, 2.7-kilogram robotic implementation carrying two real cameras, six tail motors, and a Raspberry Pi running the very same neural network, in a water-tight enclosed head.
Dropped into the sometimes-muddy, turbulent Chamberonne river near Lausanne, with plenty of river rocks, dappled light, drifting leaves, ZBot used the OMR circuit to counteract the water flow. With its visually activated OMR circuit, it stayed in the aerial drone camera’s view for much longer, 58 seconds, compared with roughly 37 seconds when its “eyes” were switched off and just 20 seconds when it drifted with the motors off (Figure 2, Video 1).
Figure 2. ZBot swimming in the Chamberonne River, Canton de Vaud, Switzerland.
It is the first demonstration that a fish could, in principle, fight a current using visual circuits alone. This overturns a long-held assumption that this behavior, rheotaxis, requires the mechanical “lateral line” network of sensory organs along the side of the fish body to detect water flow. “This is the first proof that effective rheotaxis can be achieved using visual neural circuits alone,” noted Professor Ijspeert.
Video 1. With the OMR neural model embodied, ZBot maintains its position far better than with random bouting (OMR circuit blinded) or passive drifting (motors off).
Why it matters for robotics
For robotics, the appeal is building more efficient, more autonomous agents. Most drones employ dedicated downward-facing cameras for position stabilization. By contrast, our method leverages the existing lateral cameras and requires no additional hardware, cutting both hardware and computational costs. Interestingly, this balancing circuit that compares across eyes seemed to handle poor visibility very well, likely also because the burst-and-glide swimming allowed the ZBot to integrate useful sensory information during the glide phase, when visual input was not degraded by self-motion.
The same bio-inspired burst-and-glide control of the ZBot underlies a companion study by the team showing that intermittent swimming saves energy, pointing toward lighter, longer-lasting aquatic robots that switch gaits as real fish do. Both the simulator and the robot designs are open-source, so other groups can reuse the models or build and test new circuits.
Yet, as the ZBot is roughly 200 times larger than its biological model and swims in a different fluid regime, it tests the circuit’s logic rather than the larva’s exact mechanics. The realistic simZFish’s visual circuit also holds its position only against a gentle flow, because the model currently cannot dynamically respond to different flow regimes. For the ZBot, about half the river trials were discarded for collisions or signal dropouts, probably because the stripped-down OMR circuit doesn’t allow the ZBot to avoid obstacles.
Even so, the approach the team built offers a powerful template for connecting brains, bodies, and behavior. “By integrating simulation, robotics, and animal experiments, we get a far more complete view of an animal’s neural model,” said Naumann, who runs a systems neuroscience lab testing how visual neural circuits interact in live zebrafish. Starting from these tiny baby fish, this iterative simulation-neurobiology-robotics approach is opening new ways to understand animal intelligence and to build efficient, brain-inspired aquatic robots.
Paper and team information
You can access the simZFish open-source platform here and explore more work on the Naumann Lab Github.
EPFL led simZFish development, neuromechanical modeling, and ZBot design and testing. Duke University led the zebrafish behavioral experiments, two-photon calcium imaging, and neural network modeling and data analysis. The Instituto Superior Técnico contributed modeling and theoretical analysis of the visual neuromechanical system.