Sarah Berry came to MIT as an assistant professor in MIT’s Department of Electrical Engineering and Computer Science (EECS) eager to focus on environmental challenges. She built her research career on the opportunity to use her expertise in computer vision, machine learning, and data science to solve real-world conservation and sustainability problems. Berry was drawn to the institute’s commitment to “computing for the planet” and decided to apply her methods to monitoring the environment and biodiversity on a global scale.
In the Pacific Northwest, salmon have a disproportionate impact on the health of their ecosystems, and their complex reproductive needs have caught Berry’s attention. Millions of salmon migrate each year to spawn. Their journey begins in freshwater river beds, where their eggs hatch. Young salmon smolts (newly hatched salmon) make their way to the ocean, where they spend several years maturing and reaching adulthood. As adults, salmon return to their natal streams to reproduce, ensuring the continuation of their species by depositing their eggs in the gravel of the riverbed. Both male and female salmon die shortly after providing the next generation of salmon in the river environment.
During their migration, salmon support a wide range of organisms in the ecosystems they pass through. For example, salmon bring nutrients such as carbon and nitrogen from rivers to the ocean, increasing their availability to those ecosystems. Additionally, salmon are key to many predator-prey relationships: They serve as a food source for a variety of predators such as bears, wolves, and birds, while also helping to control other populations such as insects through predation. After they die as a result of spawning, the decomposing bodies of salmon also replenish valuable nutrients for the surrounding ecosystem. Salmon migration not only maintains their species, but also plays a vital role in the overall health of the rivers and oceans in which they live.
Salmon populations play a significant role in the region, both economically and culturally. Commercial and recreational salmon fishing contributes significantly to the local economy. And for many indigenous peoples of the Pacific Northwest, salmon has significant cultural value, playing a central role in their diet, traditions, and ceremonies.
Monitoring salmon migration
Increased human activity, including overfishing and hydroelectric development, combined with habitat loss and climate change, has had a significant impact on salmon populations in the region. Effective monitoring and management of salmon fisheries is therefore important to ensure a balance between competing ecological, cultural, and human interests. Accurate counts of salmon during their seasonal migration to their natal rivers to spawn are essential for monitoring endangered populations, assessing the success of recovery strategies, guiding fishing season regulations, and supporting the management of commercial and recreational fisheries. Accurate population data helps decision-makers implement the best strategies to protect ecosystem health while adapting to human needs. Tracking salmon migration is a challenging and inefficient task.
Beery is currently leading a research project aimed at simplifying salmon tracking using advanced computer vision methods. The project aligns with Beery’s broader research interests, which focus on the interdisciplinary space between artificial intelligence, the natural world, and sustainability. Its connection to fisheries management made him a good fit for funding from MIT’s Abdul Latif Jameel Water and Food Systems (J-WAFS) Laboratory. Berry’s 2023 J-WAFS Seed Grant was the first research grant he has been awarded since joining the MIT faculty.
Historically, monitoring efforts have relied on people manually counting salmon from riverbanks using their eyes. Over the past few decades, underwater sonar systems have been used to count salmon. These sonar systems are essentially underwater video cameras, but they differ in that they use acoustics to register the presence of fish instead of light sensors. Using this method requires people to set up a tent by the river so they can count salmon based on the output of a sonar camera connected to a laptop. While this system is an improvement over the original method of tracking salmon by eye, it still relies heavily on human effort and is a laborious and time-consuming process.
Automated salmon monitoring is essential for better salmon fisheries management. “We need these technological tools,” Beery says. “We can’t keep up with the demands of monitoring, understanding and studying these really complex ecosystems that we’re working in without some automation.”
To automate the counting of migratory salmon populations in the Pacific Northwest, the project team, including Justin Kay, a doctoral student in EECS, collected data in the form of videos from sonar cameras in various rivers. The team annotates a subset of the data to train a computer vision system to independently identify and count fish as they migrate. Kay describes the process of how the model counts each migrating fish: “The computer vision algorithm is designed to determine the location of a fish in the frame, draw a box around it, and then track it over time. If a fish is detected on one side of the screen and leaves it on the other side of the screen, we count it as an upward movement. In the rivers where the team created training data for the system, it achieved robust results with counting errors of only 3 to 5 percent. This is well below the goal set by the team and partner stakeholders of a maximum of 10 percent counting errors.
Testing and Deployment: Balancing Human Effort and Using Automation
The researchers’ technology was deployed to track salmon migration in the newly restored Klamath River. Four dams on the river were recently demolished, making it the largest dam removal project in U.S. history. The dams collapsed after more than 20 years of campaigning to remove them, led by Klamath tribes in collaboration with scientists, environmental groups, and commercial fishermen. With the dams removed, 240 miles of river are now free-flowing, and nearly 800 square miles of habitat are accessible to salmon. Beery notes the almost immediate recovery of the Klamath River salmon population: “I think within eight days of the dam coming down, they started seeing salmon actually migrating upstream to the other side of the dam.” In collaboration with California Trout, the team is currently processing the new data to adjust and create a custom model that can be used to count newly migrating trout.
One of the challenges of this system revolves around training the model to accurately count fish in an unfamiliar environment with variations such as riverbed characteristics, water clarity, and lighting conditions. These factors can significantly change the appearance of fish in the sonar camera output and confuse the computer model. When deployed on new rivers where no data has been collected before, such as the Klamath, the system’s performance drops and the error rate increases significantly to 15–20 percent.
To overcome this challenge and create a scalable system that can be deployed in any location without human intervention, the researchers built an automatic adaptation algorithm into the system. This self-priming technology automatically calibrates itself to new conditions and environments to accurately count migrating fish. In testing, the automatic adaptation algorithm was able to reduce counting error to within 10 to 15 percent. The improvements in counting error through the self-priming feature mean the technology is closer to being deployable in new locations without additional human effort.
Enable real-time management with “Fishbox”
Another challenge the research team faced was developing an efficient data infrastructure. To operate the computer vision system, video generated by the sonar cameras must be delivered to the laboratory via the cloud or by manually sending hard drives from the river site. These methods have significant drawbacks: cloud access is limited by the lack of internet connectivity in remote river locations, and data transmission causes latency issues.
Instead of relying on these methods, the team built an energy-efficient computer called “Fishbox” that can be used to perform the processing in this context. The Fishbox consists of a small, lightweight computer with optimized software that fisheries managers can connect to their existing laptops and sonar cameras. The system is then able to run salmon counting models directly at sonar stations without the need for an internet connection. This allows managers to make decisions hour by hour, supporting more responsive management of salmon populations in real time.
Community development
The team is also working to unite the community around monitoring salmon fisheries management in the Pacific Northwest. “It’s very exciting to have stakeholders who are eager to access [our technology] as we deploy it and have closer integration and collaboration with them,” Beery says. “I think especially when you’re working on water and food systems, you need direct collaboration to help facilitate impact because you’re making sure that what you’re developing is truly meeting the needs of the people and organizations that you’re helping to support.”
Last June, Beery’s lab hosted a workshop in Seattle that brought together NGOs, tribes, and state and federal fish and wildlife departments to discuss the use of automated sonar systems to monitor and manage salmon populations. Kay notes that the workshop was “a fantastic opportunity to share all the different ways sonar is being used and to think about how the automated methods we’re building can fit into that workflow.” The discussion is now taking place via a shared Slack channel created by the team of more than 50 participants. The creation of this group is a significant achievement, as many of these organizations would otherwise not have the opportunity to come together and collaborate.
I’m looking forward to it.
As the team continues to fine-tune the computer vision system, improve its technology, and engage a variety of stakeholders—from indigenous communities to fisheries managers—the project is poised to significantly improve the efficiency and accuracy of salmon monitoring and management in the region. And as Beery advances his MIT group’s work, the J-WAFS seed grant helps keep challenges like fisheries management in perspective.
“The fact that there was an initial J-WAFS grant at MIT allowed us to continue working on this project until we moved here,” says Beery.
As J-WAFS celebrates its 10th anniversary this year, the program aims to continue to support and encourage MIT faculty members to pursue innovative projects that aim to expand knowledge and create practical solutions with real-world impacts on global water and food system challenges.
