How Image Recognition Enhances Human Understanding of the Natural World

For Eric Ralls, inspiration struck while playing a game of cornhole. He spotted a giant plant with beautiful flowers at a friend’s house in July 2012. Nobody at the backyard barbecue could identify the plant. An internet search didn’t help; neither did a browse of local flora.

“People are so caught up in their phones these days that they hardly notice what is happening in the real world that surrounds them,” Ralls tells Inverse. “This is just today’s reality. Rather than fight it, we feel the best way to bring people back to nature is to bring nature into their phones … use your phone to learn about nature just as you use your phone to look at what your friend just ate for lunch on Instagram.”

That’s how Ralls came up with PlantSnap, which boasts the ability to “recognize 90 percent of all known species of plants and trees” after you snap a photo of a mystery plant with your phone. Here’s how the company explains its technology: “Our algorithm analyzes a picture of a plant and uses images in our library of 90 million images and over 320,000 species to match the new photo submitted with every image in the database.”

It’s this train of thought that has inspired a new wave of apps, one that brings botany into the hands of the average smartphone user. Pl@ntNet is another app that’s identified 13,573 species of plants across 662,744 images. iNaturalist is a broader app that encourages users — it’s crowdsourced instead of using machine learning to ID plants — to identify plants and animals and share results with a community, and so far over 8 million observations of 150,000 species have been posted by over 200,000 people.

These apps are inspiring people to get creative and learn about their surroundings.

“Even in the middle of cities, you can find wild plants and animals if you know where to look,” Scott Loarie, co-director of iNaturalist, tells Inverse. “We think every curious person should be able to learn more about the most common plants, birds, and other wildlife in their area.”

Ralls’s first attempt at PlantSnap was unsuccessful. He asked the development team at RedOrbit, a science news website he founded a decade earlier, to look into a solution. After spending tens of thousands of dollars and three months, the team concluded it wasn’t possible — but the technology was just around the corner.

The shift came with the rise of deep learning, a form of artificial intelligence that learns to recognize objects through a training process. It’s the same technology that allows Adobe Photoshop to select a person, Google DeepMind to beat a human playing Go, and autonomous cars to learn how to drive.

Ralls first read about deep learning in 2016, and he grew convinced it could finally bring his idea to life four years after that summer party. He hired a team of external programmers and built the first beta in August of that year. For Ralls, the launch of the final version in 2017 was a moment that could help transform ecology.

“We find that normal people are simply taking an interest in nature now that they have an app in their phone that will tell them what they are looking at … another excuse to use your phone, almost like a game,” Ralls says.

PlantSnap has since moved from strength to strength: It’s in the final days of a crowd-funded investing round, and according to its mission statement on StartEngine, it’s now eyeing a global adoption.

The larger goal for apps like PlantSnap and iNaturalist is to use tech to enhance human understanding of the natural world. The City Nature Challenge, running from April 27-30, will see over 60 cities compete to ID the most local species and upload the results to iNaturalist. Teaching toolkits are designed to get kids as young as five snapping. The winning city will be announced May 4. Through these initiatives and more, deep learning has inspired technology to reconnect with nature.

“Thousands of groups around the world are using iNaturalist to collect data and learn more about what lives in their local parks during ‘BioBlitzes’,” Loarie says.

The apps can also help aid scientific research: Pl@ntNet has cataloged 13 “microprojects” aimed at encouraging smaller groups to get involved in research. A city farm, for example, can tell the app which plants are local to their area and get people to use the app to catalog the plants and learn more. One project, organized by Ph.D. student Christophe Botella, used geolocation data to map five invasive plant species in France, collected through the app, which could aid biodiversity conservation efforts.

“Our planet is rapidly changing before our eyes,” Ralls says. “Bringing people back to nature is vital for the future of our planet. When people notice, they start to realize how amazing this planet is, and they start to care about what happens to it and what they are leaving behind for their children.”