App We've All Been Waiting for Gets Recipes from Photos of Meals

Instagram is mainly good for one thing, which is making other people envious of your visually stunning life and food. It’s a noble mission, but with some unpleasant side effects: namely, that you might get jealous of other people’s beautiful meals and wonder how you, too, can make some of that tasty-looking stir fry. In a watershed event that rivals Google’s creation of the reverse image search, MIT researchers have come out with an app that will generate recipes based on photos you’ve taken of food.

It’s called “Pic2Recipe,” and you can try it now, even though it’s still in the demo stage. MIT’s Computer Science and Artificial Intelligence Laboratory created it using about 800,000 photos of meals matched with recipes, which they fed to their A.I. network, Recipe 1M. The A.I. analyzed connections between the food and recipes and identified patterns, which it now uses to produce recipes for newly-photographed food.

“Just like a human, it can infer the presence of invisible, homogenized or obscured ingredients using context,” Nicholas Hynes, co-author of a recent paper on the project, told NPR. “For instance, if I see a green colored soup, it probably contains peas — and most definitely salt!”

But it’s definitely not perfect yet. The app has trouble inferring flavors and textures, which, obviously, is crucial in cooking. “Without this, it might replace one ingredient with another because they’re used in similar contexts, but, doing so would significantly alter this dish,” said Hynes. “For example, there are two very similar Korean fermented ingredients called gochujang and doenjang, but the former is spicy and sweet while the latter is savory and salty.”

The app also sometimes gets confused between foods that look extremely similar, and it can’t really identify snack foods or other prepackaged foods, as The Verge found out when it tried to make the app produce recipes for potato chips and ramen.

Ultimately — once the app is refined and completely up-and-running — the researchers hope to use Pic2Recipe to track people’s health and eating habits over time and provide users with nutritional information about their food.

“In the far future, one might envision a robo-chef that fully understands food and does the cooking for you!” Hynes told NPR.

A robo-chef is cool, but honestly, right now it feels more pressing to just know how to make that delicious steak that just appeared on Instagram.