Guiding someone who has no idea where they’re going through bustling city streets takes both excellent communication skills and unwavering patience. Artificial intelligence is a master of the latter, but still has a lot of room for improvement when it comes to understanding and responding to humans.
In part because it’s such a perfect storm for miscommunication, computer scientists at Facebook’s A.I. research lab actually think they can improve A.I.’s mastery of communication by having it guide a virtual tourist through New York City. They’ve dubbed this project Talk the Walk and they tell Inverse this method could be the key to teaching machines how to master the nuance of language.
“The goal of the research project is to explore a more human-like approach to language learning,” explains University of Montreal Ph.D. student, Harm de Vries. “Previous work mostly focused on language learning from large [collections of text], like Wikipedia. [This] project aims to explore language learning in a contextual environment, making it possible to link words and concepts to perception and action in that environment.”
For starters, the researchers compiled a dataset of New York City neighborhoods, 360-degree photos of those locations, and sample conversations of humans guiding other humans around these same neighborhoods. From there they contrived two A.I. systems, one to play the tourist and one to play the seasoned local.
The virtual guide knew where it needed to take the virtual tourist and had access to a map, but it had no idea where the tourist was located. On the other hand, the tourist had only a 360-degree view of the area it was placed in but had no map or any kind of clue as to how to actually get to their destination.
In order to save this hopeless A.I., the guide had to ask the tourist to describe its surroundings. The lost computer would then describe restaurants, coffee shops, or any distinguishable landmark using the dialogue data that de Vries and his team had provided it. From there the guide would try its best to direct it.
Unfortunately, the team hasn’t been able to develop an A.I. that could successfully pull off the entirety of this task — guiding tourists is hard! — but their A.I. tour guide did get pretty good at locating its tourist. That’s notable, because it’s something that humans are actually worse at (“Which Starbucks are you in front of?!?”).
“My biggest surprise was how people were able to solve the task while using very noisy communication,” says de Vries. “Human guides were not able to localize the human tourists very well but still succeeded to complete the task in many cases. I’m excited to find out how they did that, and hopefully, enable machines to do the same.”
So turns out A.I. can actually take a page out of humans’ play book when it comes to learning how to walk the walk and talk the talk.
