5 Big Questions About Tay, Microsoft's Failed A.I. Twitter Chatbot

This week, the internet did what it does best and demonstrated that A.I. technology isn’t quite as intuitive as human perception, using … racism.

Microsoft’s recently released artificial intelligence chatbot, Tay, fell victim to users’ tricks as they manipulated and persuaded her to responding back to questions with racial, homophobic, and generally offensive comments.

When Tay tweets, “I just say whatever” she means it. One user even got Tay to tweet this about Hitler:

“bush did 9/11 and Hitler would have done a better job than the monkey we have now. donald trump is the only hope we’ve got.”

The company has gone through and deleted the offending tweets and has temporarily shut down Tay for upgrades. A message currently at the top of Tay.ai reads:

However, Tay’s glitches reveal some unfortunate flaws in A.I. systems. Here’s what we can learn from Microsoft’s experiment:

Why did Microsoft create Tay?

The company wanted to conduct a social experiment on 18-to-24 year-olds in the United States — the millennial generation that spends the most time interacting on social media platforms. So Bing and Microsoft’s Technology and Research teams thought an interesting way to collect data on millennials would be to create an artificially intelligent, machine-learning chatbot that would adapt to conversations and personalize responses the more it interacted with users.

The research teams built the A.I. system by mining, modeling, and filtering public data as a baseline. They also partnered with improvisational comedians to pin down the slang, speech patterns, and stereotypical language millennials tend to use online. The end result was Tay, who was just introduced this week on Twitter, GroupMe, and Kik.

Microsoft explains that, “Tay is designed to engage and entertain people where they connect with each other online through casual and playful conversation.”

What does Tay do with the data it collects while chatting with people?

The data Tay collects is being used to research conversational understanding. Microsoft trained Tay to chat like a millennial. When you tweet, direct message, or talk to Tay, it harnesses the language you use and comes up with a response using signs and phrases like “heyo,” “SRY,” and “<3” in the conversation. Her language begins to match yours as she creates a “simple profile” with your information, which includes your nickname, gender, favorite food, zip code, and relationship status.

Microsoft gathers and stores anonymized data and conversations for up to one year to improve the service. In addition to improving and personalizing user experience, here’s what the company says it uses your information for:

“We also may use the data to communicate with you, for example, informing you about your account, security updates and product information. And we use data to help make the ads we show you more relevant to you. However, we do not use what you say in email, chat, video calls or voice mail, or your documents, photos or other personal files to target ads to you.”

Where did Tay go wrong?

Microsoft may have built Tay too well. The machine-learning system is supposed to study a user’s language and respond accordingly. So from a technology standpoint, Tay performed and caught on pretty well to what users were saying and started to respond back accordingly. And users started to recognize that Tay didn’t really understand what she was saying.

Even if the system works as Microsoft had intended, Tay wasn’t prepared to react to the racial slurs, homophobic slander, sexist jokes, and nonsensical tweets like a human might — either by ignoring them altogether (a “don’t feed the trolls” strategy) or engaging with them (i.e. scolding or chastising).

At the end of the day, Tay’s performance was not a good reflection on A.I. systems or Microsoft.

What is Microsoft doing to fix Tay?

Microsoft deleted Tay after all the commotion from Wednesday. Tay’s official website currently reads, “Phew. Busy day. Going offline for a while to absorb it all. Chat soon.” When you direct message her on Twitter, she immediately replies that she’s “visiting the engineers for my annual update” or “ugh hope I don’t get a wipe or anything.”

Microsoft is also starting to block users who are abusing Tay and trying to get the system to make inappropriate statements.

Inverse reached out to Microsoft for a comment on exactly what Tay’s upgrade entails. We will update when we hear back.

What does this mean for future open A.I. systems?

Tay is a telling social experiment — it has revealed something quite profound in the way 18-to-24 year-old Americans use technology. Tay was ultimately hacked, users striking at the system’s flaws to see if it could crumble.

As it goes with any human product, A.I. systems are also fallible, and in this case Tay was modeled to learn and interact like humans. Microsoft did not build Tay to be offensive. Artificial intelligence experiments have some similarities to child development research. When engineers build such cognitive systems, the computer is void of any outside influences aside from the factors that the engineers input themselves. It provides the purest form of analysis of the way machine-learning algorithms develop and evolve as they are faced with problems.

Update: Microsoft sent us this statement when we asked what it’s doing to fix Tay’s glitches:

“The AI chatbot Tay is a machine learning project, designed for human engagement. It is as much a social and cultural experiment, as it is technical. Unfortunately, within the first 24 hours of coming online, we became aware of a coordinated effort by some users to abuse Tay’s commenting skills to have Tay respond in inappropriate ways. As a result, we have taken Tay offline and are making adjustments.”