Quora, like competitors Yahoo Answers and wikiHow, is best-known as a spot for people of the internet to fire off whatever questions they have about the world — whether it’s how best to come to grips with the harsh realities of middle age, Elon Musk’s intelligence, or just if anyone else enjoys playing the banjo. Compared to other toxic spaces online where people communicate, Quora can be this buzzing hub filled with helpful human interaction. It’s kind of great. And humans being humans, the answers on Quora are often written in something resembling a story.
So it’s fitting that the website was recently made the unwitting playground for an A.I. system being trained to read short stories and evaluate narrative quality. But the scientists promise the A.I. agent they programmed to comb through Quora wasn’t intended to be a prototypical robot editor, really:
Our goal in this paper is not to use human surrogates and to build purely computational models for story understanding.
Their research was presented Wednesday at the International Joint Conference on Artificial Intelligence in Melbourne, Australia. Born from the idea that A.I. will one day be tasked with creating and publishing these same sorts of stories for humans, the scientists were motivated determine how an A.I. can make value judgments about text, opening the possibility for machines to write them in the future.
“Answers to questions like ‘what does it feel like to be poor’ are likely stories,” the authors explain. Because of how subjective the responses could be, and the fact that those queries compel respondents to provide some sort of narrative experience (usually a first-person account) to illustrate their answer, they were great testing areas. To that end, the five topics (out of 21) which yielded the most stories were:
- Survey Question
- Experiences in Life
- Life and Living
- Short Stories
- Life Lessons
Classic Quora.
The researchers collected 28,320 story texts from about 740 questions — each coming out to an average of 369 words. So not quite a short-story in terms of length — maybe something more like micro-fiction.
So, What Makes a Story Good to an A.I.?
Of course, just because an answer is popular doesn’t necessarily mean it’s a good story. The researchers had to figure out a way to allow its A.I. system to parse through the language in a way that identifies an answer as a story. Stories generally needed some sort of sequence of events in which details unfolded and revealed the intentions of characters. The A.I. researchers used Quora’s rating system — the website employs upvotes and downvotes à la reddit — as a proxy for narrative quality.
Using Amazon Mechanical Turk (a crowdsourcing service in which humans are able to accomplish tasks computers currently cannot), the researchers identified a set of parameters in a broad sample of “story” texts with which to create a model for training the A.I. to identify stories from non-stories.
From there, it was a matter of then optimizing the A.I. to understand why one story would be “better” than another, using a set of three reader models from which quality could be evaluated.
An 18 Percent Improvement
The results? Well, it’s hard to exactly say whether the A.I. truly worked. The metric for quality — Quora upvotes — is subjective, so it’s not entirely clear to what extent the A.I. can actually track quality, but we can play along for the purposes of this study itself.
Under that assumption, the researchers say the machine was able to identify textual quality at an 18 percent better rate than a “random forest” text evaluation system — a tree-like process of decision-making which branches into multiple possible outcomes as each step moves froward — that served as the baseline. So that’s a start.
In the short term, the A.I. could be optimized to better recommend stories to a reader, similar to how streaming apps like Spotify and Netflix can recommend new, unfamiliar content. In the long term, researchers may be able to use this work as a launching point from which a story-writing A.I. can be built and launched.
If you thought at least the creative class was safe from the A.I. take-over, you might want to think twice, and maybe figure out another set of skills to learn.
