Microsoft CaptionBot Uses A.I. to Identify Photos Without Making Tay's Mistakes
It knows what you look like.
Microsoft is still proud of its A.I. capabilities, even if its recent chatter bot Tay turned out a little more Hitlerian than one would hope for in man or machine. So now the company has rolled out an A.I. captioning software that tries to identify images in pictures. Simply upload an image, and the CaptionBot will try to tell you what is in the picture. We cannot vouch for the bot’s politics, but its image captioning ability seems pretty strong.
The technology uses various Microsoft features to break down the images. Computer Vision dissects the important elements of a photo. The Emotion API breaks facial expressions down into emoji categories, and the Bing Image API identifies the remaining image elements. Then, a natural language software configures its conclusions in simple English.
The site is best at identifying the images of celebrities, while more abstract images tend to confuse it. One might expect simple household objects and symbols to predominate, but presumably once the bot picks out a human face from a picture, it’s not too hard to run a reverse Bing image search on it. There must be millions of images of Steph Curry online, so this muddled ID is only partly as impressive as it seems.
Steph Curry, the Ultimate Frisbee champion.
In our tests, the bot also struggled with images that contain multiple elements. Like in a photo with a boat in front of the Statue of Liberty, the algorithm picked up the boat, just not the statute. So, it wasn’t wrong, but it really wasn’t right either.
Not wrong, just not really right either.
In general, though, the bot does a pretty good job of covering its bases, rarely striking out completely.
Is it unpatriotic that the CaptionBot did not recognize the Apollo launch?
The site does mention that it collects your photos and ratings to help improve the algorithm, but it doesn’t hold onto any identifying information. That’s a good thing, so Microsoft can’t sic Tay on you for searching pictures of its competitors. In fact, Microsoft seems to be playing it a little safe with the CaptionBot.
This is not simply a car, but we'll give it to them.
So when you venture into dangerous territory, CaptionBot just shuts down. Whether it’s just a coincidence or a programmed reaction to Tay’s bad press remains to be seen, but CaptionBot would rather take a time out than play games with the internet. Give it a test here, but please be gentle. If we break another one of Microsoft’s bots, the company may not ever let us have nice things again.
Good call, bot.
