• by Laurie Sullivan , Staff Writer @lauriesullivan, January 29, 2020

Meet Meena, an open-domain chatbot developed by the Google Brain research team and trained on data mined and filtered from public domain social media conversations.

The chatbot has been trained on a neural conversational model that attempts to correct flaws in conversation. It is trained on about 40 billion words, with a focus on understanding the context of a conversation to provide a reasonable reply.

Some researchers, including  Google’s, are working on what the paper calls an open-domain chatbot to explore these aspects of artificial intelligence. Open-domain chatbots can engage in conversation on any topic, unlike closed-domain chatbots that respond to keywords or intents to accomplish specific tasks.

In one conversation the human says “Hi,” to which the Meena replies, “Hi, how are you?” The human says “All good! It’s cold.” Meena says “It’s pretty cold here too.” The human says “Where do you stay?” Meena says "Wisconsin. How about you?” to which the human replies “San Jose, California.” Meena says “Oh nice! I always wanted to go to California.” The human asks “Have you been there before?” Meena replies “No, I haven’t. Why!” The human replies “Just curious.”

The research paper, first spotted by 9to5Google, points to open-domain chatbots such as MILABOT, XiaoIce, Gunrock, Mitsuku, and Cleverbot3 that have human-like attributes, but rely on complex frameworks such as dialog managers with knowledge-based, retrieval-based, or rule-based systems.

The idea, according to researchers, is to make the chatbot human-like, so that its conversations make sense and can be specific.

Measuring the quality of Meena, the Google Brain team used the human evaluation metric Sensibleness and Specificity Average (SSA). The first part of the metric -- sensibleness -- is a requirement. To properly have a conversation with a human, a bot’s responses must make contextual sense -- the second required specificity, which prevents bots from hiding behind vague replies. This means the bot can distinguish between generic and lively responses.

Researches also considered static and interactive. In an interactive conversation humans could chat about anything they want. The better Meena accepted its training data, the more sensible and specific its chat responses became.

In the end, Meena achieved a 79% SSA score -- compared with humans, who typically score 86%.